beyondhj opened a new issue, #12589:
URL: https://github.com/apache/gravitino/issues/12589
### Version
main branch
### Describe what's wrong
**Title:** JDBC catalog operations fail with `CommunicationsException` after
idle period when using MySQL backend
---
### Describe what's wrong
When using Gravitino with Paimon or Iceberg catalogs that store metadata in
a MySQL database via JDBC, operations such as `listSchemas` (or `listDatabases`
for Paimon) occasionally fail after the system has been idle for some time.
The failure stack trace shows that the underlying MySQL connection has been
closed by the server due to exceeding the configured `wait_timeout`, but the
application attempts to reuse the stale connection, resulting in a
`CommunicationsException` with a `Broken pipe` cause.
This affects both:
- `PaimonCatalogOperations.listSchemas` → `JdbcCatalog.listDatabases`
- `IcebergCatalogOperations.listSchemas` → `JdbcCatalog.listNamespaces`
The error is observed in the Gravitino server logs when calling the REST API
or through the UI.
---
### Error message and/or stacktrace
For Paimon catalog:
```
Failed to operate object operation [LIST] under [paimon_550222763176624128],
reason [Failed to execute query: SELECT DISTINCT database_name FROM
paimon_tables WHERE catalog_key = ?]
java.lang.RuntimeException: Failed to execute query: SELECT DISTINCT
database_name FROM paimon_tables WHERE catalog_key = ?
at org.apache.paimon.jdbc.JdbcCatalog.fetch(JdbcCatalog.java:462)
at org.apache.paimon.jdbc.JdbcCatalog.listDatabases(JdbcCatalog.java:163)
...
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last
packet successfully received from the server was 53,516,254 milliseconds ago.
The last packet sent successfully to the server was 53,516,256 milliseconds
ago. is longer than the server configured value of 'wait_timeout'.
...
Caused by: java.net.SocketException: Broken pipe
```
For Iceberg catalog:
```
Failed to operate object operation [LIST] under
[iceberg_526363515375091712], reason [Failed to execute query: SELECT DISTINCT
table_namespace FROM iceberg_tables WHERE catalog_name = ?]
org.apache.iceberg.jdbc.UncheckedSQLException: Failed to execute query:
SELECT DISTINCT table_namespace FROM iceberg_tables WHERE catalog_name = ?
at org.apache.iceberg.jdbc.JdbcCatalog.fetch(JdbcCatalog.java:813)
at org.apache.iceberg.jdbc.JdbcCatalog.listNamespaces(JdbcCatalog.java:437)
...
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last
packet successfully received from the server was ... longer than wait_timeout.
...
Caused by: java.net.SocketException: Broken pipe
```
### Error message and/or stacktrace
Failed to operate object operation [LIST] under [paimon_550222763176624128],
reason [Failed to execute query: SELECT DISTINCT database_name FROM
paimon_tables WHERE catalog_key = ?]
java.lang.RuntimeException: Failed to execute query: SELECT DISTINCT
database_name FROM paimon_tables WHERE catalog_key = ?
at org.apache.paimon.jdbc.JdbcCatalog.fetch(JdbcCatalog.java:462)
at org.apache.paimon.jdbc.JdbcCatalog.listDatabases(JdbcCatalog.java:163)
at
org.apache.paimon.catalog.DelegateCatalog.listDatabases(DelegateCatalog.java:66)
at
org.apache.gravitino.catalog.lakehouse.paimon.ops.PaimonCatalogOps.listDatabases(PaimonCatalogOps.java:69)
at
org.apache.gravitino.catalog.lakehouse.paimon.PaimonCatalogOperations.listSchemas(PaimonCatalogOperations.java:144)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$0(SchemaOperationDispatcher.java:83)
at
org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:148)
at
org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86)
at
org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:143)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$1(SchemaOperationDispatcher.java:83)
at
org.apache.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:98)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$2(SchemaOperationDispatcher.java:81)
at
org.apache.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:49)
at
org.apache.gravitino.catalog.SchemaOperationDispatcher.listSchemas(SchemaOperationDispatcher.java:77)
at
org.apache.gravitino.catalog.SchemaNormalizeDispatcher.listSchemas(SchemaNormalizeDispatcher.java:60)
at
org.apache.gravitino.listener.SchemaEventDispatcher.listSchemas(SchemaEventDispatcher.java:77)
at
org.apache.gravitino.hook.SchemaHookDispatcher.listSchemas(SchemaHookDispatcher.java:62)
at
org.apache.gravitino.server.web.rest.SchemaOperations.lambda$listSchemas$0(SchemaOperations.java:110)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:53)
at org.apache.gravitino.server.web.Utils.doAs(Utils.java:209)
at
org.apache.gravitino.server.web.rest.SchemaOperations.listSchemas(SchemaOperations.java:100)
at jdk.internal.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
at
org.apache.gravitino.server.authentication.AuthenticationFilter.lambda$doFilter$0(AuthenticationFilter.java:106)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:53)
at
org.apache.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:103)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.apache.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:111)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.apache.gravitino.server.web.HttpAuditFilter.doFilter(HttpAuditFilter.java:162)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.apache.gravitino.server.web.RequestContextFilter.doFilter(RequestContextFilter.java:59)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: The last
packet successfully received from the server was 53,516,254 milliseconds ago.
The last packet sent successfully to the server was 53,516,256 milliseconds
ago. is longer than the server configured value of 'wait_timeout'. You should
consider either expiring and/or testing connection validity before use in your
application, increasing the server configured values for client timeouts, or
using the Connector/J connection property 'autoReconnect=true' to avoid this
problem.
at
com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:165)
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:55)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:988)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1056)
at org.apache.paimon.jdbc.JdbcCatalog.lambda$fetch$12(JdbcCatalog.java:453)
at
org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:68)
at org.apache.paimon.jdbc.JdbcCatalog.fetch(JdbcCatalog.java:446)
... 97 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last
packet successfully received from the server was 53,516,254 milliseconds ago.
The last packet sent successfully to the server was 53,516,256 milliseconds
ago. is longer than the server configured value of 'wait_timeout'. You should
consider either expiring and/or testing connection validity before use in your
application, increasing the server configured values for client timeouts, or
using the Connector/J connection property 'autoReconnect=true' to avoid this
problem.
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at
com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:52)
at
com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:95)
at
com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:140)
at
com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:156)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:635)
at
com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:689)
at
com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:938)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:817)
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:963)
... 101 more
Caused by: java.net.SocketException: Broken pipe
at java.base/sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:420)
at java.base/sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:440)
at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:826)
at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1035)
at
java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
at
java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
at
com.mysql.cj.protocol.a.SimplePacketSender.send(SimplePacketSender.java:48)
at
com.mysql.cj.protocol.a.TimeTrackingPacketSender.send(TimeTrackingPacketSender.java:43)
at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:628)
... 105 more
### How to reproduce
1. Deploy Gravitino with a Paimon or Iceberg catalog configured to use MySQL
as the JDBC metadata store (e.g., set `jdbc.url`, `jdbc.user`, `jdbc.password`).
2. Ensure the MySQL server has a `wait_timeout` value (e.g., 53,516,254
milliseconds) that is shorter than the typical idle period of the Gravitino
server.
3. Start Gravitino and perform a successful schema listing operation to
establish a database connection.
4. Leave the system idle for a duration longer than MySQL’s `wait_timeout`.
5. Perform another schema listing operation (e.g., via REST API or the UI).
6. The operation fails with the above `CommunicationsException`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
