Pencil-hzp opened a new issue, #68080: URL: https://github.com/apache/doris/issues/68080
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version The issue was reproduced on: - Apache Doris: 2.1.11 - JDBC Catalog: Microsoft SQL Server - HikariCP: 4.0.3 I also reviewed the current Doris master branch. The current master still uses the same general initialization pattern in `BaseJdbcExecutor`: a JDBC connection is borrowed from HikariCP before statement initialization is completed. If initialization fails after `getConnection()` succeeds, the borrowed connection needs to be explicitly released. The current master uses HikariCP 6.0.0, but HikariCP still has the `aliveBypassWindowMs` optimization, so upgrading HikariCP itself does not eliminate this trigger condition. ### What's Wrong? There appears to be a JDBC connection leak path in `BaseJdbcExecutor` when a connection has already been successfully borrowed from HikariCP, but subsequent executor initialization fails. The relevant initialization flow is conceptually: ```java conn = hikariDataSource.getConnection(); initializeStatement(conn, config, sql); ``` If: ```text getConnection() succeeds ↓ initializeStatement() fails ↓ BaseJdbcExecutor initialization fails ``` the already borrowed connection must always be returned to HikariCP. Otherwise, because initialization did not complete successfully, the normal `BaseJdbcExecutor.close()` lifecycle may never be reached and the corresponding Hikari `PoolEntry` can remain permanently `IN_USE`. ### Production symptom We reproduced a connection pool exhaustion problem using a Microsoft SQL Server JDBC Catalog. The Hikari active connection count gradually increases over time. Eventually the pool reaches: ```text total = maximumPoolSize active = maximumPoolSize idle = 0 ``` After that, JDBC Catalog queries fail because HikariCP cannot provide another connection. Changing `connection_pool_max_size` causes Doris to create another Hikari pool, which temporarily restores service. ### Runtime evidence Using Arthas attached to the JVM embedded in Doris BE, we inspected the leaked Hikari `PoolEntry` instances. The leaked entries consistently have this state: ```text PoolEntry state = IN_USE physical connection closed = true openStatements = 0 ``` Examples observed in our environment: ```text ConnectionID:13599 | physicalClosed=true | statements=0 ConnectionID:13633 | physicalClosed=true | statements=0 ConnectionID:13557 | physicalClosed=true | statements=0 ConnectionID:13578 | physicalClosed=true | statements=0 ConnectionID:13581 | physicalClosed=true | statements=0 ConnectionID:13680 | physicalClosed=true | statements=0 ConnectionID:13662 | physicalClosed=true | statements=0 ConnectionID:13665 | physicalClosed=true | statements=0 ConnectionID:13580 | physicalClosed=true | statements=0 ConnectionID:13595 | physicalClosed=true | statements=0 ConnectionID:13627 | physicalClosed=true | statements=0 ConnectionID:13660 | physicalClosed=true | statements=0 ``` These PoolEntry objects remain `IN_USE` indefinitely even though the underlying SQL Server physical connection has already been closed and there are no open statements. Therefore each occurrence permanently consumes one connection-pool slot. ### Confirmed SQL Server abort path We also traced the physical connection lifecycle with Arthas. Doris explicitly invokes `abort()` on SQL Server JDBC connections through this runtime call chain: ```text BaseJdbcExecutor.close() -> SQLServerJdbcExecutor.abortReadConnection() -> HikariProxyConnection.abort() -> SQLServerConnection.abort() ``` Therefore the closed physical connections are not caused only by an external network or SQL Server event. Doris itself can intentionally abort the SQL Server connection as part of the executor close path. Afterwards, Hikari handles recycling/eviction of that connection. ### Hikari alive bypass window We additionally traced Hikari connection borrow/recycle events. For one connection: ```text 16:59:50.549 borrow ConnectionID:13595 16:59:50.554 recycle ConnectionID:13595 16:59:50.964 borrow ConnectionID:13595 again ``` The second borrow occurred approximately 410 ms after recycle. Later this PoolEntry was observed as: ```text ConnectionID:13595 IN_USE physicalClosed=true statements=0 ``` Another example: ```text 16:59:53.451 borrow ConnectionID:13599 16:59:53.456 recycle ConnectionID:13599 16:59:53.930 borrow ConnectionID:13599 again ``` The second borrow occurred approximately 474 ms after recycle. It later remained: ```text ConnectionID:13599 IN_USE physicalClosed=true statements=0 ``` HikariCP has an `aliveBypassWindowMs` optimization whose default value is 500 ms. A recently used connection can therefore be borrowed again without a liveness check during this window. This provides a plausible trigger for the issue: ```text Doris aborts SQL Server physical connection ↓ connection is recycled/handled by Hikari ↓ connection is borrowed again very quickly ↓ Hikari liveness check may be bypassed ↓ Doris receives an already closed physical connection ↓ executor initialization fails ↓ borrowed PoolEntry is not released ↓ IN_USE + physicalClosed=true ``` The final initialization-failure step above is the suspected connection leak path based on the current `BaseJdbcExecutor` resource lifecycle. The important underlying issue is independent of why the connection is invalid: > After `hikariDataSource.getConnection()` succeeds, every subsequent > initialization failure path must guarantee that the borrowed connection is > released. ### What You Expected? Once Doris successfully obtains a JDBC connection from HikariCP, every subsequent failure during `BaseJdbcExecutor` initialization should release all partially initialized JDBC resources before propagating the exception. Expected lifecycle: ```text Hikari getConnection() ↓ initialize executor ↓ initialization fails ↓ close Statement / ResultSet if created ↓ close/return Connection ↓ throw JdbcExecutorException ``` A failed initialization should never leave a Hikari PoolEntry permanently in this state: ```text IN_USE physicalClosed=true openStatements=0 ``` Even if the physical JDBC connection has already been closed or aborted, the Hikari proxy still needs to be properly closed/released so that the PoolEntry can be evicted instead of permanently consuming a connection-pool slot. ### How to Reproduce? ### How to Reproduce? The issue is timing-sensitive, but it can be reproduced with a Microsoft SQL Server JDBC Catalog by repeatedly executing simple queries. #### 1. Create a SQL Server JDBC Catalog Using a relatively small connection pool makes the problem easier to observe. For example: ```sql CREATE CATALOG sqlserver_catalog PROPERTIES ( "type" = "jdbc", "user" = "xxx", "password" = "xxx", "jdbc_url" = "jdbc:sqlserver://xxx:1433;databaseName=testdb;encrypt=false", "driver_url" = "xxx", "driver_class" = "com.microsoft.sqlserver.jdbc.SQLServerDriver", "connection_pool_max_size" = "10" ); ``` Use the default HikariCP `aliveBypassWindowMs` configuration. Do NOT start BE with: ```text -Dcom.zaxxer.hikari.aliveBypassWindowMs=0 ``` #### 2. Repeatedly execute a simple query through the JDBC Catalog For example: ```sql SELECT TOP 100 * FROM test_table; ``` Execute the query repeatedly. Running multiple queries concurrently makes the timing issue easier to trigger. The important point is not the specific SQL statement. The issue is related to the JDBC connection lifecycle rather than any specific table or business SQL. #### 3. Observe the Hikari pool inside BE Attach Arthas to the JVM embedded in the Doris BE process. List Hikari pools: ```bash vmtool --action getInstances \ --className com.zaxxer.hikari.pool.HikariPool \ --limit 500 \ --express 'instances.{#this.toString()+" | total="+#this.getTotalConnections()+" | active="+#this.getActiveConnections()+" | idle="+#this.getIdleConnections()+" | waiting="+#this.getThreadsAwaitingConnection()}' ``` During reproduction, active connections may gradually accumulate. Eventually the affected pool may reach: ```text total=10 active=10 idle=0 ``` At this point, subsequent JDBC operations fail to acquire a connection. #### 4. Inspect the leaked PoolEntry objects Identify the affected pool, for example `HikariPool-XXX`, and execute: ```bash vmtool --action getInstances \ --className com.zaxxer.hikari.pool.PoolEntry \ --limit 500 \ --express 'instances.{?#this.getPoolName().equals("HikariPool-XXX") && #this.getState()==1}.{#this.connection.toString()+" | physicalClosed="+#this.connection.isClosed()+" | statements="+#this.openStatements.size()}' ``` The leaked entries can be observed in the following state: ```text IN_USE physicalClosed=true statements=0 ``` These entries remain `IN_USE` indefinitely even though the underlying physical JDBC connection is already closed. #### 5. Trace the SQL Server abort path The following Arthas watch can be used to observe SQL Server connection aborts: ```bash watch com.microsoft.sqlserver.jdbc.SQLServerConnection abort \ '{target.toString(),@java.lang.Thread@currentThread().getName()}' \ -b -x 3 ``` The observed call chain in our environment is: ```text com.microsoft.sqlserver.jdbc.SQLServerConnection.abort() com.zaxxer.hikari.pool.HikariProxyConnection.abort() org.apache.doris.jdbc.SQLServerJdbcExecutor.abortReadConnection() org.apache.doris.jdbc.BaseJdbcExecutor.close() ``` #### 6. Verify the workaround Restart BE with: ```text -Dcom.zaxxer.hikari.aliveBypassWindowMs=0 ``` Ensure that a new Hikari pool is created after applying the property. Verify the actual value: ```bash vmtool --action getInstances \ --className com.zaxxer.hikari.pool.HikariPool \ --limit 500 \ --express 'instances.{#this.toString()+" | aliveBypassWindowMs="+#this.aliveBypassWindowMs}' ``` The new pool should report: ```text HikariPool-XXX | aliveBypassWindowMs=0 ``` Repeat the same query workload. In our environment, after `aliveBypassWindowMs=0` was actually applied to a newly created pool, the `IN_USE + physicalClosed=true` leak could no longer be reproduced. ### Anything Else? ### Workaround The following JVM system property currently prevents the issue in our environment: ```text -Dcom.zaxxer.hikari.aliveBypassWindowMs=0 ``` This should only be considered a workaround. It prevents HikariCP from quickly handing an invalid recently used connection back to Doris without a liveness check, but it does not fix the resource management issue in `BaseJdbcExecutor`. It may also add additional connection validation overhead. ### Why this should be fixed in Doris A JDBC connection can become invalid for many reasons: - Doris intentionally aborting the physical connection - database-side disconnect - socket/network failure - database restart - firewall/NAT timeout - JDBC driver error - other transient connection failures Therefore Doris should not rely on HikariCP always returning a usable connection. The lifecycle should guarantee: ```text getConnection() succeeds ↓ ownership of the borrowed connection belongs to BaseJdbcExecutor ↓ ANY later initialization failure ↓ connection must be released ``` ### Possible fix A possible fix would be to make `BaseJdbcExecutor.init()` clean up all resources that may have been created before the initialization failure. Conceptually: ```java try { conn = hikariDataSource.getConnection(); initializeStatement(conn, config, sql); } catch (Exception e) { try { if (resultSet != null) { resultSet.close(); } } catch (Exception closeException) { LOG.warn("Failed to close ResultSet after JDBC initialization failure", closeException); } try { if (stmt != null) { stmt.close(); } } catch (Exception closeException) { LOG.warn("Failed to close Statement after JDBC initialization failure", closeException); } try { if (conn != null) { conn.close(); } } catch (Exception closeException) { LOG.warn("Failed to release JDBC connection after initialization failure", closeException); } resultSet = null; stmt = null; conn = null; throw new JdbcExecutorException("Initialize datasource failed: ", e); } ``` This is only an example. The actual fix should reuse the existing Doris JDBC resource cleanup mechanism where appropriate and preserve the original exception. The essential requirement is: > After a successful `getConnection()`, every failure path during executor > initialization must guarantee that the Hikari connection is released. ### Current master Although our runtime reproduction was performed on Doris 2.1.11, the issue is being reported against the current code path because the same resource ownership problem can still occur if statement initialization fails after a Hikari connection has already been borrowed. Current master uses HikariCP 6.0.0. HikariCP still has the alive-bypass optimization, so upgrading from the HikariCP version used by Doris 2.1.11 does not by itself eliminate this type of trigger. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
