Andrey Khitrin created IGNITE-26969:
---------------------------------------
Summary: IgniteClient: OutOfMemoryError when working with JDCB
under C3P0 pool
Key: IGNITE-26969
URL: https://issues.apache.org/jira/browse/IGNITE-26969
Project: Ignite
Issue Type: Bug
Components: thin clients ai3
Affects Versions: 3.2
Reporter: Andrey Khitrin
h2. How to reproduce
Use a C3P0 connection pool to work with AI3 cluster.
{code:java}
ComboPooledDataSource c3p0Pool = new ComboPooledDataSource();
c3p0Pool.setJdbcUrl("jdbc:ignite:thin://" + thinClientEndpoint);
try (Connection connection = cluster().dataSource().getConnection()) {
...
{code}
After some amount of time, the following error may occur:
{code}
java.sql.SQLException: Connections could not be acquired from the underlying
database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at
org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.createTables(ClusterFailoverTestBase.java:193)
at
org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.getFilledTables(ClusterFailoverTestBase.java:182)
at
org.gridgain.ai3tests.tests.failover.ClusterFailover1NodeTest.singleKillAndRestartNodeWhenDataIsLoaded(ClusterFailover1NodeTest.java:60)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:787)
at
org.junit.platform.commons.support.ReflectionSupport.invokeMethod(ReflectionSupport.java:479)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.gridgain.ai3tests.tests.extensions.cluster.IgniteClusterByIgniteRunnerExtension.interceptTestTemplateMethod(IgniteClusterByIgniteRunnerExtension.java:146)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:112)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:94)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:112)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:94)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.api.AssertTimeoutPreemptively.lambda$submitTask$3(AssertTimeoutPreemptively.java:95)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A
ResourcePool could not acquire a resource from its primary factory or source.
at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 23 common frames omitted
Caused by: java.sql.SQLException: Failed to connect to server
at
org.apache.ignite.internal.jdbc.JdbcConnection.<init>(JdbcConnection.java:141)
at
org.apache.ignite.jdbc.IgniteJdbcDriver.connect(IgniteJdbcDriver.java:177)
at
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:161)
at
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:161)
at
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:147)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:202)
at
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
at
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
at
com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
at
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: org.apache.ignite.lang.IgniteException: Cannot reserve 131072 bytes
of direct buffer memory (allocated: 536847170, limit: 536870912)
at
org.apache.ignite.internal.util.ViewUtils.ensurePublicException(ViewUtils.java:72)
at org.apache.ignite.internal.util.ViewUtils.sync(ViewUtils.java:46)
at
org.apache.ignite.internal.jdbc.JdbcConnection.buildClient(JdbcConnection.java:189)
at
org.apache.ignite.internal.jdbc.JdbcConnection.<init>(JdbcConnection.java:139)
... 10 common frames omitted
Caused by: java.lang.OutOfMemoryError: Cannot reserve 131072 bytes of direct
buffer memory (allocated: 536847170, limit: 536870912)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
at io.netty.util.internal.CleanerJava9.allocate(CleanerJava9.java:86)
at
io.netty.util.internal.PlatformDependent.allocateDirect(PlatformDependent.java:600)
at
io.netty.buffer.UnpooledDirectByteBuf.allocateDirectBuffer(UnpooledDirectByteBuf.java:138)
at
io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:81)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:48)
at
io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:698)
at
io.netty.buffer.AdaptiveByteBufAllocator$DirectChunkAllocator.allocate(AdaptiveByteBufAllocator.java:115)
at
io.netty.buffer.AdaptivePoolingAllocator$SizeClassChunkController.newChunkAllocation(AdaptivePoolingAllocator.java:580)
at
io.netty.buffer.AdaptivePoolingAllocator$Magazine.allocate(AdaptivePoolingAllocator.java:983)
at
io.netty.buffer.AdaptivePoolingAllocator$Magazine.tryAllocate(AdaptivePoolingAllocator.java:854)
at
io.netty.buffer.AdaptivePoolingAllocator$MagazineGroup.allocate(AdaptivePoolingAllocator.java:396)
at
io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:261)
at
io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:248)
at
io.netty.buffer.AdaptiveByteBufAllocator.newDirectBuffer(AdaptiveByteBufAllocator.java:67)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:154)
at
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:88)
at
org.apache.ignite.internal.client.io.netty.NettyClientConnection.getBuffer(NettyClientConnection.java:86)
at
org.apache.ignite.internal.client.TcpClientChannel.handshakeReqAsync(TcpClientChannel.java:710)
at
org.apache.ignite.internal.client.TcpClientChannel.handshakeAsync(TcpClientChannel.java:678)
at
org.apache.ignite.internal.client.TcpClientChannel.lambda$initAsync$0(TcpClientChannel.java:202)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
at
org.apache.ignite.internal.client.io.netty.NettyClientConnectionMultiplexer.lambda$openAsync$1(NettyClientConnectionMultiplexer.java:177)
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604)
at
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:597)
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:573)
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:506)
at
io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:650)
at
io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:639)
at
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:119)
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:350)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:385)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:432)
at
io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)
at
io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)
at
io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:571)
at
io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:512)
at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)
at
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
{code}
h2. Context
It's highly likely that this issue may appear after IGNITE-26719. There are two
arguments for this:
1. It wasn't observed before 2025-10-31, and after this date it has started to
appear quite often. IGNITE-26719 was
[merged|https://github.com/apache/ignite-3/commit/dd091a9ab1a70cae3f9ab0d38cc5b90c7293e2bc]
at 2025-10-29.
2. A provided stack trace includes TcpClientChannel.handshakeAsync method which
was changed by the provided issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
