Mikhail Petrov created IGNITE-21399: ---------------------------------------
Summary: IgniteCache#getAll invoked on transactional cache can lead to a node failure with OutOfMemoryError Key: IGNITE-21399 URL: https://issues.apache.org/jira/browse/IGNITE-21399 Project: Ignite Issue Type: Bug Reporter: Mikhail Petrov Exception: {code:java} 2024-01-18 09:28:07.374 [ERROR][sys-#4371][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError]] java.lang.OutOfMemoryError: null at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:214) ~[?:?] at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:206) ~[?:?] at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:173) ~[?:?] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538) ~[?:?] at java.lang.StringBuilder.append(StringBuilder.java:178) ~[?:?] at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?] at java.util.concurrent.ConcurrentHashMap.toString(ConcurrentHashMap.java:1329) ~[?:?] at java.lang.String.valueOf(String.java:2951) ~[?:?] at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?] at org.apache.ignite.internal.processors.cache.GridCacheUtils$18.toString(GridCacheUtils.java:646) ~[ignite-core-14.1.2.jar:14.1.2] at java.lang.String.valueOf(String.java:2951) ~[?:?] at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:102) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:910) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1121) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1055) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:622) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:561) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.future.GridCompoundFuture.toString(GridCompoundFuture.java:421) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.future.GridCompoundIdentityFuture.toString(GridCompoundIdentityFuture.java:47) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.processors.cache.distributed.dht.CacheDistributedGetFutureAdapter.toString(CacheDistributedGetFutureAdapter.java:377) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedGetFuture.toString(GridPartitionedGetFuture.java:664) ~[ignite-core-14.1.2.jar:14.1.2] at java.lang.String.valueOf(String.java:2951) ~[?:?] at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpLongRunningOperations0(GridCachePartitionExchangeManager.java:2290) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpLongRunningOperations(GridCachePartitionExchangeManager.java:2439) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.lambda$null$0(IgniteTxManager.java:398) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7437) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) ~[ignite-core-14.1.2.jar:14.1.2] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) ~[ignite-core-14.1.2.jar:14.1.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2024-01-18 09:28:07.411 [WARN ][sys-#4371][org.apache.ignite.internal.processors.cache.CacheDiagnosticManager] Page locks dump: {code} Java version: OpenJDK Runtime Environment 11.0.20+8-LTS IgniteCache#getAll is called on a transactional cache with a huge number of keys (> 1000000). Near node creates a future for this operation and sends requests for cache entries to remote nodes. All received entries are accumulated in the near node reducer (CacheDistributedGetFutureAdapter#rdc). If the mentioned future is not completed for a long time, long running operation detector will dump all pending futures in a log. The result includes mentioned reducer and string representation of all received cache entries. The `toString` implementation in the reducer just invokes ConcurrentHashMap#toString which can result in the OutOfMemoryError if huge amount of keys are requested. The OutOfMemoryError in this case does not mean that no free heap space is available (see https://github.com/openjdk/jdk/blob/d7ca08a5cc64c8d3941493d98423a49b5bc1b922/src/java.base/share/classes/java/lang/AbstractStringBuilder.java#L269) -- This message was sent by Atlassian Jira (v8.20.10#820010)