Mikhail Petrov created IGNITE-21399:
---------------------------------------
Summary: IgniteCache#getAll invoked on transactional cache can
lead to a node failure with OutOfMemoryError
Key: IGNITE-21399
URL: https://issues.apache.org/jira/browse/IGNITE-21399
Project: Ignite
Issue Type: Bug
Reporter: Mikhail Petrov
Exception:
{code:java}
2024-01-18 09:28:07.374 [ERROR][sys-#4371][] Critical system error detected.
Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.OutOfMemoryError]]
java.lang.OutOfMemoryError: null
at
java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:214)
~[?:?]
at
java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:206)
~[?:?]
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:173)
~[?:?]
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538) ~[?:?]
at java.lang.StringBuilder.append(StringBuilder.java:178) ~[?:?]
at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?]
at
java.util.concurrent.ConcurrentHashMap.toString(ConcurrentHashMap.java:1329)
~[?:?]
at java.lang.String.valueOf(String.java:2951) ~[?:?]
at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils$18.toString(GridCacheUtils.java:646)
~[ignite-core-14.1.2.jar:14.1.2]
at java.lang.String.valueOf(String.java:2951) ~[?:?]
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:102)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:910)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1121)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1055)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:622)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:561)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.future.GridCompoundFuture.toString(GridCompoundFuture.java:421)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.future.GridCompoundIdentityFuture.toString(GridCompoundIdentityFuture.java:47)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.distributed.dht.CacheDistributedGetFutureAdapter.toString(CacheDistributedGetFutureAdapter.java:377)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedGetFuture.toString(GridPartitionedGetFuture.java:664)
~[ignite-core-14.1.2.jar:14.1.2]
at java.lang.String.valueOf(String.java:2951) ~[?:?]
at java.lang.StringBuilder.append(StringBuilder.java:172) ~[?:?]
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpLongRunningOperations0(GridCachePartitionExchangeManager.java:2290)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpLongRunningOperations(GridCachePartitionExchangeManager.java:2439)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.lambda$null$0(IgniteTxManager.java:398)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7437)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
~[ignite-core-14.1.2.jar:14.1.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-01-18 09:28:07.411 [WARN
][sys-#4371][org.apache.ignite.internal.processors.cache.CacheDiagnosticManager]
Page locks dump:
{code}
Java version: OpenJDK Runtime Environment 11.0.20+8-LTS
IgniteCache#getAll is called on a transactional cache with a huge number of
keys (> 1000000). Near node creates a future for this operation and sends
requests for cache entries to remote nodes. All received entries are
accumulated in the near node reducer (CacheDistributedGetFutureAdapter#rdc). If
the mentioned future is not completed for a long time, long running operation
detector will dump all pending futures in a log. The result includes mentioned
reducer and string representation of all received cache entries.
The `toString` implementation in the reducer just invokes
ConcurrentHashMap#toString which can result in the OutOfMemoryError if huge
amount of keys are requested. The OutOfMemoryError in this case does not mean
that no free heap space is available (see
https://github.com/openjdk/jdk/blob/d7ca08a5cc64c8d3941493d98423a49b5bc1b922/src/java.base/share/classes/java/lang/AbstractStringBuilder.java#L269)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
