[
https://issues.apache.org/jira/browse/IGNITE-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024440#comment-17024440
]
Igor Kamyshnikov edited comment on IGNITE-7913 at 1/27/20 3:43 PM:
-------------------------------------------------------------------
The same on Ignite 2.7.6. At least the way to disable the execution of the
logic of
org.apache.ignite.internal.IgniteDiagnosticPrepareContext.CompoundInfoClosure#apply
is required (or some throttling)
It does not seem that the configuration property 'IGNITE_DIAGNOSTIC_ENABLED'
disables all the diag tasks (e.g. TX problems)
The test case leading to the sys-# pool thread starvation:
1) take a cluster of several ignite server nodes (3 or 5) and backups=1
2) load it with a lot of concurrent put requests
3) kill one node and immediately restart it
There are chances that the *Latch release* message coming from the client will
be processed on the server *in several minutes* because the sys-# pool is busy
with diagnostic messages. Not receiving the *latch release* message in time
leads to failed SQL queries.
was (Author: kamyshnikov):
The same on Ignite 2.7.6. At least the way to disable the execution of the
logic of
org.apache.ignite.internal.IgniteDiagnosticPrepareContext.CompoundInfoClosure#apply
is required.
It does not seem that the configuration property 'IGNITE_DIAGNOSTIC_ENABLED'
disables all the diag tasks (e.g. TX problems)
The test case leading to the sys-# pool thread starvation:
1) take a cluster of several ignite server nodes (3 or 5) and backups=1
2) load it with a lot of concurrent put requests
3) kill one node and immediately restart it
There are chances that the *Latch release* message coming from the client will
be processed on the server *in several minutes* because the sys-# pool is busy
with diagnostic messages. Not receiving the *latch release* message in time
leads to failed SQL queries.
> Current implementation of Internal Diagnostics may cause OOM on server nodes.
> -----------------------------------------------------------------------------
>
> Key: IGNITE-7913
> URL: https://issues.apache.org/jira/browse/IGNITE-7913
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 2.3
> Reporter: Alexey Scherbakov
> Priority: Major
> Fix For: 2.9
>
>
> If many transactions are active in grid, Internal Diagnostics can cause OOM
> on server nodes serving IgniteDiagnosticMessage because of heap buffering.
> See the stack trace demonstrating the issue:
> {noformat}
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1012)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:762)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:710)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.toString(GridDhtCacheEntry.java:818)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:826)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:783)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxEntry.toString(IgniteTxEntry.java:1267)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at java.util.AbstractMap.toString(AbstractMap.java:559)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:864)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxRemoteStateImpl.toString(IgniteTxRemoteStateImpl.java:180)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:826)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:783)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.toString(GridDistributedTxRemoteAdapter.java:926)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxRemote.toString(GridDhtTxRemote.java:373)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:826)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:783)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter$TxFinishFuture.toString(IgniteTxAdapter.java:2405)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at java.util.AbstractCollection.toString(AbstractCollection.java:462)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.ignite.internal.processors.cache.CacheObjectsReleaseFuture.toString(CacheObjectsReleaseFuture.java:58)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at java.util.AbstractCollection.toString(AbstractCollection.java:462)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.ignite.internal.processors.cache.CacheObjectsReleaseFuture.toString(CacheObjectsReleaseFuture.java:58)
> at java.lang.String.valueOf(String.java:2994)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.SBLimitedLength.a(SBLimitedLength.java:88)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:939)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:1005)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:685)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:621)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.toString(GridDhtPartitionsExchangeFuture.java:3519)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.ignite.internal.IgniteDiagnosticMessage$ExchangeInfoClosure.apply(IgniteDiagnosticMessage.java:368)
> at
> org.apache.ignite.internal.IgniteDiagnosticMessage$ExchangeInfoClosure.apply(IgniteDiagnosticMessage.java:346)
> at
> org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.moreInfo(IgniteDiagnosticPrepareContext.java:220)
> at
> org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.apply(IgniteDiagnosticPrepareContext.java:202)
> at
> org.apache.ignite.internal.IgniteDiagnosticPrepareContext$CompoundInfoClosure.apply(IgniteDiagnosticPrepareContext.java:163)
> at
> org.apache.ignite.internal.processors.cluster.ClusterProcessor$2.onMessage(ClusterProcessor.java:175)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We must reconsider way to build string representation of transactions and use
> streaming-like approach.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
