Artem Zinnatullin created IGNITE-9952: -----------------------------------------
Summary: Failed to process write update request in FULL_ASYNC mode Key: IGNITE-9952 URL: https://issues.apache.org/jira/browse/IGNITE-9952 Project: Ignite Issue Type: Bug Affects Versions: 2.6 Environment: * Kubernetes * Official Ignite 2.6.0 Docker image Reporter: Artem Zinnatullin Cluster of 12 Ignite server nodes, 4 client nodes. Following cache configuration: {code:java} <property name="cacheConfiguration"> <bean class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="mycache"/> <property name="dataRegionName" value="Default_Region"/> <!-- Distribute cache across server nodes. --> <property name="cacheMode" value="PARTITIONED"/> <!-- Store a backup of each key/value pair, allows survive node downtime at cost of storing less data. --> <property name="backups" value="1"/> <!-- Do not wait for the end of cache write operation(s). --> <property name="writeSynchronizationMode" value="FULL_ASYNC"/> <!-- Throttle rebalancing messages by such pause (ms), otherwise it can over-consume CPU/memory. --> <property name="rebalanceThrottle" value="20"/> <!-- Size of rebalancing batch in bytes (default is 512 KB). --> <property name="rebalanceBatchSize" value="#{2L * 1024 * 1024}"/> </bean> </property> {code} Ignite is only used as key/value cache. Periodically these exceptions can be found in Ignite server nodes logs: {code:java} [23:17:23,586][INFO][grid-timeout-worker-#23][IgniteKernal] FreeList [name=null, buckets=256, dataPages=32070, reusePages=1435] [23:17:28,586][INFO][grid-timeout-worker-#23][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=37ea392d, uptime=51:52:58.760] ^-- H/N/C [hosts=16, nodes=16, CPUs=128] ^-- CPU [cur=0.4%, avg=0.44%, GC=0%] ^-- PageMemory [pages=2827437] ^-- Heap [used=1866MB, free=63.54%, comm=5120MB] ^-- Non heap [used=94MB, free=-1%, comm=95MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=2, qSize=0] ^-- System thread pool [active=0, idle=8, qSize=0] [23:17:28,586][INFO][grid-timeout-worker-#23][IgniteKernal] FreeList [name=null, buckets=256, dataPages=31943, reusePages=1435] [23:17:30,506][SEVERE][sys-stripe-5-#6][GridDhtAtomicCache] <mycache> Unexpected exception during cache update java.lang.NullPointerException [23:17:30,506][SEVERE][sys-stripe-5-#6][GridDhtAtomicCache] <mycache> Failed to process write update request in FULL_ASYNC mode for keys: [KeyCacheObjectImpl [part=637, val=b/cb65d27d2837d6864edb03cce62aa9c29c540669, hasValBytes=true]] class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node. at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1785) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1628) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3056) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:130) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:261) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511) at java.lang.Thread.run(Thread.java:748) Suppressed: java.lang.NullPointerException [23:17:33,588][INFO][grid-timeout-worker-#23][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=37ea392d, uptime=51:53:03.763] ^-- H/N/C [hosts=16, nodes=16, CPUs=128] ^-- CPU [cur=1.67%, avg=0.44%, GC=0%] ^-- PageMemory [pages=2827437] ^-- Heap [used=2123MB, free=58.52%, comm=5120MB] ^-- Non heap [used=94MB, free=-1%, comm=95MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=2, qSize=0] ^-- System thread pool [active=0, idle=8, qSize=0] [23:17:33,588][INFO][grid-timeout-worker-#23][IgniteKernal] FreeList [name=null, buckets=256, dataPages=32119, reusePages=1435] {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)