I encountered a problem when restarting ignite, a node seems to be unable to
process the request anymore, because its CacheStore did not process the
update operations of database. Here is the log it printed:
2018-12-12 07:06:27:556 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:06:37:557 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:06:47:558 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:06:57:559 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:07:05:177 [db-checkpoint-thread-#47] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Skipping checkpoint (no
pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=0ms,
reason='timeout']
2018-12-12 07:07:05:565 [grid-timeout-worker-#23] INFO
o.a.i.i.IgniteKernal:478 -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=14135687, uptime=00:09:00.052]
^-- H/N/C [hosts=13, nodes=13, CPUs=44]
^-- CPU [cur=0.2%, avg=37.69%, GC=0%]
^-- PageMemory [pages=324003]
^-- Heap [used=1876MB, free=38.91%, comm=3072MB]
^-- Non heap [used=70MB, free=-1%, comm=72MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=8, qSize=0]
2018-12-12 07:07:05:565 [grid-timeout-worker-#23] INFO
o.a.i.i.IgniteKernal:478 - FreeList [name=null, buckets=256,
dataPages=204429, reusePages=0]
2018-12-12 07:07:07:559 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:07:17:560 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:07:27:561 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:07:37:562 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:07:47:563 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:07:57:563 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
2018-12-12 07:08:05:574 [grid-timeout-worker-#23] INFO
o.a.i.i.IgniteKernal:478 -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=14135687, uptime=00:10:00.054]
^-- H/N/C [hosts=13, nodes=13, CPUs=44]
^-- CPU [cur=0.17%, avg=36.59%, GC=0%]
^-- PageMemory [pages=324003]
^-- Heap [used=1878MB, free=38.85%, comm=3072MB]
^-- Non heap [used=70MB, free=-1%, comm=72MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=8, qSize=0]
2018-12-12 07:08:05:575 [grid-timeout-worker-#23] INFO
o.a.i.i.IgniteKernal:478 - FreeList [name=null, buckets=256,
dataPages=204429, reusePages=0]
2018-12-12 07:08:07:564 [exchange-worker-#42] WARN
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:488 - Unable to await
partitions release latch within timeout: ClientLatch
[coordinator=ZookeeperClusterNode [id=6b9fc6e4-5b6a-4a98-be4d-6bc1aa5c014c,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3,
loc=false, client=false], ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=45, minorTopVer=1]]]
In addition, other nodes also printed some logs. Please review the
attachment.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
