Hello,
Please can anyone advise below Log overflow & Threads hanged warning . My system is running in openshift4 platform, ignite 2.13.0. Observed some "Log overflow" and "Threads hanged" warning logs. It's not happening frequently but a few times every day. The only activities around that time were lots of invoking addData by two IgniteDataStreamer . And both are having below setting and so the addData did not trigger cache's writethrough . Can those warnings cause breaks? any idea what the root cause is, and how to fix it ? stmr.allowOverwrite(true); stmr.skipStore(true); stmr.autoFlushFrequency(1000L); -------------------some logging [ignite-poc-datanode-131-z5x4r] 23-06-20 10:43:42.221 [INFO ] grid-timeout-worker-#22%poc% o.a.i.i.IgniteKernal%poc:117 - [ignite-poc-datanode-131-z5x4r] Data storage metrics for local node (to disable set 'metricsLogFrequency' to 0) [ignite-poc-datanode-131-z5x4r] ^-- Off-heap memory [used=1239MB, free=78.63%, allocated=1619MB] [ignite-poc-datanode-131-z5x4r] ^-- Page memory [pages=315385] [ignite-poc-datanode-131-6ph7w] 23-06-20 10:44:02.312 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Threads hanged: [(query-#2324%poc%-2369, TIMED_WAITING)] [ignite-poc-datanode-131-6ph7w] 23-06-20 10:44:02.320 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Page locks dump: [ignite-poc-datanode-131-6ph7w] Log overflow, size:512, headIdx=512 [structureId=50, pageIdpageId=844420635199783 [pageIdHex=0002ffff00008927, partId=65535, pageIdx=35111, flags=00000002]] [ignite-poc-datanode-131-6ph7w] Thread=[name=query-#2324%poc%, id=2369], state=TIMED_WAITING [ignite-poc-datanode-131-6ph7w] Locked pages = [844420635199783[0002ffff00008927](r=1|w=0),844420635191247[0002ffff000067cf](r=1|w=0)] [ignite-poc-datanode-131-6ph7w] Locked pages log: name=query-#2324%poc% time=(1687272172791, 2023-06-20 10:42:52.791) [ignite-poc-datanode-131-6ph7w] L=1 -> Read lock pageId=844420635164763, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, partId=65535, pageIdx=91, flags=00000002] [ignite-poc-datanode-131-6ph7w] L=2 -> Read lock pageId=844420635195566, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff000078ae, partId=65535, pageIdx=30894, flags=00000002] [ignite-poc-datanode-131-6ph7w] L=1 <- Read unlock pageId=844420635164763, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, partId=65535, pageIdx=91, flags=00000002] [ignite-poc-datanode-131-6ph7w] L=2 -> Read lock pageId=844420635192482, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff00006ca2, partId=65535, pageIdx=27810, flags=00000002] [ignite-poc-datanode-131-6ph7w] L=1 <- Read unlock pageId=844420635195566, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff000078ae, partId=65535, pageIdx=30894, flags=00000002] .......... [ignite-poc-datanode-131-6ph7w] 23-06-20 10:44:02.323 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Failed to save locks dump file. [ignite-poc-datanode-131-6ph7w] org.apache.ignite.IgniteCheckedException: Work directory does not exist and cannot be created: /ignite/work [ignite-poc-datanode-131-6ph7w] at org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:9900) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-6ph7w] at org.apache.ignite.internal.util.IgniteUtils.defaultWorkDirectory(IgniteUtils.java:9840) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-6ph7w] at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.PageLockTrackerManager.onHangThreads(PageLockTrackerManager.java:153) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-6ph7w] at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.iteration(SharedPageLockTracker.java:340) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-6ph7w] at org.apache.ignite.internal.util.worker.CycleThread.run(CycleThread.java:49) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-x4fzx] 23-06-20 10:44:05.190 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Threads hanged: [(query-#2270%poc%-2315, TIMED_WAITING)] [ignite-poc-datanode-131-x4fzx] 23-06-20 10:44:05.198 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Page locks dump: [ignite-poc-datanode-131-x4fzx] Log overflow, size:512, headIdx=512 [structureId=50, pageIdpageId=844420635196176 [pageIdHex=0002ffff00007b10, partId=65535, pageIdx=31504, flags=00000002]] [ignite-poc-datanode-131-x4fzx] Thread=[name=query-#2270%poc%, id=2315], state=TIMED_WAITING [ignite-poc-datanode-131-x4fzx] Locked pages = [844420635196176[0002ffff00007b10](r=1|w=0),844420635199249[0002ffff00008711](r=1|w=0)] [ignite-poc-datanode-131-x4fzx] Locked pages log: name=query-#2270%poc% time=(1687272172791, 2023-06-20 10:42:52.791) [ignite-poc-datanode-131-x4fzx] L=1 -> Read lock pageId=844420635164763, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, partId=65535, pageIdx=91, flags=00000002] [ignite-poc-datanode-131-x4fzx] L=2 -> Read lock pageId=844420635197184, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff00007f00, partId=65535, pageIdx=32512, flags=00000002] [ignite-poc-datanode-131-x4fzx] L=1 <- Read unlock pageId=844420635164763, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, partId=65535, pageIdx=91, flags=00000002] .......... [ignite-poc-datanode-131-x4fzx] L=2 -> Read lock pageId=844420635199249, structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff00008711, partId=65535, pageIdx=34577, flags=00000002] [ignite-poc-datanode-131-x4fzx] 23-06-20 10:44:05.201 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Failed to save locks dump file. [ignite-poc-datanode-131-x4fzx] org.apache.ignite.IgniteCheckedException: Work directory does not exist and cannot be created: /ignite/work [ignite-poc-datanode-131-x4fzx] at org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:9900) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-x4fzx] at org.apache.ignite.internal.util.IgniteUtils.defaultWorkDirectory(IgniteUtils.java:9840) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-x4fzx] at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.PageLockTrackerManager.onHangThreads(PageLockTrackerManager.java:153) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-x4fzx] at org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.iteration(SharedPageLockTracker.java:340) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-x4fzx] at org.apache.ignite.internal.util.worker.CycleThread.run(CycleThread.java:49) ~[ignite-core-2.13.0.jar:2.13.0] [ignite-poc-datanode-131-z5x4r] 23-06-20 10:44:18.623 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Threads hanged: [(query-#2331%poc%-2376, TIMED_WAITING)] [ignite-poc-datanode-131-z5x4r] 23-06-20 10:44:18.632 [WARN ] page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Page locks dump: [ignite-poc-datanode-131-z5x4r] Log overflow, size:512, headIdx=512 [structureId=50, pageIdpageId=844420635194620 [pageIdHex=0002ffff000074fc, partId=65535, pageIdx=29948, flags=00000002]] [ignite-poc-datanode-131-z5x4r] Thread=[name=query-#2331%poc%, id=2376], state=TIMED_WAITING [ignite-poc-datanode-131-z5x4r] Locked pages = [844420635194620[0002ffff000074fc](r=1|w=0),844420635201360[0002ffff00008f50](r=1|w=0)] Thanks, MJ
