unsubscribe On Wed, 21 Jun, 2023, 06:23 MJ, <6733...@qq.com> wrote:
> Hello, > > Please can anyone advise below Log overflow & Threads hanged warning . > My system is running in openshift4 platform, ignite 2.13.0. Observed some > "Log overflow" and "Threads hanged" warning logs. It's not happening > frequently but a few times every day. > The only activities around that time were lots of invoking addData by two > IgniteDataStreamer . And both are having below setting and so the addData > did not trigger cache's writethrough . > Can those warnings cause breaks? any idea what the root cause is, and how > to fix it ? > > stmr.allowOverwrite(true); > stmr.skipStore(true); > stmr.autoFlushFrequency(1000L); > > -------------------some logging > [ignite-poc-datanode-131-z5x4r] 23-06-20 10:43:42.221 [INFO ] > grid-timeout-worker-#22%poc% o.a.i.i.IgniteKernal%poc:117 - > [ignite-poc-datanode-131-z5x4r] Data storage metrics for local node (to > disable set 'metricsLogFrequency' to 0) > [ignite-poc-datanode-131-z5x4r] ^-- Off-heap memory [used=1239MB, > free=78.63%, allocated=1619MB] > [ignite-poc-datanode-131-z5x4r] ^-- Page memory [pages=315385] > [ignite-poc-datanode-131-6ph7w] 23-06-20 10:44:02.312 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Threads > hanged: [(query-#2324%poc%-2369, TIMED_WAITING)] > [ignite-poc-datanode-131-6ph7w] 23-06-20 10:44:02.320 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Page > locks dump: > [ignite-poc-datanode-131-6ph7w] Log overflow, size:512, headIdx=512 > [structureId=50, pageIdpageId=844420635199783 [pageIdHex=0002ffff00008927, > partId=65535, pageIdx=35111, flags=00000002]] > [ignite-poc-datanode-131-6ph7w] Thread=[name=query-#2324%poc%, id=2369], > state=TIMED_WAITING > [ignite-poc-datanode-131-6ph7w] Locked pages = > [844420635199783[0002ffff00008927](r=1|w=0),844420635191247[0002ffff000067cf](r=1|w=0)] > [ignite-poc-datanode-131-6ph7w] Locked pages log: name=query-#2324%poc% > time=(1687272172791, 2023-06-20 10:42:52.791) > [ignite-poc-datanode-131-6ph7w] L=1 -> Read lock pageId=844420635164763, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, > partId=65535, pageIdx=91, flags=00000002] > [ignite-poc-datanode-131-6ph7w] L=2 -> Read lock pageId=844420635195566, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff000078ae, > partId=65535, pageIdx=30894, flags=00000002] > [ignite-poc-datanode-131-6ph7w] L=1 <- Read unlock pageId=844420635164763, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, > partId=65535, pageIdx=91, flags=00000002] > [ignite-poc-datanode-131-6ph7w] L=2 -> Read lock pageId=844420635192482, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff00006ca2, > partId=65535, pageIdx=27810, flags=00000002] > [ignite-poc-datanode-131-6ph7w] L=1 <- Read unlock pageId=844420635195566, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff000078ae, > partId=65535, pageIdx=30894, flags=00000002] > .......... > [ignite-poc-datanode-131-6ph7w] 23-06-20 10:44:02.323 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Failed > to save locks dump file. > [ignite-poc-datanode-131-6ph7w] org.apache.ignite.IgniteCheckedException: > Work directory does not exist and cannot be created: /ignite/work > [ignite-poc-datanode-131-6ph7w] at > org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:9900) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-6ph7w] at > org.apache.ignite.internal.util.IgniteUtils.defaultWorkDirectory(IgniteUtils.java:9840) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-6ph7w] at > org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.PageLockTrackerManager.onHangThreads(PageLockTrackerManager.java:153) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-6ph7w] at > org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.iteration(SharedPageLockTracker.java:340) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-6ph7w] at > org.apache.ignite.internal.util.worker.CycleThread.run(CycleThread.java:49) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-x4fzx] 23-06-20 10:44:05.190 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Threads > hanged: [(query-#2270%poc%-2315, TIMED_WAITING)] > [ignite-poc-datanode-131-x4fzx] 23-06-20 10:44:05.198 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Page > locks dump: > [ignite-poc-datanode-131-x4fzx] Log overflow, size:512, headIdx=512 > [structureId=50, pageIdpageId=844420635196176 [pageIdHex=0002ffff00007b10, > partId=65535, pageIdx=31504, flags=00000002]] > [ignite-poc-datanode-131-x4fzx] Thread=[name=query-#2270%poc%, id=2315], > state=TIMED_WAITING > [ignite-poc-datanode-131-x4fzx] Locked pages = > [844420635196176[0002ffff00007b10](r=1|w=0),844420635199249[0002ffff00008711](r=1|w=0)] > [ignite-poc-datanode-131-x4fzx] Locked pages log: name=query-#2270%poc% > time=(1687272172791, 2023-06-20 10:42:52.791) > [ignite-poc-datanode-131-x4fzx] L=1 -> Read lock pageId=844420635164763, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, > partId=65535, pageIdx=91, flags=00000002] > [ignite-poc-datanode-131-x4fzx] L=2 -> Read lock pageId=844420635197184, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff00007f00, > partId=65535, pageIdx=32512, flags=00000002] > [ignite-poc-datanode-131-x4fzx] L=1 <- Read unlock pageId=844420635164763, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff0000005b, > partId=65535, pageIdx=91, flags=00000002] > .......... > [ignite-poc-datanode-131-x4fzx] L=2 -> Read lock pageId=844420635199249, > structureId=1560569965__key_PK##H2Tree [pageIdHex=0002ffff00008711, > partId=65535, pageIdx=34577, flags=00000002] > [ignite-poc-datanode-131-x4fzx] 23-06-20 10:44:05.201 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Failed > to save locks dump file. > [ignite-poc-datanode-131-x4fzx] org.apache.ignite.IgniteCheckedException: > Work directory does not exist and cannot be created: /ignite/work > [ignite-poc-datanode-131-x4fzx] at > org.apache.ignite.internal.util.IgniteUtils.workDirectory(IgniteUtils.java:9900) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-x4fzx] at > org.apache.ignite.internal.util.IgniteUtils.defaultWorkDirectory(IgniteUtils.java:9840) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-x4fzx] at > org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.PageLockTrackerManager.onHangThreads(PageLockTrackerManager.java:153) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-x4fzx] at > org.apache.ignite.internal.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.iteration(SharedPageLockTracker.java:340) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-x4fzx] at > org.apache.ignite.internal.util.worker.CycleThread.run(CycleThread.java:49) > ~[ignite-core-2.13.0.jar:2.13.0] > [ignite-poc-datanode-131-z5x4r] 23-06-20 10:44:18.623 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Threads > hanged: [(query-#2331%poc%-2376, TIMED_WAITING)] > [ignite-poc-datanode-131-z5x4r] 23-06-20 10:44:18.632 [WARN ] > page-lock-tracker-timeout o.a.i.i.p.c.CacheDiagnosticManager:127 - Page > locks dump: > [ignite-poc-datanode-131-z5x4r] Log overflow, size:512, headIdx=512 > [structureId=50, pageIdpageId=844420635194620 [pageIdHex=0002ffff000074fc, > partId=65535, pageIdx=29948, flags=00000002]] > [ignite-poc-datanode-131-z5x4r] Thread=[name=query-#2331%poc%, id=2376], > state=TIMED_WAITING > [ignite-poc-datanode-131-z5x4r] Locked pages = > [844420635194620[0002ffff000074fc](r=1|w=0),844420635201360[0002ffff00008f50](r=1|w=0)] > > > > > Thanks, > MJ >
