Oleg Valuyskiy created IGNITE-21770:
---------------------------------------
Summary: Improve logging for the ‘Probably disk is too busy,
please check your device’ event (when reading from disk)
Key: IGNITE-21770
URL: https://issues.apache.org/jira/browse/IGNITE-21770
Project: Ignite
Issue Type: Improvement
Reporter: Oleg Valuyskiy
Assignee: Oleg Valuyskiy
Make logging messages more specific and include more information that could be
necessary for analysis.
Stack trace example:
{code:java}
2024-03-15 04:23:25.395 [ERROR][ForkJoinPool.commonPool-worker-41][] Critical
system error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to read page
[file=/opt/ignite/ssd/data/cell_6_node_4/cacheGroup-part/part-15909.bin,
pageId=349803611487542]]]
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to read page
[file=/opt/ignite/ssd/data/cell_6_node_4/cacheGroup-part/part-15909.bin,
pageId=349803611487542]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:547)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:487)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.read(PageReadWriteManagerImpl.java:69)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:522)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:930)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:741)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:730)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:304)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:129)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:424)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:6168)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5917)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5971)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next(BPlusTree.java:6221)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:913)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:56)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.verify.IdleVerifyUtility.calculatePartitionHash(IdleVerifyUtility.java:269)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.verify.VerifyBackupPartitionsTaskV2$VerifyBackupPartitionsJobV2.lambda$calculatePartitionHashAsync$3(VerifyBackupPartitionsTaskV2.java:478)
~[ignite-core-14.1.2.jar:14.1.2]
at
java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1448)
~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
~[?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
~[?:?]
Caused by: java.io.IOException: Probably disk is too busy, please check your
device.
at
org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.fully(AbstractFileIO.java:66)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.readFully(AbstractFileIO.java:88)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.readWithFailover(FilePageStore.java:897)
~[ignite-core-14.1.2.jar:14.1.2]
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:512)
~[ignite-core-14.1.2.jar:14.1.2]
... 27 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
