Hi folks, We have been experiencing node failures with the error "Failed to write buffer." recently. Any ideas or optimizations not to get the error and node failure?
Thanks... [2019-08-22T01:20:55,916][ERROR][wal-write-worker%null-#221][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.StorageException: Failed to write buffer.]] org.apache.ignite.internal.processors.cache.persistence.StorageException: Failed to write buffer. at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.writeBuffer(FileWriteAheadLogManager.java:3484) [ignite-core-2.7.5.jar:2.7.5] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.body(FileWriteAheadLogManager.java:3301) [ignite-core-2.7.5.jar:2.7.5] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.5.jar:2.7.5] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[?:1.8.0_201] at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:253) ~[?:1.8.0_201] at org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.position(RandomAccessFileIO.java:48) ~[ignite-core-2.7.5.jar:2.7.5] at org.apache.ignite.internal.processors.cache.persistence.file.FileIODecorator.position(FileIODecorator.java:41) ~[ignite-core-2.7.5.jar:2.7.5] at org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.writeFully(AbstractFileIO.java:111) ~[ignite-core-2.7.5.jar:2.7.5] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.writeBuffer(FileWriteAheadLogManager.java:3477) ~[ignite-core-2.7.5.jar:2.7.5] ... 3 more [2019-08-22T01:20:55,921][WARN ][wal-write-worker%null-#221][FailureProcessor] No deadlocked threads detected. [2019-08-22T01:20:56,347][WARN ][wal-write-worker%null-#221][FailureProcessor] Thread dump at 2019/08/22 01:20:56 UTC *Ignite version*: 2.7.5 *Cluster size*: 16 *Client size*: 22 *Cluster OS version*: Centos 7 *Cluster Kernel version*: 4.4.185-1.el7.elrepo.x86_64 *Java version* : java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) Current disk sizes; Screen_Shot_2019-08-22_at_12.png <http://apache-ignite-users.70518.x6.nabble.com/file/t2515/Screen_Shot_2019-08-22_at_12.png> Ignite and gc logs; ignite-9.zip <http://apache-ignite-users.70518.x6.nabble.com/file/t2515/ignite-9.zip> Ignite configuration file; default-config.xml <http://apache-ignite-users.70518.x6.nabble.com/file/t2515/default-config.xml> ----- İbrahim Halil Altun Senior Software Engineer @ Segmentify -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/