Ivan, Alex Goncharuk, The exception trace is not helpful, it's not obvious what might be a reason and how to address it. How do we tackle these problems?
Ibrahim, please attach all the log files for a detailed look. - Denis On Thu, Aug 22, 2019 at 3:08 AM ihalilaltun <ibrahim.al...@segmentify.com> wrote: > Hi folks, > > We have been experiencing node failures with the error "Failed to write > buffer." recently. Any ideas or optimizations not to get the error and node > failure? > > Thanks... > > [2019-08-22T01:20:55,916][ERROR][wal-write-worker%null-#221][] Critical > system error detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, > SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext > [type=CRITICAL_ERROR, err=class > o.a.i.i.processors.cache.persistence.StorageException: Failed to write > buffer.]] > org.apache.ignite.internal.processors.cache.persistence.StorageException: > Failed to write buffer. > at > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.writeBuffer(FileWriteAheadLogManager.java:3484) > [ignite-core-2.7.5.jar:2.7.5] > at > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.body(FileWriteAheadLogManager.java:3301) > [ignite-core-2.7.5.jar:2.7.5] > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.7.5.jar:2.7.5] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] > Caused by: java.nio.channels.ClosedChannelException > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_201] > at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:253) > ~[?:1.8.0_201] > at > > org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.position(RandomAccessFileIO.java:48) > ~[ignite-core-2.7.5.jar:2.7.5] > at > > org.apache.ignite.internal.processors.cache.persistence.file.FileIODecorator.position(FileIODecorator.java:41) > ~[ignite-core-2.7.5.jar:2.7.5] > at > > org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.writeFully(AbstractFileIO.java:111) > ~[ignite-core-2.7.5.jar:2.7.5] > at > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.writeBuffer(FileWriteAheadLogManager.java:3477) > ~[ignite-core-2.7.5.jar:2.7.5] > ... 3 more > [2019-08-22T01:20:55,921][WARN > ][wal-write-worker%null-#221][FailureProcessor] No deadlocked threads > detected. > [2019-08-22T01:20:56,347][WARN > ][wal-write-worker%null-#221][FailureProcessor] Thread dump at 2019/08/22 > 01:20:56 UTC > > > *Ignite version*: 2.7.5 > *Cluster size*: 16 > *Client size*: 22 > *Cluster OS version*: Centos 7 > *Cluster Kernel version*: 4.4.185-1.el7.elrepo.x86_64 > *Java version* : > java version "1.8.0_201" > Java(TM) SE Runtime Environment (build 1.8.0_201-b09) > Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) > > Current disk sizes; > Screen_Shot_2019-08-22_at_12.png > < > http://apache-ignite-users.70518.x6.nabble.com/file/t2515/Screen_Shot_2019-08-22_at_12.png> > > Ignite and gc logs; > ignite-9.zip > <http://apache-ignite-users.70518.x6.nabble.com/file/t2515/ignite-9.zip> > Ignite configuration file; > default-config.xml > < > http://apache-ignite-users.70518.x6.nabble.com/file/t2515/default-config.xml> > > > > > ----- > İbrahim Halil Altun > Senior Software Engineer @ Segmentify > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
