Hello! It's hard to say outright. Can you provide full log before node crash? Is there a chance that you ran out of disk space? What's your WALMode?
Regards, -- Ilya Kasnacheev пт, 1 февр. 2019 г. в 08:16, radha jai <[email protected]>: > Hi, > Ignite has been deployed on k8s has 12 ignite-servers, which are spread > out one on each worker node. The limits are 1 CPU 32GB RAM, with maximum > of 8 CPU and 64GB. Each ignite-server has a WAL and Persistent storage > volume of 30GB. > Getting below error after inserting the 60GB of data to ignite cluster, one > of the nodes crashes, and never recovers. The error on startup indicates > that the WAL fails to restore memory state, > type=CRITICAL_ERROR, err=class o.a.i.i.pagemem.wal.StorageException: > Failed to restore memory state (checkpoint marker is present on disk, but > checkpoint record is missed in WAL) > > following warning message are seen in some of the server logs. > > [03:53:53,375][WARNING][jvm-pause-detector-worker][] Possible too long JVM > pause: 1022 milliseconds. > > > The snippet of ignite configuration is below: > > > <property name="peerClassLoadingEnabled" value="true"/> > > <property name="dataStorageConfiguration"> > > <bean > class="org.apache.ignite.configuration.DataStorageConfiguration"> > > <!-- Enable metrics for Ignite persistence --> > > <property name="metricsEnabled" value="true"/> > > <property name="defaultDataRegionConfiguration"> > > <bean > class="org.apache.ignite.configuration.DataRegionConfiguration"> > > > <property name="name" value="Default_Region"/> > > <property name="initialSize" value="#{32L * 1024 * 1024 > * 1024}"/> > > <property name="maxSize" value="#{64L * 1024 * 1024 * > 1024}"/> > > <!-- Enabling Apache Ignite Persistent Store. --> > > <property name="persistenceEnabled" value="true"/> > > <!-- Enable metrics for this data region --> > > <property name="metricsEnabled" value="true"/> > > </bean> > > </property> > > <property name="storagePath" value="/opt/ignite/persistence/"/> > > <property name="walPath" value="/opt/ignite/wal/"/> > > </bean> > > </property> > > > Ignite JVM configuration: -server -Xms1g -Xmx1g -XX:+AlwaysPreTouch > -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC > > > Thanks > > radha > > > >
