checkpoint marker is present on disk, but checkpoint record is missed in WAL

[radha jai]

Hi,
   Ignite has been deployed on k8s has 12 ignite-servers, which are spread
out one on each worker node.  The limits are 1 CPU 32GB RAM, with maximum
of 8 CPU and 64GB.  Each ignite-server has a WAL and Persistent storage
volume of 30GB.
   Getting below error after inserting the 60GB of data to ignite cluster, one
of the nodes crashes, and never recovers.  The error on startup indicates
that the WAL fails to restore memory state,
   type=CRITICAL_ERROR, err=class o.a.i.i.pagemem.wal.StorageException:
Failed to restore memory state (checkpoint marker is present on disk, but
checkpoint record is missed in WAL)

following warning message are seen in some of the server logs.

[03:53:53,375][WARNING][jvm-pause-detector-worker][] Possible too long JVM
pause: 1022 milliseconds.


The snippet of ignite configuration is below:


<property name="peerClassLoadingEnabled" value="true"/>

 <property name="dataStorageConfiguration">

      <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">

          <!-- Enable metrics for Ignite persistence  -->

          <property name="metricsEnabled" value="true"/>

          <property name="defaultDataRegionConfiguration">

              <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">


                  <property name="name" value="Default_Region"/>

                  <property name="initialSize" value="#{32L * 1024 * 1024 *
1024}"/>

                  <property name="maxSize" value="#{64L * 1024 * 1024 *
1024}"/>

                  <!-- Enabling Apache Ignite Persistent Store. -->

                  <property name="persistenceEnabled" value="true"/>

                  <!-- Enable metrics for this data region  -->

                  <property name="metricsEnabled" value="true"/>

              </bean>

          </property>

          <property name="storagePath" value="/opt/ignite/persistence/"/>

          <property name="walPath" value="/opt/ignite/wal/"/>

      </bean>

  </property>


Ignite JVM configuration:  -server -Xms1g -Xmx1g -XX:+AlwaysPreTouch
-XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC


Thanks

radha