Ignite Node crashed in middle of checkpoint and data loss

[garima.j]

Hello,

We have an ignite cluster of 3 nodes (16GB RAM, 50GB disk space each)and
have given 10GB (off-heap) to data region and (Xms) 2GB and (Xmx) 3GB to the
nodes. 

One node went down and while restarting the node, I get the exception that
Ignite node crashed in the middle of checkpoint and JVM crash after that. 

Ignite configuration : 
 <property name="transactionConfiguration">
            <bean
class="org.apache.ignite.configuration.TransactionConfiguration">
                
                <property name="TxTimeoutOnPartitionMapExchange"
value="20000"/>
                
            </bean>
        </property>
        
        <property name="publicThreadPoolSize" value="64"/>
        <property name="systemThreadPoolSize" value="32"/>

Data Storage configuration : 
<property name="dataStorageConfiguration">
            <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="defaultDataRegionConfiguration">
                    <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="metricsEnabled" value="true"/>
                        <property name="initialSize" value="#{4L * 1024 *
1024 * 1024}"/>
                        <property name="maxSize" value="#{10L * 1024 * 1024
* 1024}"/>
                        <property name="checkpointPageBufferSize"
value="#{1L * 1024 * 1024 * 1024}"/>
                        <property name="pageEvictionMode"
value="RANDOM_2_LRU"/>
                    </bean>
                </property>
                <property name="storagePath" value="/data1/data/datastore"/>
                <property name="walPath" value="/data2/data/wal"/>
                <property name="walArchivePath"
value="/data2/data/wal/archive"/>
                <property name="walMode" value="LOG_ONLY"/>
                <property name="pageSize" value="#{4 * 1024}"/>
                <property name="writeThrottlingEnabled" value="true"/>
                <property name="walHistorySize" value="1"/>
                <property name="metricsEnabled" value="true"/>
                <property name="checkpointFrequency" value="60000"/>
            </bean>
        </property>

Cache configuration : 
<bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="XXX"/>
                    <property name="sqlSchema" value="PUBLIC"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="queryParallelism" value="8"/>
                    <property name="backups" value="1"/>
                    <property name="rebalanceThrottle" value="100"/>
                    <property name="rebalanceThreadPoolSize" value="4"/>
                    <property name="atomicityMode"
value="TRANSACTIONAL_SNAPSHOT"/>

Please find the logs(FINE level) and JVM crash logs.

hs_err_pid6456.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2241/hs_err_pid6456.log>  
ignite-8393e373.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2241/ignite-8393e373.log> 
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/