Ignite Load Hangs at the end while using DataStreamer

[pg31]

Hi

Cluster Configuration:
3 Nodes (112 GB Memory / 512 GB Disk)

Ignite Configuration:
1. Persistence Enabled
2. Version: 2.6.0
3. Configuration is as follows:
            <property name="authenticationEnabled" value="true"/>
            <property name="failureDetectionTimeout" value="30000"/>
            <property name="workDirectory" value="/persistence/work"/>          
  
            <property name="dataStorageConfiguration">
                <bean
class="org.apache.ignite.configuration.DataStorageConfiguration">               
     
                    <property name="defaultDataRegionConfiguration">
                        <bean
class="org.apache.ignite.configuration.DataRegionConfiguration">                
            
                            <property name="name" value="Default_Region"/>
                            <property name="persistenceEnabled"
value="true"/>                                          
                            <property name="maxSize" value="#{80 * 1024 *
1024 * 1024}"/>
                        </bean>
                    </property>
                    <property name="storagePath" value="/persistence"/>
                    <property name="walPath" value="/wal"/>
                    <property name="walArchivePath" value="/wal/archive"/>
                    <property name="walMode" value="LOG_ONLY"/>
                    <property name="walCompactionEnabled" value="true" />
                    <property name="walHistorySize" value="2" />
                </bean>
            </property>


Issue: Data Loading with Spark gets stuck at the end. 
Description:
I am trying to load 65M (million) rows into the Ignite cluster. Everything
runs well till 64.5 Million rows and then all of sudden, the data ingestion
just hangs. (I am able to ingest 64.5 Million rows in about 10 minutes)

There is enough memory which is still free on all the nodes (Approximately
80 GB of memory remains free on all the nodes). 

I am using the following code to ingest data into ignite. 

dataFrame
      .write.format(IgniteDataFrameSettings.FORMAT_IGNITE)
      .option(IgniteDataFrameSettings.OPTION_TABLE, tableName)
      .option(IgniteDataFrameSettings.OPTION_CONFIG_FILE, igniteConfig)
     
.option(IgniteDataFrameSettings.OPTION_CREATE_TABLE_PRIMARY_KEY_FIELDS,
primaryKey)
      .option("user", igniteUsername)
      .option("password", ignitePassword)
      .mode(SaveMode.Overwrite) //Overwriting entire table.
      .save()

I am not sure, but it looks like that the Ignite Thread is hanging
somewhere. 

This is what I can see in the thread dumps of all the executors. 


<http://apache-ignite-users.70518.x6.nabble.com/file/t2770/Spark_Thread_Stuck.png>
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/