Hi all - Why data streamer take lot of time randomly after 500+ million
writes, it frequently and very consistently takes lot of time to finish the
writes and to the extent of 25 to 45 seconds for write. May be its flushing
the data as i have flush frequency set but why not in the beginning and why
only in the end. And also I see heap size going up after sometime and trend
is consistently upwards.
Here is the streamer configuration:
dataStreamer.autoFlushFrequency(10000);
dataStreamer.perNodeBufferSize(32 * 1024);
dataStreamer.perNodeParallelOperations(32);
Not sure if this is of any use but here is the dataStorageConfiguration
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property
name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property
name="initialSize" value="#{512L * 1024 * 1024}"></property>
<property
name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
<property
name="persistenceEnabled" value="true" />
<property
name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
</bean>
</property>
<property name="pageSize"
value="#{4 * 1024}" />
<property
name="storagePath" value="${grid.data}" />
<property name="walPath"
value="${grid.wal}" />
<property
name="walArchivePath" value="${grid.wal}/archive" />
<property name="walMode"
value="BACKGROUND" />
<property
name="walFlushFrequency" value="5000"></property>
</bean>
</property>
