Hello,
Seems like checkpoint-runner thread is idle. There's no active
checkpoint at the moment of dump, and thus there's no work for
checkpoint thread.
If the problem persists, please share full thread dump. It would be
better to take 2-3 consecutive thread dumps when your data streamer stops.
Best Regards,
Ivan Rakov
On 19.02.2018 0:41, lawrencefinn wrote:
I am testing a very large durable cache that mostly cannot fit into memory.
I start loading in a lot of data via a data streamer. At some point the
data becomes too large to fit into memory so ignite starts writing a lot to
disk during checkpoints. But after some point after that, the datastreamer
stops streaming. It's stuck. Doing a jstack on ignite i see the
datastreamer threads are all stuck in a parked state. Any thoughts?
"data-streamer-stripe-4-#37" #62 prio=5 os_prio=0 tid=0x00007fe8c212d000
nid=0x1410 waiting on condition [0x00007fe73d1d6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
at java.lang.Thread.run(Thread.java:745)
The checkpoint thread seems to be parked too
"checkpoint-runner-#354" #397 prio=5 os_prio=0 tid=0x00007fe49c04f000
nid=0x177c waiting on condition [0x00007fdc54a1c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000002c08a6ef8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
My configuration is as follows:
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="publicThreadPoolSize" value="128"/>
<property name="systemThreadPoolSize" value="128"/>
<property name="dataStreamerThreadPoolSize" value="128"/>
<property name="deploymentMode" value="SHARED"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="storagePath" value="/per/persistence"/>
<property name="walPath" value="/wal"/>
<property name="walMode" value="BACKGROUND"/>
<property name="walFlushFrequency" value="10000"/>
<property name="checkpointThreads" value="128"/>
<property name="writeThrottlingEnabled" value="true"/>
<property name="walArchivePath" value="/wal/archive"/>
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="maxSize" value="#{100L * 1024 *
1024}"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="20G_Region_Eviction_P"/>
<property name="initialSize" value="#{10L * 1024 * 1024}"/>
<property name="maxSize" value="#{20 * 1024L * 1024 *
1024}"/>
<property name="persistenceEnabled" value="true"/>
<property name="pageEvictionMode" value="RANDOM_2_LRU"/>
<property name="checkpointPageBufferSize" value="#{2 * 1024L
* 1024 * 1024}"/>
</bean>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
