Hi,
We are using Ignite 2.6 as persistent store in Partitioned Mode having 6
cluster node running, each node is running on different machine.
We have noticed that on all the server nodes were trying to rebalance due to
'too many dirty pages':
2018-12-22 14:56:17,161 INFO
[db-checkpoint-thread-#104%d66a2109-94b4-4eb3-bb3c-e611aa842a2a%] {}
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager
- Checkpoint started [
checkpointId=cabfb20d-d53d-4dd5-8e2b-d270f062c010, startPtr=FileWALPointer
[idx=5025, fileOff=65789181, len=5487], checkpointLockWait=34ms,
checkpointLockHoldTime=20ms, walCpRecordFsyncDuration=68ms, pages=1
820016, reason='too many dirty pages']
then I can see following log after a minute:
2018-12-22 14:57:26,040 WARN
[exchange-worker-#102%d66a2109-94b4-4eb3-bb3c-e611aa842a2a%] {}
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture
- Unable to await partitions release latch within timeout: ClientLatch
After this cluster became unresponsive for 30 minutes and timeout started
happening while writing to cluster at the client end.
But at the same time number of ignite related objects increased by many
threshold over the time.
Following is histogram from one of the client machine :
1: 675519770 16212474480
org.apache.ignite.internal.util.future.GridFutureAdapter$Node
2: 6637390 3837988712 [Ljava.lang.Object;
3: 6557471 262298840
org.apache.ignite.internal.util.future.GridCompoundFuture
4: 6627708 159064992 java.util.ArrayList
5: 177242 36609304 [B
And eventually client machine went in Full GC mode.
Following is the code to write in the ignite cache :
try(IgniteDataStreamer<BenchmarkCacheStoreKey, OptionalDouble> streamer =
ignite.dataStreamer(cacheName)){
igniteMap.forEach((key,value) -> streamer.addData(key, value));
}catch(CacheException|IgniteInterruptedException|IllegalStateException|IgniteDataStreamerTimeoutException
e){
ignite.log().error("Entries not written to Ignite Cache, please
check the logs.");
throw new IgniteException(e);
}
Any help will be much appreciated.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
