I had the same problem in this thread http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-td16247.html
Basically, I'm using ignite-spark's saveParis method(which is also a IgniteDataStreamer) to ingest 550 million entries of data into Ignite, and the ingesting speed got slow down after first few minutes and seems stuck for now with persistent store enabled. My setup is 4 nodes with 16GB heap size and 32GB off heap size. After initial analysis, I found that when the 32GB off heap space is used up, the writing is almost stuck. >From the document, my understanding is if I enable persistent store the old data will be swapped to disc. But the writing stuck. Here's the dstat result for about a minute when the ingesting is stuck. [7l----total-cpu-usage---- ------memory-usage----- -dsk/total- ---paging-- ----swap--- --filesystem- usr sys idl wai hiq siq| used buff cach free| read writ| in out | used free|files inodes 1 0 99 0 0 0|89.9G 3970M 157G 507G| 30k 967k| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 12M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 9548k| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 15M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 2 1 94 3 0 0|89.9G 3970M 157G 507G| 0 15M| 0 0 | 0 0 | 8176 453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 26M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 10M| 0 0 | 0 0 | 8176 453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 31M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 23M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 21M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 21M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 28M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0 0 | 8176 453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 29M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 23M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 62M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 42M| 0 0 | 0 0 | 8064 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 41M| 0 0 | 0 0 | 8064 453k 0 0 97 2 0 0|89.9G 3970M 157G 507G| 0 28M| 0 0 | 0 0 | 8064 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 36M| 0 0 | 0 0 | 8064 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 31M| 0 0 | 0 0 | 8064 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0 0 | 8064 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8064 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 28M| 0 0 | 0 0 | 8064 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 40M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 34M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 16M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 33M| 0 0 | 0 0 | 8176 453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 53M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 30M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 34M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 22M| 0 0 | 0 0 | 8176 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 20M| 0 0 | 0 0 | 8176 453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 0 17M| 0 0 | 0 0 | 8288 453k 2 1 94 3 0 0|89.9G 3970M 157G 507G| 0 18M| 0 0 | 0 0 | 8288 453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 0 41M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 19M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 32M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 49M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 19M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 25M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 24M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 27M| 0 0 | 0 0 | 8288 453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 21M| 0 0 | 0 0 | 8288 453k And the GC logs, please scroll down to see today's log. http://apache-ignite-users.70518.x6.nabble.com/file/t1346/ignitegc.current <http://apache-ignite-users.70518.x6.nabble.com/file/t1346/ignitegc.current> -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
