Hello, first of all thanks for your advice.
Before reading your post I runned another session (at 15:45 about) of test with the same rate before changing the frequency of checkpoint you suggested but with data already loaded. I observed that performance degrades suddenly and not after a while (about 45 minutes) as observed in my first session when the cache was empty. Below you can see the statistics: 2019-02-22 15:49:03.992 INFO 5271 --- [oint-thread-#67] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721, markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426], walSegmentsCleared=11, walSegmentsCovered=[460 - 469], markDuration=387ms, pagesWrite=1149ms, fsync=227229ms, total=228765ms] 2019-02-22 15:53:25.681 INFO 5271 --- [oint-thread-#67] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943, markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426], walSegmentsCleared=14, walSegmentsCovered=[470 - 483], markDuration=1316ms, pagesWrite=1043ms, fsync=259329ms, total=261688ms] 2019-02-22 15:56:07.878 INFO 5271 --- [oint-thread-#67] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514, markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426], walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms, pagesWrite=239ms, fsync=161070ms, total=162197ms] What looks strange in this second session is that I see a delay between 228 and 260 sec to write 129721 / 178943 pages. I don't observe this behaviour when I start from scratch (with no data in cache): it usually takes about 45 minutes before observing this performance degradation. Any clues on what can be the cause? I started today another test session with the suggested frequency (checkpointingFrequency=600000) and I also configured direct IO and set the WAL mode to NONE as suggested in tuning performance but the result observed is quite the same 2019-02-25 09:34:02.284 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms] 2019-02-25 10:04:04.082 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, fsync=123ms, total=1827ms] 2019-02-25 10:14:27.275 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, fsync=21ms, total=25013ms] 2019-02-25 10:24:55.492 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, fsync=93ms, total=53230ms] 2019-02-25 10:35:22.512 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms, total=80241ms] 2019-02-25 10:45:34.524 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms, total=92245ms] 2019-02-25 10:55:43.139 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, fsync=16ms, total=100854ms] 2019-02-25 11:06:04.437 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, fsync=154ms, total=122147ms] 2019-02-25 11:16:08.333 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, fsync=25ms, total=126037ms] 2019-02-25 11:18:53.622 INFO 1208 --- [oint-thread-#66] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277, markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, fsync=344ms, total=48239ms] Do you have any other suggestions considering also the behaviour described above? Thanks, Antonio -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
