Hello! Can you please provide full log? It's hard to say what is going on here. Can you please also share your Ignite/cache configuration and describe your work load?
Regards, -- Ilya Kasnacheev пн, 25 февр. 2019 г. в 17:32, Dodong Juan <[email protected]>: > I did observe the same thing with ignite and was not able to resolve it at > all. > So I am very interested on a resolution for this too. > > > On Feb 25, 2019, at 9:30 AM, Antonio Conforti <[email protected]> > wrote: > > > > Hello, > > > > first of all thanks for your advice. > > > > Before reading your post I runned another session (at 15:45 about) of > test > > with the same rate before changing the frequency of checkpoint you > suggested > > but with data already loaded. > > > > > > I observed that performance degrades suddenly and not after a while > (about > > 45 minutes) as observed in my first session when the cache was empty. > > Below you can see the statistics: > > > > 2019-02-22 15:49:03.992 INFO 5271 --- [oint-thread-#67] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721, > > markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426], > > walSegmentsCleared=11, walSegmentsCovered=[460 - 469], > markDuration=387ms, > > pagesWrite=1149ms, fsync=227229ms, total=228765ms] > > 2019-02-22 15:53:25.681 INFO 5271 --- [oint-thread-#67] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943, > > markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426], > > walSegmentsCleared=14, walSegmentsCovered=[470 - 483], > markDuration=1316ms, > > pagesWrite=1043ms, fsync=259329ms, total=261688ms] > > 2019-02-22 15:56:07.878 INFO 5271 --- [oint-thread-#67] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514, > > markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426], > > walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms, > > pagesWrite=239ms, fsync=161070ms, total=162197ms] > > > > What looks strange in this second session is that I see a delay between > 228 > > and 260 sec to write 129721 / 178943 pages. > > I don't observe this behaviour when I start from scratch (with no data in > > cache): it usually takes about 45 minutes before observing this > performance > > degradation. > > Any clues on what can be the cause? > > > > > > I started today another test session with the suggested frequency > > (checkpointingFrequency=600000) and I also configured direct IO and set > the > > WAL mode to NONE as suggested in tuning performance but the result > observed > > is quite the same > > > > > > > > 2019-02-25 09:34:02.284 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, > markPos=FileWALPointer > > [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], > > markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms] > > 2019-02-25 10:04:04.082 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, > fsync=123ms, > > total=1827ms] > > 2019-02-25 10:14:27.275 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, > fsync=21ms, > > total=25013ms] > > 2019-02-25 10:24:55.492 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, > fsync=93ms, > > total=53230ms] > > 2019-02-25 10:35:22.512 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms, > > total=80241ms] > > 2019-02-25 10:45:34.524 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms, > > total=92245ms] > > 2019-02-25 10:55:43.139 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, > fsync=16ms, > > total=100854ms] > > 2019-02-25 11:06:04.437 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, > fsync=154ms, > > total=122147ms] > > 2019-02-25 11:16:08.333 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, > fsync=25ms, > > total=126037ms] > > 2019-02-25 11:18:53.622 INFO 1208 --- [oint-thread-#66] > > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > > [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277, > > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > > walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, > fsync=344ms, > > total=48239ms] > > > > > > Do you have any other suggestions considering also the behaviour > described > > above? > > > > Thanks, > > Antonio > > > > > > > > -- > > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > >
