sorry for misprint. I meant thread dumps, of course вт, 17 окт. 2017 г. в 18:16, Dmitry Pavlov <[email protected]>:
> Hi Ray, > > Thank you for your reply. In addition to checkpoint marker, setting page > size to 4K (already default in newer versions of Ignite) and WAL history > size to value “1” may help to reduce overhead and space used and make > loading a little faster. > > I apologize if you already mentioned this, but are there any trade dumps > for the client and servers at the time of the supposed hangup? > > My estimate based on first test for index size for this case is around > 80Gb (20Gb per node). Can be data of such volume fit fully in memory? How > much cluster RAM available? > > Sincerely, > Dmitriy Pavlov > > > вт, 17 окт. 2017 г. в 17:09, Ray <[email protected]>: > >> I'm using ignite 2.1. >> >> The phenomenon I observed is that for the first 130M entries the speed is >> OK, but after about 130M entries it slowed down tremendously and finally >> it >> will stuck. >> The problem is that when I ingest small amount of data like 20M, it works >> OK >> and the performance is acceptable. >> But when the entries number grow, the issue will happen for sure. >> Please try ingesting more data and see if you can reproduce this issue. >> >> My servers have HDD, and I tested the random write speed using this >> command >> " fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=randwrite >> -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 >> -group_reporting -name=mytest >> " >> The result is >> mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, >> iodepth=1 >> ... >> mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, >> iodepth=1 >> fio-2.0.13 >> Starting 30 threads >> Jobs: 11 (f=11): [__wwww_ww__w_w___w____w______w] [5.0% done] [0K/3072K/0K >> /s] [0 /192 /0 iops] [eta 05h:16m:40s] >> mytest: (groupid=0, jobs=30): err= 0: pid=17321: Tue Oct 17 13:54:12 2017 >> write: io=5835.2MB, bw=5969.5KB/s, iops=373 , runt=1000960msec >> clat (usec): min=149 , max=2013.4K, avg=80349.31, stdev=53474.06 >> lat (usec): min=150 , max=2013.4K, avg=80351.00, stdev=53474.13 >> clat percentiles (msec): >> | 1.00th=[ 4], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 48], >> | 30.00th=[ 67], 40.00th=[ 78], 50.00th=[ 86], 60.00th=[ 92], >> | 70.00th=[ 99], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 139], >> | 99.00th=[ 180], 99.50th=[ 215], 99.90th=[ 424], 99.95th=[ 611], >> | 99.99th=[ 2008] >> bw (KB/s) : min= 7, max= 8440, per=3.36%, avg=200.43, stdev=72.14 >> lat (usec) : 250=0.01%, 500=0.04%, 750=0.05%, 1000=0.01% >> lat (msec) : 2=0.09%, 4=1.71%, 10=3.23%, 20=9.55%, 50=5.60% >> lat (msec) : 100=50.94%, 250=28.38%, 500=0.33%, 750=0.02%, 1000=0.01% >> lat (msec) : 2000=0.01%, >=2000=0.02% >> cpu : usr=0.00%, sys=0.00%, ctx=331635, majf=0, >> minf=18446744073700073665 >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >> >=64=0.0% >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >> >=64=0.0% >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >> >=64=0.0% >> issued : total=r=0/w=373449/d=0, short=r=0/w=0/d=0 >> >> Run status group 0 (all jobs): >> WRITE: io=5835.2MB, aggrb=5969KB/s, minb=5969KB/s, maxb=5969KB/s, >> mint=1000960msec, maxt=1000960msec >> >> Disk stats (read/write): >> sda: ios=166/373605, merge=0/108, ticks=203/30925452, in_queue=30935375, >> util=100.00% >> >> >> >> The estimated field length for key+value is 300 bytes each entry. >> So the total amount will be write to disc will be >> (300+300)*1.3*550*1000*1000 bytes = 426GB >> Because I have four ignite servers, so the total time will be 426 / 4 / >> (5969 / 1000 / 1000) / 60 / 60 = 5 hours. >> >> So apparently the disk writing speed is not an issue, it took ignite more >> 13.5 hours to ingest 250M entries. >> >> By the way, I launched six ignite clients on the same node ignite server >> runs. >> >> >> >> >> >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
