Hi Ray, Do you see "Page evictions started, this will affect storage performance" message in the log? If so, dramatic performance drop you observe might indicate that we have an issue with page replacement algorithm that we need to investigate. Can you please check the message?
2017-10-17 17:09 GMT+03:00 Ray <[email protected]>: > I'm using ignite 2.1. > > The phenomenon I observed is that for the first 130M entries the speed is > OK, but after about 130M entries it slowed down tremendously and finally it > will stuck. > The problem is that when I ingest small amount of data like 20M, it works > OK > and the performance is acceptable. > But when the entries number grow, the issue will happen for sure. > Please try ingesting more data and see if you can reproduce this issue. > > My servers have HDD, and I tested the random write speed using this command > " fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=randwrite > -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 > -group_reporting -name=mytest > " > The result is > mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, > iodepth=1 > ... > mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, > iodepth=1 > fio-2.0.13 > Starting 30 threads > Jobs: 11 (f=11): [__wwww_ww__w_w___w____w______w] [5.0% done] [0K/3072K/0K > /s] [0 /192 /0 iops] [eta 05h:16m:40s] > mytest: (groupid=0, jobs=30): err= 0: pid=17321: Tue Oct 17 13:54:12 2017 > write: io=5835.2MB, bw=5969.5KB/s, iops=373 , runt=1000960msec > clat (usec): min=149 , max=2013.4K, avg=80349.31, stdev=53474.06 > lat (usec): min=150 , max=2013.4K, avg=80351.00, stdev=53474.13 > clat percentiles (msec): > | 1.00th=[ 4], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 48], > | 30.00th=[ 67], 40.00th=[ 78], 50.00th=[ 86], 60.00th=[ 92], > | 70.00th=[ 99], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 139], > | 99.00th=[ 180], 99.50th=[ 215], 99.90th=[ 424], 99.95th=[ 611], > | 99.99th=[ 2008] > bw (KB/s) : min= 7, max= 8440, per=3.36%, avg=200.43, stdev=72.14 > lat (usec) : 250=0.01%, 500=0.04%, 750=0.05%, 1000=0.01% > lat (msec) : 2=0.09%, 4=1.71%, 10=3.23%, 20=9.55%, 50=5.60% > lat (msec) : 100=50.94%, 250=28.38%, 500=0.33%, 750=0.02%, 1000=0.01% > lat (msec) : 2000=0.01%, >=2000=0.02% > cpu : usr=0.00%, sys=0.00%, ctx=331635, majf=0, > minf=18446744073700073665 > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > issued : total=r=0/w=373449/d=0, short=r=0/w=0/d=0 > > Run status group 0 (all jobs): > WRITE: io=5835.2MB, aggrb=5969KB/s, minb=5969KB/s, maxb=5969KB/s, > mint=1000960msec, maxt=1000960msec > > Disk stats (read/write): > sda: ios=166/373605, merge=0/108, ticks=203/30925452, in_queue=30935375, > util=100.00% > > > > The estimated field length for key+value is 300 bytes each entry. > So the total amount will be write to disc will be > (300+300)*1.3*550*1000*1000 bytes = 426GB > Because I have four ignite servers, so the total time will be 426 / 4 / > (5969 / 1000 / 1000) / 60 / 60 = 5 hours. > > So apparently the disk writing speed is not an issue, it took ignite more > 13.5 hours to ingest 250M entries. > > By the way, I launched six ignite clients on the same node ignite server > runs. > > > > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
