Hi Ray, Thank you for your reply. In addition to checkpoint marker, setting page size to 4K (already default in newer versions of Ignite) and WAL history size to value “1” may help to reduce overhead and space used and make loading a little faster.
I apologize if you already mentioned this, but are there any trade dumps for the client and servers at the time of the supposed hangup? My estimate based on first test for index size for this case is around 80Gb (20Gb per node). Can be data of such volume fit fully in memory? How much cluster RAM available? Sincerely, Dmitriy Pavlov вт, 17 окт. 2017 г. в 17:09, Ray <[email protected]>: > I'm using ignite 2.1. > > The phenomenon I observed is that for the first 130M entries the speed is > OK, but after about 130M entries it slowed down tremendously and finally it > will stuck. > The problem is that when I ingest small amount of data like 20M, it works > OK > and the performance is acceptable. > But when the entries number grow, the issue will happen for sure. > Please try ingesting more data and see if you can reproduce this issue. > > My servers have HDD, and I tested the random write speed using this command > " fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=randwrite > -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 > -group_reporting -name=mytest > " > The result is > mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, > iodepth=1 > ... > mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, > iodepth=1 > fio-2.0.13 > Starting 30 threads > Jobs: 11 (f=11): [__wwww_ww__w_w___w____w______w] [5.0% done] [0K/3072K/0K > /s] [0 /192 /0 iops] [eta 05h:16m:40s] > mytest: (groupid=0, jobs=30): err= 0: pid=17321: Tue Oct 17 13:54:12 2017 > write: io=5835.2MB, bw=5969.5KB/s, iops=373 , runt=1000960msec > clat (usec): min=149 , max=2013.4K, avg=80349.31, stdev=53474.06 > lat (usec): min=150 , max=2013.4K, avg=80351.00, stdev=53474.13 > clat percentiles (msec): > | 1.00th=[ 4], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 48], > | 30.00th=[ 67], 40.00th=[ 78], 50.00th=[ 86], 60.00th=[ 92], > | 70.00th=[ 99], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 139], > | 99.00th=[ 180], 99.50th=[ 215], 99.90th=[ 424], 99.95th=[ 611], > | 99.99th=[ 2008] > bw (KB/s) : min= 7, max= 8440, per=3.36%, avg=200.43, stdev=72.14 > lat (usec) : 250=0.01%, 500=0.04%, 750=0.05%, 1000=0.01% > lat (msec) : 2=0.09%, 4=1.71%, 10=3.23%, 20=9.55%, 50=5.60% > lat (msec) : 100=50.94%, 250=28.38%, 500=0.33%, 750=0.02%, 1000=0.01% > lat (msec) : 2000=0.01%, >=2000=0.02% > cpu : usr=0.00%, sys=0.00%, ctx=331635, majf=0, > minf=18446744073700073665 > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, > >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, > >=64=0.0% > issued : total=r=0/w=373449/d=0, short=r=0/w=0/d=0 > > Run status group 0 (all jobs): > WRITE: io=5835.2MB, aggrb=5969KB/s, minb=5969KB/s, maxb=5969KB/s, > mint=1000960msec, maxt=1000960msec > > Disk stats (read/write): > sda: ios=166/373605, merge=0/108, ticks=203/30925452, in_queue=30935375, > util=100.00% > > > > The estimated field length for key+value is 300 bytes each entry. > So the total amount will be write to disc will be > (300+300)*1.3*550*1000*1000 bytes = 426GB > Because I have four ignite servers, so the total time will be 426 / 4 / > (5969 / 1000 / 1000) / 60 / 60 = 5 hours. > > So apparently the disk writing speed is not an issue, it took ignite more > 13.5 hours to ingest 250M entries. > > By the way, I launched six ignite clients on the same node ignite server > runs. > > > > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
