Hello! I would start with tuning thread pool sizes if I see CPU under-loaded:
https://apacheignite.readme.io/docs/thread-pools You will need to do thread dump mid-benchmark to see which thread pools are full. Regards, -- Ilya Kasnacheev чт, 22 нояб. 2018 г. в 20:28, summasumma <[email protected]>: > Hi all, > > I am trying to benchmark ignite with ycsb tool as follows: > > Ignite Cluster: > ---------------- > 3 ignite-nodes each running in 1 VM (3 seperate VMs in cluster) > > VM details: > -------------- > CPU: 4 V-CPU (Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz) > RAM: 16 Gb > SSD: 40 GB > Network: 1 Gb/s > > Apache Ignite Parameters > ------------------------------ > Version: Apache Ignite 2.6 > Persistence: Disabled (pure in-memory key-value data store). > Caches mode: Partitioned / ATOMIC. > Backups: 1 > Write synchronization mode: PRIMARY_SYNC. > Max_Size/Initial_Size: 12 GB / 12 GB > > YCSB Workload Configuration: > --------------------------------- > recordcount=6000000 ----------------> 6 million records each of > size 1kb (total size:; 5.7 GB) > operationcount=6000000 > workload=com.yahoo.ycsb.workloads.CoreWorkload > readallfields=true > insertproportion=1 ------------------> 100% only INSERT > operation > readproportion=0 > updateproportion=0 > scanproportion=0 > requestdistribution=zipfian > threadcount=100 -------------------> 100 threads > target=75000 -------------------> 75k Operations Per > Second (OPS) > > Benchmark Results: > ---------------------- > [OVERALL], RunTime(ms), 64877 > [OVERALL], Throughput(ops/sec), 46241.34901428858 ---------------> > 46k OPS******* > [TOTAL_GCS_PS_Scavenge], Count, 87 > [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 162 > [TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.2497032846771583 > [TOTAL_GCS_PS_MarkSweep], Count, 2 > [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 66 > [TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.10173096783143488 > [TOTAL_GCs], Count, 89 > [TOTAL_GC_TIME], Time(ms), 228 > [TOTAL_GC_TIME_%], Time(%), 0.35143425250859317 > [CLEANUP], Operations, 100 > [CLEANUP], AverageLatency(us), 359.8 > [CLEANUP], MinLatency(us), 0 > [CLEANUP], MaxLatency(us), 35775 > [CLEANUP], 95thPercentileLatency(us), 6 > [CLEANUP], 99thPercentileLatency(us), 11 > [INSERT], Operations, 3000000 > [INSERT], AverageLatency(us), 2089.05358 -----------------> 2 MilliSeconds > Latency > [INSERT], MinLatency(us), 328 > [INSERT], MaxLatency(us), 64927 > [INSERT], 95thPercentileLatency(us), 3969 > [INSERT], 99thPercentileLatency(us), 6835 > > As you can see from the benchmark results that: > Expected OPS: 75k OPS > Actual OPS result: only 46k OPS > > During this run, i have captured the system resource details as shown > below: > CPU (Ignite / YCSB ) : 40% / 35% > Mem(Ignite / YCSB ) : 6.5 GB / 1.1 GB > Network(Ignite/YCSB in Mbps): 700rx-350tx / 30rx-500tx > > My question is: > 1. There is no system resource limit hit in both ignite and ycsb, So why is > that the actual result is not going above 46k OPS ? what is stopping here? > I > even tried increasing threads in ycsb but not getting any improvement even > with that. > > 2. On ignite machine, in the top command i can see below ignite app showing > > 100%: > ==CPU% MEM% VIRT RES PID USER TIME+ THR NI S R/s > W/s > Command > ==141.8 37.1 16.6G 5.76G 19099 root 10:39 86 0 S 0 > 0 > /usr/bin/java -Xms1g -Xmx1g > > But the overall cpu display(not individual process cpu as shown above) at > the top command was never crossed 50% (Load factor is 2 during this time). > Since its not crossed 50% i assume CPU still has juice to take more ignite > ops. Is this correct? > > 3. Is the latency displayed in ycsb output above(2 millisecond) be still > optimized? Again network/cpu is not exhausted. So it should be possible? Is > there a ignite-latency measurement tool (similar to redis?) available? > > 4. Any other parameters in the system is blocking to this limit? like tcp > open connections/tcp buffer size etc) ? > > Any help highly appreciated. > Please clarify. > > Thanks > summa > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
