Hello Abhinav, Have you followed all of our suggested tuning in this document?
http://docs.basho.com/riak/kv/2.1.4/using/performance/ Specifically, these settings may help a lot in your environment: http://docs.basho.com/riak/kv/2.1.4/using/performance/#optional-i-o-settings -- Luke Bakken Engineer lbak...@basho.com On Mon, Jun 13, 2016 at 12:00 AM, Abhinav Tripathi <abhi.160...@gmail.com> wrote: > Hi, > I am seeing quite slow get and put operations on our Riak KV cluster. We are > new to Riak and have tried to follow the instructions in the documentation. > It would be great if you could take a look and let us know what are we doing > wrong. > > We are using Riak as a distributed cache for our application. The 95th > percentile (and above) get operations are >10ms. Similarly, 99th and above > percentile for put operations is getting close to 10ms. Ideally, we want the > 99% gets to stay around 1ms and 99% puts to stay <5ms. > > I have attached the configurations for all 3 nodes in our cluster. Also, > attached are recent console.log files. I have attached a recent one-minute > stats file as well from which the below values are. > > GETs > "node_get_fsm_time_100": 71429, > "node_get_fsm_time_95": 16551, > "node_get_fsm_time_99": 27513, > "node_get_fsm_time_mean": 4393, > "node_get_fsm_time_median": 989 > > PUTs > "node_put_fsm_time_100": 10785, > "node_put_fsm_time_95": 2679, > "node_put_fsm_time_99": 7800, > "node_put_fsm_time_mean": 1752, > "node_put_fsm_time_median": 1608 > > I have recorded many one-minute stats for our cluster. Most of them for the > past couple of days show high values like these. > > All three Riak nodes are 4-core, 16GB RAM, 128 GB SSD machines ... basically > m3.xlarge AWS instances. > > Also, I can see many such lines in the console.log, > > 2016-06-13 00:03:32.545 [info] <0.23701.3269> Merged > {["/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/75.bitcask.data","/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/76.bitcask.data","/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/77.bitcask.data","/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/78.bitcask.data"],[]} > in 596.530484 seconds. > > These merge operations are usually taking close to 600 seconds. Could that > be a problem as well? > > Thanks, > Abhinav. > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
