Hi everyone, we are running Riak 1.4.1 on RHEL 6.2 using bitcask. We are using protobufs with the Java client, and our binary objects are typically a few hundred KB in size. I have noticed a persistent anomaly with riak reads and writes. It seems like often, maybe 0.5% of the time, writing to Riak takes 60 seconds longer than it should. Here is a prime example I just trimmed from a log file (see the last time entry below).
This is not bitcask merge related because it happens before any of the bitcask slabs are large enough to merge. I am seeing lots of busy_dist_port messages in the Riak logs. One unique setting is that we have a small zdbbl setting of 128K because this seemed to prevent congestive collapse of throughput at high sustained loads. I believe that this 60 second timeout persisted across the various zdbbl settings we tried. Also we see this occasional 60 second delay on both VMs and real server hardware. Does anyone know where this 60 second delay comes from? Thanks! -Matt 2014-03-11 20:40:00,747 INFO [Thread-61] sally.ReportHandler - Riak load time for 88: 0.087 seconds 2014-03-11 20:40:01,137 INFO [Thread-62] sally.ReportHandler - Riak load time for 70: 0.185 seconds 2014-03-11 20:40:01,958 INFO [Thread-63] sally.ReportHandler - Riak load time for 97: 0.054 seconds 2014-03-11 20:40:02,566 INFO [Thread-64] sally.ReportHandler - Riak load time for 90: 0.043 seconds 2014-03-11 20:40:02,830 INFO [Thread-65] sally.ReportHandler - Riak load time for 85: 0.051 seconds 2014-03-11 20:40:04,162 INFO [Thread-66] sally.ReportHandler - Riak load time for 101: 0.075 seconds 2014-03-11 20:40:04,503 INFO [Thread-67] sally.ReportHandler - Riak load time for 103: 0.048 seconds 2014-03-11 20:40:05,745 INFO [Thread-68] sally.ReportHandler - Riak load time for 98: 0.031 seconds 2014-03-11 20:40:06,041 INFO [Thread-69] sally.ReportHandler - Riak load time for 102: 0.063 seconds 2014-03-11 20:40:06,444 INFO [Thread-70] sally.ReportHandler - Riak load time for 92: 0.022 seconds 2014-03-11 20:40:06,903 INFO [Thread-71] sally.ReportHandler - Riak load time for 99: 0.039 seconds 2014-03-11 20:40:09,847 INFO [Thread-72] sally.ReportHandler - Riak load time for 106: 0.019 seconds 2014-03-11 20:40:10,107 INFO [Thread-73] sally.ReportHandler - Riak load time for 108: 0.043 seconds 2014-03-11 20:40:47,820 INFO [Thread-52] sally.ReportHandler - Riak load time for 62: 1 minutes, 0.190 seconds
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
