Thanks for the suggestion Christian. Right now I am running an experiment with pb_backlog set to 64 up from the default of 5. The way our application uses the client there would usually be 4 connections initiated at the same time, but there could be as many as 12 connections initiated at the same time right now. I wonder if that is causing the client to time out and try back in 60 seconds!
I will report my results. Best regards, -Matt On Tue, Mar 11, 2014 at 11:38 PM, Christian Dahlqvist <[email protected]>wrote: > Hi Matthew, > > I believe 60 seconds is the default timeout in the client, so it is > possible the `busy_dist_port` issues have caused a timeout and that the > automatic retry then has succeeded. > > A small +zdbbl value will cause the internal buffers to fill up and result > in `busy_dist_port` messages, which will cause performance problems. I > would recommend setting +zdbbl to 16384 (16MB) or 32768 (32MB) and verify > that you stop seeing busy_dist_post messages in the logs. If problems > persist it may be required to set it even higher. > > It is also important to note that `busy_dist_port` messages can be caused > by individual large objects even if +zdbbl is set to a reasonably large > value as outlined above. In Riak 1.4.8, logging of large objects has been > introduced, which will allow you to identify large objects that could cause > problems by going through the logs. You can also track large objects by > trending the `node_get_fsm_objsize_100` statistic. > > Best regards, > > Christian > > > > On Wed, Mar 12, 2014 at 5:43 AM, Matthew MacClary < > [email protected]> wrote: > >> Hi everyone, we are running Riak 1.4.1 on RHEL 6.2 using bitcask. We are >> using protobufs with the Java client, and our binary objects are typically >> a few hundred KB in size. I have noticed a persistent anomaly with riak >> reads and writes. It seems like often, maybe 0.5% of the time, writing to >> Riak takes 60 seconds longer than it should. Here is a prime example I just >> trimmed from a log file (see the last time entry below). >> >> This is not bitcask merge related because it happens before any of the >> bitcask slabs are large enough to merge. I am seeing lots of busy_dist_port >> messages in the Riak logs. One unique setting is that we have a small zdbbl >> setting of 128K because this seemed to prevent congestive collapse of >> throughput at high sustained loads. I believe that this 60 second timeout >> persisted across the various zdbbl settings we tried. Also we see this >> occasional 60 second delay on both VMs and real server hardware. >> >> Does anyone know where this 60 second delay comes from? >> >> Thanks! >> >> -Matt >> >> >> 2014-03-11 20:40:00,747 INFO [Thread-61] sally.ReportHandler - Riak load >> time for 88: 0.087 seconds >> 2014-03-11 20:40:01,137 INFO [Thread-62] sally.ReportHandler - Riak load >> time for 70: 0.185 seconds >> 2014-03-11 20:40:01,958 INFO [Thread-63] sally.ReportHandler - Riak load >> time for 97: 0.054 seconds >> 2014-03-11 20:40:02,566 INFO [Thread-64] sally.ReportHandler - Riak load >> time for 90: 0.043 seconds >> 2014-03-11 20:40:02,830 INFO [Thread-65] sally.ReportHandler - Riak load >> time for 85: 0.051 seconds >> 2014-03-11 20:40:04,162 INFO [Thread-66] sally.ReportHandler - Riak load >> time for 101: 0.075 seconds >> 2014-03-11 20:40:04,503 INFO [Thread-67] sally.ReportHandler - Riak load >> time for 103: 0.048 seconds >> 2014-03-11 20:40:05,745 INFO [Thread-68] sally.ReportHandler - Riak load >> time for 98: 0.031 seconds >> 2014-03-11 20:40:06,041 INFO [Thread-69] sally.ReportHandler - Riak load >> time for 102: 0.063 seconds >> 2014-03-11 20:40:06,444 INFO [Thread-70] sally.ReportHandler - Riak load >> time for 92: 0.022 seconds >> 2014-03-11 20:40:06,903 INFO [Thread-71] sally.ReportHandler - Riak load >> time for 99: 0.039 seconds >> 2014-03-11 20:40:09,847 INFO [Thread-72] sally.ReportHandler - Riak load >> time for 106: 0.019 seconds >> 2014-03-11 20:40:10,107 INFO [Thread-73] sally.ReportHandler - Riak load >> time for 108: 0.043 seconds >> 2014-03-11 20:40:47,820 INFO [Thread-52] sally.ReportHandler - Riak load >> time for 62: 1 minutes, 0.190 seconds >> >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
