Interesting, so the Java client is just honoring the server's request that it try back in 60 seconds? Do you happen to know if there is a configuration variable to tune that default timeout value?
Our automated testing since yesterday has shown that changing pb_backlog did resolve this 60 second timeout issue for our system. -Matt On Wed, Mar 12, 2014 at 9:18 PM, Brian Roach <[email protected]> wrote: > Just as a clarification, there is no default timeout in the Java client. > Riak 1.4 introduced server-side timeouts on operations and the default is > 60 seconds (60k milliseconds). By default, the client does retry after > receiving the timeout error message which is the behavior you're seeing. > > - Roach > On Mar 12, 2014 7:42 AM, "Matthew MacClary" < > [email protected]> wrote: > >> Thanks for the suggestion Christian. Right now I am running an experiment >> with pb_backlog set to 64 up from the default of 5. The way our application >> uses the client there would usually be 4 connections initiated at the same >> time, but there could be as many as 12 connections initiated at the same >> time right now. I wonder if that is causing the client to time out and try >> back in 60 seconds! >> >> I will report my results. >> >> Best regards, >> >> -Matt >> >> >> On Tue, Mar 11, 2014 at 11:38 PM, Christian Dahlqvist < >> [email protected]> wrote: >> >>> Hi Matthew, >>> >>> I believe 60 seconds is the default timeout in the client, so it is >>> possible the `busy_dist_port` issues have caused a timeout and that the >>> automatic retry then has succeeded. >>> >>> A small +zdbbl value will cause the internal buffers to fill up and >>> result in `busy_dist_port` messages, which will cause performance problems. >>> I would recommend setting +zdbbl to 16384 (16MB) or 32768 (32MB) and verify >>> that you stop seeing busy_dist_post messages in the logs. If problems >>> persist it may be required to set it even higher. >>> >>> It is also important to note that `busy_dist_port` messages can be >>> caused by individual large objects even if +zdbbl is set to a reasonably >>> large value as outlined above. In Riak 1.4.8, logging of large objects has >>> been introduced, which will allow you to identify large objects that could >>> cause problems by going through the logs. You can also track large objects >>> by trending the `node_get_fsm_objsize_100` statistic. >>> >>> Best regards, >>> >>> Christian >>> >>> >>> >>> On Wed, Mar 12, 2014 at 5:43 AM, Matthew MacClary < >>> [email protected]> wrote: >>> >>>> Hi everyone, we are running Riak 1.4.1 on RHEL 6.2 using bitcask. We >>>> are using protobufs with the Java client, and our binary objects are >>>> typically a few hundred KB in size. I have noticed a persistent anomaly >>>> with riak reads and writes. It seems like often, maybe 0.5% of the time, >>>> writing to Riak takes 60 seconds longer than it should. Here is a prime >>>> example I just trimmed from a log file (see the last time entry below). >>>> >>>> This is not bitcask merge related because it happens before any of the >>>> bitcask slabs are large enough to merge. I am seeing lots of busy_dist_port >>>> messages in the Riak logs. One unique setting is that we have a small zdbbl >>>> setting of 128K because this seemed to prevent congestive collapse of >>>> throughput at high sustained loads. I believe that this 60 second timeout >>>> persisted across the various zdbbl settings we tried. Also we see this >>>> occasional 60 second delay on both VMs and real server hardware. >>>> >>>> Does anyone know where this 60 second delay comes from? >>>> >>>> Thanks! >>>> >>>> -Matt >>>> >>>> >>>> 2014-03-11 20:40:00,747 INFO [Thread-61] sally.ReportHandler - Riak >>>> load time for 88: 0.087 seconds >>>> 2014-03-11 20:40:01,137 INFO [Thread-62] sally.ReportHandler - Riak >>>> load time for 70: 0.185 seconds >>>> 2014-03-11 20:40:01,958 INFO [Thread-63] sally.ReportHandler - Riak >>>> load time for 97: 0.054 seconds >>>> 2014-03-11 20:40:02,566 INFO [Thread-64] sally.ReportHandler - Riak >>>> load time for 90: 0.043 seconds >>>> 2014-03-11 20:40:02,830 INFO [Thread-65] sally.ReportHandler - Riak >>>> load time for 85: 0.051 seconds >>>> 2014-03-11 20:40:04,162 INFO [Thread-66] sally.ReportHandler - Riak >>>> load time for 101: 0.075 seconds >>>> 2014-03-11 20:40:04,503 INFO [Thread-67] sally.ReportHandler - Riak >>>> load time for 103: 0.048 seconds >>>> 2014-03-11 20:40:05,745 INFO [Thread-68] sally.ReportHandler - Riak >>>> load time for 98: 0.031 seconds >>>> 2014-03-11 20:40:06,041 INFO [Thread-69] sally.ReportHandler - Riak >>>> load time for 102: 0.063 seconds >>>> 2014-03-11 20:40:06,444 INFO [Thread-70] sally.ReportHandler - Riak >>>> load time for 92: 0.022 seconds >>>> 2014-03-11 20:40:06,903 INFO [Thread-71] sally.ReportHandler - Riak >>>> load time for 99: 0.039 seconds >>>> 2014-03-11 20:40:09,847 INFO [Thread-72] sally.ReportHandler - Riak >>>> load time for 106: 0.019 seconds >>>> 2014-03-11 20:40:10,107 INFO [Thread-73] sally.ReportHandler - Riak >>>> load time for 108: 0.043 seconds >>>> 2014-03-11 20:40:47,820 INFO [Thread-52] sally.ReportHandler - Riak >>>> load time for 62: 1 minutes, 0.190 seconds >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >>
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
