Re: 60 second timeout in Riak
Hi Matthew, I believe 60 seconds is the default timeout in the client, so it is possible the `busy_dist_port` issues have caused a timeout and that the automatic retry then has succeeded. A small +zdbbl value will cause the internal buffers to fill up and result in `busy_dist_port` messages, which will cause performance problems. I would recommend setting +zdbbl to 16384 (16MB) or 32768 (32MB) and verify that you stop seeing busy_dist_post messages in the logs. If problems persist it may be required to set it even higher. It is also important to note that `busy_dist_port` messages can be caused by individual large objects even if +zdbbl is set to a reasonably large value as outlined above. In Riak 1.4.8, logging of large objects has been introduced, which will allow you to identify large objects that could cause problems by going through the logs. You can also track large objects by trending the `node_get_fsm_objsize_100` statistic. Best regards, Christian On Wed, Mar 12, 2014 at 5:43 AM, Matthew MacClary < maccl...@lifetime.oregonstate.edu> wrote: > Hi everyone, we are running Riak 1.4.1 on RHEL 6.2 using bitcask. We are > using protobufs with the Java client, and our binary objects are typically > a few hundred KB in size. I have noticed a persistent anomaly with riak > reads and writes. It seems like often, maybe 0.5% of the time, writing to > Riak takes 60 seconds longer than it should. Here is a prime example I just > trimmed from a log file (see the last time entry below). > > This is not bitcask merge related because it happens before any of the > bitcask slabs are large enough to merge. I am seeing lots of busy_dist_port > messages in the Riak logs. One unique setting is that we have a small zdbbl > setting of 128K because this seemed to prevent congestive collapse of > throughput at high sustained loads. I believe that this 60 second timeout > persisted across the various zdbbl settings we tried. Also we see this > occasional 60 second delay on both VMs and real server hardware. > > Does anyone know where this 60 second delay comes from? > > Thanks! > > -Matt > > > 2014-03-11 20:40:00,747 INFO [Thread-61] sally.ReportHandler - Riak load > time for 88: 0.087 seconds > 2014-03-11 20:40:01,137 INFO [Thread-62] sally.ReportHandler - Riak load > time for 70: 0.185 seconds > 2014-03-11 20:40:01,958 INFO [Thread-63] sally.ReportHandler - Riak load > time for 97: 0.054 seconds > 2014-03-11 20:40:02,566 INFO [Thread-64] sally.ReportHandler - Riak load > time for 90: 0.043 seconds > 2014-03-11 20:40:02,830 INFO [Thread-65] sally.ReportHandler - Riak load > time for 85: 0.051 seconds > 2014-03-11 20:40:04,162 INFO [Thread-66] sally.ReportHandler - Riak load > time for 101: 0.075 seconds > 2014-03-11 20:40:04,503 INFO [Thread-67] sally.ReportHandler - Riak load > time for 103: 0.048 seconds > 2014-03-11 20:40:05,745 INFO [Thread-68] sally.ReportHandler - Riak load > time for 98: 0.031 seconds > 2014-03-11 20:40:06,041 INFO [Thread-69] sally.ReportHandler - Riak load > time for 102: 0.063 seconds > 2014-03-11 20:40:06,444 INFO [Thread-70] sally.ReportHandler - Riak load > time for 92: 0.022 seconds > 2014-03-11 20:40:06,903 INFO [Thread-71] sally.ReportHandler - Riak load > time for 99: 0.039 seconds > 2014-03-11 20:40:09,847 INFO [Thread-72] sally.ReportHandler - Riak load > time for 106: 0.019 seconds > 2014-03-11 20:40:10,107 INFO [Thread-73] sally.ReportHandler - Riak load > time for 108: 0.043 seconds > 2014-03-11 20:40:47,820 INFO [Thread-52] sally.ReportHandler - Riak load > time for 62: 1 minutes, 0.190 seconds > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
60 second timeout in Riak
Hi everyone, we are running Riak 1.4.1 on RHEL 6.2 using bitcask. We are using protobufs with the Java client, and our binary objects are typically a few hundred KB in size. I have noticed a persistent anomaly with riak reads and writes. It seems like often, maybe 0.5% of the time, writing to Riak takes 60 seconds longer than it should. Here is a prime example I just trimmed from a log file (see the last time entry below). This is not bitcask merge related because it happens before any of the bitcask slabs are large enough to merge. I am seeing lots of busy_dist_port messages in the Riak logs. One unique setting is that we have a small zdbbl setting of 128K because this seemed to prevent congestive collapse of throughput at high sustained loads. I believe that this 60 second timeout persisted across the various zdbbl settings we tried. Also we see this occasional 60 second delay on both VMs and real server hardware. Does anyone know where this 60 second delay comes from? Thanks! -Matt 2014-03-11 20:40:00,747 INFO [Thread-61] sally.ReportHandler - Riak load time for 88: 0.087 seconds 2014-03-11 20:40:01,137 INFO [Thread-62] sally.ReportHandler - Riak load time for 70: 0.185 seconds 2014-03-11 20:40:01,958 INFO [Thread-63] sally.ReportHandler - Riak load time for 97: 0.054 seconds 2014-03-11 20:40:02,566 INFO [Thread-64] sally.ReportHandler - Riak load time for 90: 0.043 seconds 2014-03-11 20:40:02,830 INFO [Thread-65] sally.ReportHandler - Riak load time for 85: 0.051 seconds 2014-03-11 20:40:04,162 INFO [Thread-66] sally.ReportHandler - Riak load time for 101: 0.075 seconds 2014-03-11 20:40:04,503 INFO [Thread-67] sally.ReportHandler - Riak load time for 103: 0.048 seconds 2014-03-11 20:40:05,745 INFO [Thread-68] sally.ReportHandler - Riak load time for 98: 0.031 seconds 2014-03-11 20:40:06,041 INFO [Thread-69] sally.ReportHandler - Riak load time for 102: 0.063 seconds 2014-03-11 20:40:06,444 INFO [Thread-70] sally.ReportHandler - Riak load time for 92: 0.022 seconds 2014-03-11 20:40:06,903 INFO [Thread-71] sally.ReportHandler - Riak load time for 99: 0.039 seconds 2014-03-11 20:40:09,847 INFO [Thread-72] sally.ReportHandler - Riak load time for 106: 0.019 seconds 2014-03-11 20:40:10,107 INFO [Thread-73] sally.ReportHandler - Riak load time for 108: 0.043 seconds 2014-03-11 20:40:47,820 INFO [Thread-52] sally.ReportHandler - Riak load time for 62: 1 minutes, 0.190 seconds ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to disassociate a bucket with a yokozuna index?
On Tue, Mar 11, 2014 at 4:17 AM, EmiNarcissus wrote: > > Now I'm working with riak 2.0 pre17, have tried both set bucket property > search_index to other index or _dont_index_, but still cannot delete the > index. > > > Failure: riakasaurus.exceptions.RiakPBCException: Can't delete index with > associate buckets [<<"riakasaurus.tests.test_search">>] (0) > What are the bucket properties for that bucket? I have a hunch of what it might be but need to see the properties to verify. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[ANN] Kria, an async Clojure driver for Riak 2
Kria (a right rotation of "Riak") is an open source asynchronous Clojure driver for Riak 2.0 built on top of Java 7's NIO.2. It uses Riak's protocol buffer interface. https://github.com/bluemont/kria https://clojars.org/kria In my work projects, we have found that Clojure's core.async works great as a layer on top of Kria. Just create a core.async channel in advance and have the callback put the desired return value in the core.async channel. You might also try Clojure atoms or promises; Kria doesn't care. There are, of course, several Riak drivers for Java and Clojure. I hope some people find this one useful. I have a section in the README about why I made it. To summarize: * The 1.4.X Java driver wasn't doing it for me. * I started this before the new Java client was far along. * I didn't expect to like a Java style API. * I used Java 7 NIO.2 instead of Netty -- because it is simpler and has less dependencies. * First class functions in Clojure make async callbacks easier. * Using Clojure made it easy to abstract out the boilerplate in the API. I welcome your feedback. It does not yet support all of the Riak API, but the essentials are there. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Re: Questions about the Riak Horizontal Scalability
A few things:
1. can you provide the output from `riak-admin member-status`?
2. using five VMs per physical node likely means that you're bottleneck is
that host running the VMs not Riak
3. uses a shared disk via iSCSI for storage is most certainly also a
bottleneck
A benchmark that scales linearly uses distinct resources, not shared
resources. That's the dynamo model, that's the way to test Riak. Setup 5
distinct machines, run the basho bench test. Add five more, run it again.
You'll see a difference. Also, if possible use two network interfaces: 1)
for inbound TCP/PB or HTTP/TCP traffic from basho_bench (clients
communicating with the cluster), 2) for communication between nodes
(disterl messages) and handoff.
Also, make sure that the system running basho_bench is not also a Riak
node. That'd skew results as well.
-greg
@gregburd | Basho Technologies | Riak | http://basho.com | @basho
On Mon, Mar 10, 2014 at 2:43 AM, yanqun@mesheven.com <
yanqun@mesheven.com> wrote:
> Greate thanks to all of you!
> I'm very sorry for a late response!
> Is it polite to make a reply to all in one email? (If not, ask for
> forgiveness)
>
> To Alexander:
> There are 2 physical machines in our cluster, and 5 riak VMs on each
> machine.
> Two physical CPUs in each machine (totally 12 cores), and I assigned 2
> cores for each riak VM.
> All the 12 riak VMs are based on the same shared storage server (by
> FreeNAS+iSCSI), whicn means all riak VMs don't use machines' local disks as
> their storage.
> And they are in a 1Gb LAN, each physical machine has 6 NICs, 4 of them are
> for iSCSI(multipath).
> So the network bandwidth is probably not the bottleneck, but disk IO
> probably be.
>
> To Sean Cribbs:
> Yes, as you said, more "concurrent" means more load for basho benchmark
> machine.
> But during my previous tests, I noticed that cpu utilization of basho
> benchmark machine is about 50%.
> But I want to try run basho benchmark tool on two machines as you
> suggested.
> I noticed the command "basho_bench" has an option "-J --join", I tried but
> faild with error report "escript: exception throw: {no_host,'
> root@10.11.20.44'} ".
> I have configured SSH public-key&no-password login for those two basho
> benchmark machines(they have same environments like OS,erlang and basho
> benchmark versions). I searched about this, but didn't find how to run
> basho benchmark tool on two separate machines.
> Could you tell me how to do or is there any guidance about this?
>
> To Christian Dahlqvist:
> Thanks for your tips, and I think we will try
> "basho_bench_driver_riakc_pb".
> Not only because of this "Horizontal Scalability" problem, but also for
> better performance.
> And also, I will change the key generator for subsequent tests.
>
> To Evan Vigil-McClanahan:
> Yes, I am using Riak 2.0 for tests.
> And "numa zones", "sbt"? Sorry I know little about this, but I will read
> more about this to check if it can help me.
>
> I will run more tests with the suggestions from all of you!
> Thank you again!
> And I will let you know if get any progress or result.
>
>
>
>
>
>
>
>
>
>
>
> *From:* Sean Cribbs
> *Date:* 2014-03-08 06:54
> *To:* Alexander Sicular
> *CC:* yanqun@mesheven.com; riak-users
> *Subject:* Re: Questions about the Riak Horizontal Scalability
> Note as well that the 'concurrent' setting will *not* typically increase
> throughput beyond a certain point (instead you will simply have many
> workers waiting in the run queue). You are also only driving load from a
> single machine with limits on its CPU and network throughput. Start by
> trying a more reasonable concurrency level, like two (2) times the number
> of CPU cores in the basho_bench machine. Also consider running basho_bench
> from multiple machines at once. Consider the line rate of your network as
> well; in the past I have easily been able to saturate a gigabit network
> with basho_bench.
>
>
> On Fri, Mar 7, 2014 at 4:45 PM, Alexander Sicular wrote:
>
>> How many physical machines (disks/cores/nics) are in your cluster?
>>
>> -Alexander
>>
>> @siculars
>> http://siculars.posthaven.com
>>
>> Sent from my iRotaryPhone
>>
>> On Mar 6, 2014, at 22:07, "yanqun@mesheven.com" <
>> yanqun@mesheven.com> wrote:
>>
>> Greetings all...
>>
>> I am running tests with Riak Cluster.
>> When this cluster has only one node, the max throughput is about 1500
>> ops/s(CPU becomes the bottleneck).
>> But after I added 4 more nodes into the cluster, no matter how I changed
>> the configuration file of Basho benchmark tool,
>> the number of throughput have never been more than ~3500 ops/s.
>> CPU utilization on each node is about 60~65% (captured by iostat and sar)
>>
>> And when I added 5 more nodes(10 nodes totally), it is the same result.
>> And CPU utilization on each node is about 60%.
>> I can't understand why.
>>
>> During the tests, I tried:
>> 1)Change ring size from 64 to 128
>> 2)Change basho benchmark configuration file, like "mode"
Re: What will happen when I remove a node from the cluster?
Hi Yaochitc, Answers below: On Mar 11, 2014, at 2:47 AM, yaochitc wrote: > Hello, I'm trying to do some test with a riak consist of 8 nodes. I tried to > leave a node from the cluster, and I can see the ownership handoff through > ring-status.But when I tried to remove a node, such process didn't occur.I > have several questions about node-leaving and node-removing.Please tell me > the answers if U know, thanks a lot! > 1. After I leave a node from the cluster, it takes several hours before the > ownership handoff finishes, and another hours before the nodes repartition > finishes.During this time, is the cluster available for reading and writing? Yes, the cluster is available for reading and writing while a node is being removed or added. > 2. I saw nothing like repartition happened between nodes still in the cluster > through ring-status command after a node removed, what will happen to make > the ring works well (I mean, operations like filling the lost copies on the > removed node) , if there are such processes, how can I observe them? Is the > cluster available for reading and writing before they finish? What process did you use to “leave” a node and “remove” a node? The correct process should be to run a “riak-admin cluster leave ” on the node you’d like to take out of the cluster, and this should handoff the node’s partitions and shut it down afterwards. You can find more info about the command here: http://docs.basho.com/riak/latest/ops/running/nodes/adding-removing/#Removing-a-Node-From-a-Cluster. Thanks, Alex___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: How to disassociate a bucket with a yokozuna index?
Hi Ryan, Thanks for the help. Now I’m working with riak 2.0 pre17, have tried both set bucket property search_index to other index or _dont_index_, but still cannot delete the index. Failure: riakasaurus.exceptions.RiakPBCException: Can't delete index with associate buckets [<<"riakasaurus.tests.test_search">>] (0) On 2014年3月11日 at 上午3:51:25, Ryan Zezeski (rzeze...@basho.com) wrote: > On Sun, Mar 9, 2014 at 8:31 AM, EmiNarcissus wrote: > > > > I'm testing on yokozuna api now, but found every time I call > > delete_search_index function it alerts cannot delete because of pre-existed > > associated bucket. I've tried to set the bucket search-index to another > > index, still have the same error. > > > > Is this part not implemented yet? or is something I missed from? > > > > Hi Tim, > > An index may not be deleted if it has any buckets associated with > it. The 'search_index' property (not 'search-index') must be changed > to either a different index or the sentinel value '_dont_index_'. > > -Z > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
