I suspect that, given the large heap messages, you're seeing the known issues where when a custom bucket is created and the moving that metadata around the ring takes increasingly long. It isn't recommended to create a large number of custom buckets at the moment.
What the process likely looks like is this: 1. you do the set. 2. bucket metadata starts being gossip around the ring. 3. you do the put. it succeeds, but on some nodes, the metadata hasn't been committed, so it's put into the default backend. 4. gossip finishes. 5. you do the get. it fails because now the bucket for this backend has made 2 replicas unreachable. 6. read repair happens, repopulating the 2 missing replicas. 7. you re-get and it works. On Wed, Apr 30, 2014 at 4:04 PM, Oleksiy Krivoshey <[email protected]> wrote: > The objects are really small (50 - 500 bytes). > > It doesn't happen if the bucket was already created. > It also does't happen if I don't call SetBucket at all (so using default > backend and options). > And it seems it doesn't happen if I call SetBucket but don't set `backend` > property. > > > > On 1 May 2014 00:51, Evan Vigil-McClanahan <[email protected]> wrote: >> >> Does this continue if the bucket hasn't been created recently? Does >> it matter how large the object is? Is it particularly large in this >> case? >> >> On Wed, Apr 30, 2014 at 3:47 PM, Oleksiy Krivoshey <[email protected]> >> wrote: >> > Hi guys, >> > >> > can someone please suggest what can be the reason for 'get' immediately >> > following successful 'put' to fail? >> > >> > I'm running a fully connected, healthy, 5 node Riak 2.0-beta1 cluster. >> > Using >> > a multiple backend feature, so the order of operations is: >> > >> > 1. 'SetBucket' for a new bucket with a backend name. Wait for successful >> > reply >> > 2. 'Put' object to bucket. Wait for successful reply (return_head: true, >> > so >> > I get clock, vtag, etc back) >> > 3. 'Get' object from step 2. Returns empty response (missing object) >> > >> > If I repeat 'Get' after just few moments - I will successfully retrieve >> > the >> > object. >> > >> > CAP options are all defaults (n: 3, etc). Happens with both custom >> > bitcask >> > and custom leveldb backends. >> > >> > There are no errors in riak logs, but a lot of the following: >> > >> > 2014-04-30 21:41:05.656 [info] >> > <0.95.0>@riak_core_sysmon_handler:handle_event:92 monitor large_heap >> > <0.10689.74> >> > >> > [{initial_call,{erlang,apply,2}},{almost_current_function,{eval_bits,expr_grp,4}},{message_queue_len,0}] >> > >> > [{old_heap_block_size,0},{heap_block_size,79467343},{mbuf_size,0},{stack_size,394},{old_heap_size,0},{heap_size,28858704}] >> > >> > -- >> > Oleksiy >> > >> > _______________________________________________ >> > riak-users mailing list >> > [email protected] >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > > > > -- > Oleksiy Krivoshey _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
