Hi JB,
The gossip crashes you are seeing are the result in some changes to the way
gossiping takes place. There can be a spike in gossip requests around
transfer at the moment. The next RC should have a change to mitigate it if
we can get it reviewed in time. I didn't see any errors related to leveldb
in there.
Jon Meredith
Basho Technologies.
On Mon, Sep 26, 2011 at 12:37 PM, JB Smith <jbsmith_...@mac.com> wrote:
> We are seeing similar issues using levelDB with 1.0.0RC1
>
> We run a 6 way cluster on EC2 64bit micro instances.
>
> Initially we had assumed that there may be some issue with small memory
> environments.
>
> on 0.14.2 things would seem to balance out and stabilize.
>
> on 1.0.0RC1, once this sort of report starts showing up, it tends to run
> out of control until the node crashes.
>
> We also use a multi backend configuration as follows:
>
> {storage_backend, riak_kv_multi_backend},
> {multi_backend_default, <<"eleveldb">>},
> {multi_backend, [
> {<<"eleveldb">>, riak_kv_eleveldb_backend, []},
> {<<"sessions">>, riak_kv_bitcask_backend, [
> {expiry_secs, 7200},
> {data_root, "/riak/data/bitcask_sessions"}]}
> ]},
>
> We are now storing over 150,000 keys in the levelDB buckets
>
> 17:17:03.282 [error] CRASH REPORT Process [] with 0 neighbours crashed with
> reason:
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,122743301572712549771012593372656581728916209664,'
> riak...@riak004.xx.com','riak...@riak005.xx.com',riak_pipe_vnode}]}}
> 17:17:04.299 [error] Supervisor riak_core_vnode_sup had child undefined
> started with {riak_core_vnode,start_link,undefined} at <0.16220.0> exit with
> reason
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,122743301572712549771012593372656581728916209664,'
> riak...@riak004.xx.com','riak...@riak005.xx.com',riak_pipe_vnode}]}} in
> context child_terminated
> 17:17:05.814 [info] monitor long_gc <0.44.0>
> [{name,lager_crash_log},{initial_call,{lager_crash_log,init,1}},{almost_current_function,{gen,do_call,4}}]
> [{timeout,1501},{old_heap_block_size,0},{heap_block_size,1597},{mbuf_size,0},{stack_size,285},{old_heap_size,0},{heap_size,1179}]
> 17:17:07.349 [error] gen_fsm <0.16589.0> in state active terminated with
> reason:
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'
> riak...@riak004.xx.com','riak...@riak006.xx.com',riak_pipe_vnode}]}}
> 17:17:07.351 [info] monitor long_gc <0.101.0>
> [{name,riak_core_ring_manager},{initial_call,{gen,init_it,7}},{almost_current_function,{erl_syntax,is_tree,1}}]
> [{timeout,1510},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,2363},{old_heap_size,0},{heap_size,275697}]
> 17:17:10.345 [error] CRASH REPORT Process [] with 0 neighbours crashed with
> reason:
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'
> riak...@riak004.xx.com','riak...@riak006.xx.com',riak_pipe_vnode}]}}
> 17:17:10.348 [info] monitor long_gc <0.17891.0>
> [{initial_call,{erlang,apply,2}},{almost_current_function,{erl_scan,set_attribute,3}}]
> [{timeout,1481},{old_heap_block_size,0},{heap_block_size,121393},{mbuf_size,0},{stack_size,99},{old_heap_size,0},{heap_size,116704}]
> 17:17:11.847 [error] gen_server <0.16590.0> terminated with reason:
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'
> riak...@riak004.xx.com','riak...@riak006.xx.com',riak_pipe_vnode}]}}
> 17:17:13.339 [info] monitor long_gc <0.17891.0>
> [{initial_call,{erlang,apply,2}},{almost_current_function,{erl_expand_records,expr,2}}]
> [{timeout,1473},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,1364},{old_heap_size,0},{heap_size,151085}]
> 17:17:13.342 [error] CRASH REPORT Process [] with 0 neighbours crashed with
> reason:
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'
> riak...@riak004.xx.com','riak...@riak006.xx.com',riak_pipe_vnode}]}}
> 17:17:14.842 [error] Supervisor riak_core_vnode_sup had child undefined
> started with {riak_core_vnode,start_link,undefined} at <0.16589.0> exit with
> reason
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,161992613122126446500115457532517697979441741824,'
> riak...@riak004.xx.com','riak...@riak006.xx.com',riak_pipe_vnode}]}} in
> context child_terminated
> 17:17:14.857 [error] gen_fsm <0.15083.0> in state active terminated with
> reason:
> {timeout,{gen_server,call,[riak_core_gossip,{finish_handoff,72076008481650973993443441457199504387328704512,'
> riak...@riak004.xx.com','riak...@riak006.xx.com',riak_kv_vnode}]}}
> 17:17:16.338 [info] monitor long_gc <0.17891.0>
> [{initial_call,{erlang,apply,2}},{almost_current_function,{zlib,deflateInit,2}}]
> [{timeout,1470},{old_heap_block_size,0},{heap_block_size,75025},{mbuf_size,0},{stack_size,34},{old_heap_size,0},{heap_size,36434}]
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
