We're running a small, 2 node riak cluster (on 2 m1.large boxes) using the
LevelDB backend and have been trying to write ~250 keys a second at it. With a
small dataset everything was running smoothly. However, after storing several
hundred thousand keys some performance issues started to show up.
* We're running out of file descriptors which is causing nodes to crash with
the following error:
2011-09-24 00:23:52.097 [error] <0.110.0> CRASH REPORT Process [] with 0
neighbours crashed with reason: {error,accept_failed}
2011-09-24 00:23:52.098 [error] <0.121.0> application: mochiweb, "Accept failed
error", "{error,emfile}
Setting the max_open_files limit in the app.config doesn't seem to help.
* Writes have slowed down by an order of magnitude. I even set the n_val, w,
and dw bucket properties to 1 without any noticeable difference. Also we
switched to using protocol buffers to make sure there wasn't any extra overhead
when using HTTP.
* Running map reduce jobs that use a range query on a secondary index started
returning an error, {"error":"map_reduce_error"}, once our dataset increased in
size. Feeding a list of keys works fine, but querying the index for keys seems
to be timing out:
2011-09-26 16:37:57.192 [error] <0.136.0> Supervisor riak_pipe_fitting_sup had
child undefined started with {riak_pipe_fitting,start_link,undefined} at
<0.3497.0> exit with reason
{timeout,{gen_server,call,[{riak_pipe_vnode_master,'riak@10.206.105.52'},{return_vnode,{'riak_vnode_req_v1',502391187832497878132516661246222288006726811648,{raw,#Ref<0.0.1.88700>,<0.3500.0>},{cmd_enqueue,{fitting,<0.3499.0>,#Ref<0.0.1.88700>,#Fun<riak_kv_mrc_pipe.0.133305895>,#Fun<riak_kv_mrc_pipe.1.125635227>},{<<"ip_queries">>,<<"uaukXZn5rZQ0LrSED3pi-fE-JjU">>},infinity,[{502391187832497878132516661246222288006726811648,'riak@10.206.105.52'}]}}}]}}
in context child_terminated
Is anyone familiar with these problems or is there anything else I can try to
increase the performance when using LevelDB?
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
