Hi Patrick, I suggested increasing ports as you had an emfile on a socket accept call. Erlang uses ports for things like network sockets and file handles when opened by *erlang* processes. However, the leveldb library manages it's own sockets as it is a C++ library dynamically loaded by the emulator and so doesn't count towards ports.
Is it possible that Riak started getting more client load if request latency increased? Changing max_open_files will keep the number of process-level file handles lower, but will cause more opening and closing of files to search them. If you have a nice modern OS with lots of file handles available, you may be able to increase max_open_files for increased performance. If you want to check how many ports you are using you can run this from the riak console. (dev1@127.0.0.1)7> length(erlang:ports()). 39 Try increasing your max_open_ports and checking how many file handles are in use using a tool like lsof and check the number of ports you have opened. Cheers, Jon. On Mon, Sep 26, 2011 at 12:44 PM, Patrick Van Stee <vans...@highgroove.com>wrote: > Thanks for the quick response Jon. I bumped it from 4096 up to the max I > have set in /etc/riak/defaults and writes actually slowed down a little bit > (~10 less writes per second). Shouldn't the max_open_files setting keep the > total amount of fd's pretty low? Maybe I'm misunderstanding what that option > is used for. > > Patrick > > On Sep 26, 2011, at 2:34 PM, Jon Meredith wrote: > > Hi Patrick, > > You may be running out of ports which erlang uses for TCP sockets - try > increasing ERL_MAX_PORTS in etc/vm.args > > Cheers, Jon > Basho Technologies. > > On Mon, Sep 26, 2011 at 12:17 PM, Patrick Van Stee <vans...@highgroove.com > > wrote: > >> We're running a small, 2 node riak cluster (on 2 m1.large boxes) using the >> LevelDB backend and have been trying to write ~250 keys a second at it. With >> a small dataset everything was running smoothly. However, after storing >> several hundred thousand keys some performance issues started to show up. >> >> * We're running out of file descriptors which is causing nodes to crash >> with the following error: >> >> 2011-09-24 00:23:52.097 [error] <0.110.0> CRASH REPORT Process [] with 0 >> neighbours crashed with reason: {error,accept_failed} >> 2011-09-24 00:23:52.098 [error] <0.121.0> application: mochiweb, "Accept >> failed error", "{error,emfile} >> >> Setting the max_open_files limit in the app.config doesn't seem to help. >> >> * Writes have slowed down by an order of magnitude. I even set the n_val, >> w, and dw bucket properties to 1 without any noticeable difference. Also we >> switched to using protocol buffers to make sure there wasn't any extra >> overhead when using HTTP. >> >> * Running map reduce jobs that use a range query on a secondary index >> started returning an error, {"error":"map_reduce_error"}, once our dataset >> increased in size. Feeding a list of keys works fine, but querying the index >> for keys seems to be timing out: >> >> 2011-09-26 16:37:57.192 [error] <0.136.0> Supervisor riak_pipe_fitting_sup >> had child undefined started with {riak_pipe_fitting,start_link,undefined} at >> <0.3497.0> exit with reason >> {timeout,{gen_server,call,[{riak_pipe_vnode_master,'riak@10.206.105.52 >> '},{return_vnode,{'riak_vnode_req_v1',502391187832497878132516661246222288006726811648,{raw,#Ref<0.0.1.88700>,<0.3500.0>},{cmd_enqueue,{fitting,<0.3499.0>,#Ref<0.0.1.88700>,#Fun<riak_kv_mrc_pipe.0.133305895>,#Fun<riak_kv_mrc_pipe.1.125635227>},{<<"ip_queries">>,<<"uaukXZn5rZQ0LrSED3pi-fE-JjU">>},infinity,[{502391187832497878132516661246222288006726811648,' >> riak@10.206.105.52'}]}}}]}} in context child_terminated >> >> Is anyone familiar with these problems or is there anything else I can try >> to increase the performance when using LevelDB? >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com