Hi Patrick,

I suggested increasing ports as you had an emfile on a socket accept call.
Erlang uses ports for things like network sockets and file handles when
opened by *erlang* processes.  However, the leveldb library manages it's own
sockets as it is a C++ library dynamically loaded by the emulator and so
doesn't count towards ports.

Is it possible that Riak started getting more client load if request latency
increased?  Changing max_open_files will keep the number of process-level
file handles lower, but will cause more opening and closing of files to
search them.  If you have a nice modern OS with lots of file handles
available, you may be able to increase max_open_files for increased
performance.

If you want to check how many ports you are using you can run this from the
riak console.

  (dev1@127.0.0.1)7> length(erlang:ports()).
  39

Try increasing your max_open_ports and checking how many file handles are in
use using a tool like lsof and check the number of ports you have opened.

Cheers, Jon.

On Mon, Sep 26, 2011 at 12:44 PM, Patrick Van Stee
<vans...@highgroove.com>wrote:

> Thanks for the quick response Jon. I bumped it from 4096 up to the max I
> have set in /etc/riak/defaults and writes actually slowed down a little bit
> (~10 less writes per second). Shouldn't the max_open_files setting keep the
> total amount of fd's pretty low? Maybe I'm misunderstanding what that option
> is used for.
>
> Patrick
>
> On Sep 26, 2011, at 2:34 PM, Jon Meredith wrote:
>
> Hi Patrick,
>
> You may be running out of ports which erlang uses for TCP sockets - try
> increasing ERL_MAX_PORTS in etc/vm.args
>
> Cheers, Jon
> Basho Technologies.
>
> On Mon, Sep 26, 2011 at 12:17 PM, Patrick Van Stee <vans...@highgroove.com
> > wrote:
>
>> We're running a small, 2 node riak cluster (on 2 m1.large boxes) using the
>> LevelDB backend and have been trying to write ~250 keys a second at it. With
>> a small dataset everything was running smoothly. However, after storing
>> several hundred thousand keys some performance issues started to show up.
>>
>> * We're running out of file descriptors which is causing nodes to crash
>> with the following error:
>>
>> 2011-09-24 00:23:52.097 [error] <0.110.0> CRASH REPORT Process [] with 0
>> neighbours crashed with reason: {error,accept_failed}
>> 2011-09-24 00:23:52.098 [error] <0.121.0> application: mochiweb, "Accept
>> failed error", "{error,emfile}
>>
>> Setting the max_open_files limit in the app.config doesn't seem to help.
>>
>> * Writes have slowed down by an order of magnitude. I even set the n_val,
>> w, and dw bucket properties to 1 without any noticeable difference. Also we
>> switched to using protocol buffers to make sure there wasn't any extra
>> overhead when using HTTP.
>>
>> * Running map reduce jobs that use a range query on a secondary index
>> started returning an error, {"error":"map_reduce_error"}, once our dataset
>> increased in size. Feeding a list of keys works fine, but querying the index
>> for keys seems to be timing out:
>>
>> 2011-09-26 16:37:57.192 [error] <0.136.0> Supervisor riak_pipe_fitting_sup
>> had child undefined started with {riak_pipe_fitting,start_link,undefined} at
>> <0.3497.0> exit with reason
>> {timeout,{gen_server,call,[{riak_pipe_vnode_master,'riak@10.206.105.52
>> '},{return_vnode,{'riak_vnode_req_v1',502391187832497878132516661246222288006726811648,{raw,#Ref<0.0.1.88700>,<0.3500.0>},{cmd_enqueue,{fitting,<0.3499.0>,#Ref<0.0.1.88700>,#Fun<riak_kv_mrc_pipe.0.133305895>,#Fun<riak_kv_mrc_pipe.1.125635227>},{<<"ip_queries">>,<<"uaukXZn5rZQ0LrSED3pi-fE-JjU">>},infinity,[{502391187832497878132516661246222288006726811648,'
>> riak@10.206.105.52'}]}}}]}} in context child_terminated
>>
>> Is anyone familiar with these problems or is there anything else I can try
>> to increase the performance when using LevelDB?
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Reply via email to