Hi Luke,
Thanks for the fast reply!
Ok, yes, our limit is being inherited (I just raised it to 131072 after our
latest issue a couple of hours ago):
$ riak attach
Remote Shell: Use "Ctrl-C a" to quit. q() or init:stop() will terminate the
riak node.
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:16:16] [async-threads:0]
[kernel-poll:false]
Eshell V5.9.1 (abort with ^G)
(riak@riak-01)1> os:cmd('ulimit -n').
"131072\n"
(riak@riak-01)2>
--
Dave Brady
----- Original Message -----
From: "Luke Bakken" <[email protected]>
To: "Dave Brady" <[email protected]>
Cc: "riak-users" <[email protected]>
Sent: Lundi 11 Novembre 2013 21:10:54
Subject: Re: max_files_limit and AAE
Hi Dave,
Just to confirm that the ulimit settings "stuck", could you please run riak
attach and execute the following Erlang snippet?
os:cmd('ulimit -n').
The period is significant and please exit using CTRL-C twice.
Thanks!
--
Luke Bakken
CSE
[email protected]
On Mon, Nov 11, 2013 at 11:56 AM, Dave Brady < [email protected] > wrote:
Hey Everyone,
We have a five-node, 128 partition cluster running 1.4.2 on Debian.
Is there a doc somewhere that explains how to size max_open_files as it applies
to AAE?
I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst
files in one VNode's data directory, and the boxes have plenty of RAM.
I set max_open_files in the AAE section to 3000, too, on whim after we had our
first issue. Still got these in the logs on a couple of nodes after running for
less than one day:
===============
2013-11-09 11:37:12.438 [info]
<0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142
riak_kv/125597796958124469533129165311555572001681702912: unable to start
index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK:
Too many open
files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422>
terminated with reason: no match of right hand value {error,{db_write,"IO
error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log:
Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process
<0.5209.2422> with 1 neighbours exited with reason: no match of right hand
value {error,{db_write,"IO error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log:
Too many open files"}} in hashtree:flush_buffer/1 line 302 in
gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process
<0.19959.2426> with 0 neighbours exited with reason: no match of right hand
value {error,{db_open,"IO error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK:
Too many open files"}} in hashtree:new_segment_store/2 line 499 in
gen_server:init_it/6 line 328
===============
Our init script has "ulimit -n 65536" in it, which I *thought* that would be
high enough. Maybe not?
I also made the necessary tweaks to /etc/pam.d/common-session*, so that
/etc/security/limts.conf would be read, and that did not help.
Much obliged for any suggestions!
--
Dave Brady
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
