Hey Everyone,
We have a five-node, 128 partition cluster running 1.4.2 on Debian.
Is there a doc somewhere that explains how to size max_open_files as it applies
to AAE?
I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst
files in one VNode's data directory, and the boxes have plenty of RAM.
I set max_open_files in the AAE section to 3000, too, on whim after we had our
first issue. Still got these in the logs on a couple of nodes after running for
less than one day:
===============
2013-11-09 11:37:12.438 [info]
<0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142
riak_kv/125597796958124469533129165311555572001681702912: unable to start
index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK:
Too many open
files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422>
terminated with reason: no match of right hand value {error,{db_write,"IO
error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log:
Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process
<0.5209.2422> with 1 neighbours exited with reason: no match of right hand
value {error,{db_write,"IO error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log:
Too many open files"}} in hashtree:flush_buffer/1 line 302 in
gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process
<0.19959.2426> with 0 neighbours exited with reason: no match of right hand
value {error,{db_open,"IO error:
/var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK:
Too many open files"}} in hashtree:new_segment_store/2 line 499 in
gen_server:init_it/6 line 328
===============
Our init script has "ulimit -n 65536" in it, which I *thought* that would be
high enough. Maybe not?
I also made the necessary tweaks to /etc/pam.d/common-session*, so that
/etc/security/limts.conf would be read, and that did not help.
Much obliged for any suggestions!
--
Dave Brady
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
