FWIW, on ubuntu you can drop a file in /etc/security/limits.d/ called 'riak.conf' with the appropriate stanzas you would normally include in limits.conf and it will get read.
If you use chef or puppet, this is much easier than trying to have multiple resources attempt to manage limits.conf together. I do this for our java apps as well. On Wed, Aug 1, 2012 at 8:20 PM, Sean Carey <[email protected]> wrote: > John, > Glad things are starting to run smoothly. This pam.d setting has tripped me > a couple of times. > > Best, > > Sean > > On Wednesday, August 1, 2012 at 8:16 PM, John Roy wrote: > > All -- > > The pam.d/su and limits.conf changes seem to have brought us back to > reliability -- so far so good. The time consuming part was the reboot. I > double checked the ulimit in the risk console and all came up to 8192 -- my > new limit. > > thanks for all your help, > > John > > > On Aug 1, 2012, at 2:56 PM, Sean Carey wrote: > > John, > Please make sure in /etc/pam.d/su, that the following line is uncommented: > > session required pam_limits.so > > I have noticed lately in Ubuntu that this line is commented out by default. > > > Best, > > > Sean > > On Wednesday, August 1, 2012 at 5:47 PM, Jared Morrow wrote: > > You will need to make the adjustments in the /etc/security/limits.conf file > as described here http://wiki.basho.com/Open-Files-Limit.html > > -Jared > > > On Aug 1, 2012, at 3:33 PM, John Roy <[email protected]> wrote: > > Hi Reid -- > > I added a risk.conf file in /etc/default with the line: > > ulimit -n 8192 > > then rebooted, restarted risk and then did the attach. > > I got this line (which is also in the crash.log), then the limit of 1024. > See below: > > 16:28:58.041 [error] Hintfile > '/disk1/riak/bitcask/159851741583067506678528028578343455274867621888/12.bitcask.hint' > contains pointer 118308596 570 that is greater than total data size > 118308864 > >> os:cmd("ulimit -n"). > "1024\n" > > I also set in manually prior to the reboot and got the same result. > > > On Aug 1, 2012, at 2:09 PM, Reid Draper wrote: > > ulimit of 4096 might be too low. I'd also double-check the ulimit > has taken effect either by attaching to the node (riak attach) or > starting the node in the console (riak console), then type this: > > os:cmd("ulimit -n"). > > Be sure to include the period (.) that > is above as well. > > Reid > > > On Aug 1, 2012, at 5:00 PM, John Roy wrote: > > Hi -- > > Riak 1.1.1 > three nodes > Ubuntu 10.04.1 LTS > downtime means one node drops off then the other two follow so the entire > cluster falls down. > > On Aug 1, 2012, at 1:48 PM, Mark Phillips wrote: > > Hey John, > > First questions would be: > > * What version of Riak? > * How many nodes? > * Which OS? > * When you say "downtime" do you mean the entire cluster? Or just a subset > of your nodes? > > Mark > > On Wed, Aug 1, 2012 at 1:42 PM, John Roy <[email protected]> wrote: > > I'm seeing significant downtime on Riak now. Much like the "Riak Crashing > Constantly" thread. However in this case we get a "Too many open files" > error, and also "contains pointer that is greater than the total data size." > See the error messages below for more details. > > If others have an idea on this I'd appreciate your help. > > Thanks! > > John > > 2012-08-01 14:53:28 =ERROR REPORT==== > Hintfile > '/disk1/riak/bitcask/1347321821914426127719021955160323408745312813056/12.bitcask.hint' > contains pointer 119561351 4415 that is greater than total data size > 119562240 > > 2012-08-01 14:54:26 =CRASH REPORT==== > crasher: > initial call: riak_core_vnode:init/1 > pid: <0.29239.0> > registered_name: [] > exception exit: [{riak_kv_eleveldb_backend,{db_open,"IO error: > /disk1/riak/leveldb/753586781748746817198774991869333432010090217472/CURRENT: > Too many open files"}}] > in function gen_fsm:init_it/6 > in call from proc_lib:init_p_do_apply/3 > ancestors: [riak_core_vnode_sup,riak_core_sup,<0.88.0>] > messages: [] > links: [<0.92.0>] > dictionary: > [{#Ref<0.0.0.37922>,{bc_state,"/disk1/riak/bitcask/753586781748746817198774991869333432010090217472",fresh,undefined,[{filestate,read_only,"/disk1/riak/bitcask/753586781748746817198774991869333432010090217472/1.bitcask.data",1,<<>>,undefined,0,0},{filestate,read_only,"/disk1/riak/bitcask/753586781748746817198774991869333432010090217472/2.bitcask.data",2,<<>>,undefined,0,0}],2147483648,[{expiry_secs,-1},{read_write,true}],<<>>}},{random_seed,{17770,26756,17419}}] > trap_exit: true > status: running > heap_size: 4181 > stack_size: 24 > reductions: 13734 > neighbours: > 2012-08-01 14:54:26 =SUPERVISOR REPORT==== > Supervisor: {local,riak_core_vnode_sup} > Context: child_terminated > Reason: [{riak_kv_eleveldb_backend,{db_open,"IO error: > /disk1/riak/leveldb/753586781748746817198774991869333432010090217472/CURRENT: > Too many open files"}}] > Offender: > [{pid,<0.29239.0>},{name,undefined},{mfargs,{riak_core_vnode,start_link,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,worker}] > > 2012-08-01 14:54:26 =ERROR REPORT==== > ** Generic server riak_core_vnode_manager terminating > ** Last message in was > {753586781748746817198774991869333432010090217472,riak_kv_vnode,get_vnode} > ** When Server state == > {state,12308,[{{riak_kv_vnode,0},undefined},{{riak_kv_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_kv_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_kv_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_kv_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_kv_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_kv_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_kv_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_kv_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_kv_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_kv_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_kv_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_kv_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_kv_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_kv_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_kv_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_kv_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_kv_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_kv_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_kv_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_kv_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_kv_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_kv_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_kv_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_kv_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_kv_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_kv_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_kv_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_kv_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_kv_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_kv_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_kv_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_kv_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_kv_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_kv_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_kv_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_kv_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_kv_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_kv_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_kv_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_kv_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_kv_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_kv_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_kv_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_kv_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_kv_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_kv_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_kv_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_kv_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_kv_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_kv_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_kv_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_kv_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_kv_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_kv_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_kv_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_kv_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_kv_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_kv_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_kv_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_kv_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_kv_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_kv_vnode,1438665674247607560106752257205091097473808596992},undefined},{{riak_pipe_vnode,0},undefined},{{riak_pipe_vnode,22835963083295358096932575511191922182123945984},undefined},{{riak_pipe_vnode,45671926166590716193865151022383844364247891968},undefined},{{riak_pipe_vnode,68507889249886074290797726533575766546371837952},undefined},{{riak_pipe_vnode,91343852333181432387730302044767688728495783936},undefined},{{riak_pipe_vnode,114179815416476790484662877555959610910619729920},undefined},{{riak_pipe_vnode,137015778499772148581595453067151533092743675904},undefined},{{riak_pipe_vnode,159851741583067506678528028578343455274867621888},undefined},{{riak_pipe_vnode,182687704666362864775460604089535377456991567872},undefined},{{riak_pipe_vnode,205523667749658222872393179600727299639115513856},undefined},{{riak_pipe_vnode,228359630832953580969325755111919221821239459840},undefined},{{riak_pipe_vnode,251195593916248939066258330623111144003363405824},undefined},{{riak_pipe_vnode,274031556999544297163190906134303066185487351808},undefined},{{riak_pipe_vnode,296867520082839655260123481645494988367611297792},undefined},{{riak_pipe_vnode,319703483166135013357056057156686910549735243776},undefined},{{riak_pipe_vnode,342539446249430371453988632667878832731859189760},undefined},{{riak_pipe_vnode,365375409332725729550921208179070754913983135744},undefined},{{riak_pipe_vnode,388211372416021087647853783690262677096107081728},undefined},{{riak_pipe_vnode,411047335499316445744786359201454599278231027712},undefined},{{riak_pipe_vnode,433883298582611803841718934712646521460354973696},undefined},{{riak_pipe_vnode,456719261665907161938651510223838443642478919680},undefined},{{riak_pipe_vnode,479555224749202520035584085735030365824602865664},undefined},{{riak_pipe_vnode,502391187832497878132516661246222288006726811648},undefined},{{riak_pipe_vnode,525227150915793236229449236757414210188850757632},undefined},{{riak_pipe_vnode,548063113999088594326381812268606132370974703616},undefined},{{riak_pipe_vnode,570899077082383952423314387779798054553098649600},undefined},{{riak_pipe_vnode,593735040165679310520246963290989976735222595584},undefined},{{riak_pipe_vnode,616571003248974668617179538802181898917346541568},undefined},{{riak_pipe_vnode,639406966332270026714112114313373821099470487552},undefined},{{riak_pipe_vnode,662242929415565384811044689824565743281594433536},undefined},{{riak_pipe_vnode,685078892498860742907977265335757665463718379520},undefined},{{riak_pipe_vnode,707914855582156101004909840846949587645842325504},undefined},{{riak_pipe_vnode,730750818665451459101842416358141509827966271488},undefined},{{riak_pipe_vnode,753586781748746817198774991869333432010090217472},undefined},{{riak_pipe_vnode,776422744832042175295707567380525354192214163456},undefined},{{riak_pipe_vnode,799258707915337533392640142891717276374338109440},undefined},{{riak_pipe_vnode,822094670998632891489572718402909198556462055424},undefined},{{riak_pipe_vnode,844930634081928249586505293914101120738586001408},undefined},{{riak_pipe_vnode,867766597165223607683437869425293042920709947392},undefined},{{riak_pipe_vnode,890602560248518965780370444936484965102833893376},undefined},{{riak_pipe_vnode,913438523331814323877303020447676887284957839360},undefined},{{riak_pipe_vnode,936274486415109681974235595958868809467081785344},undefined},{{riak_pipe_vnode,959110449498405040071168171470060731649205731328},undefined},{{riak_pipe_vnode,981946412581700398168100746981252653831329677312},undefined},{{riak_pipe_vnode,1004782375664995756265033322492444576013453623296},undefined},{{riak_pipe_vnode,1027618338748291114361965898003636498195577569280},undefined},{{riak_pipe_vnode,1050454301831586472458898473514828420377701515264},undefined},{{riak_pipe_vnode,1073290264914881830555831049026020342559825461248},undefined},{{riak_pipe_vnode,1096126227998177188652763624537212264741949407232},undefined},{{riak_pipe_vnode,1118962191081472546749696200048404186924073353216},undefined},{{riak_pipe_vnode,1141798154164767904846628775559596109106197299200},undefined},{{riak_pipe_vnode,1164634117248063262943561351070788031288321245184},undefined},{{riak_pipe_vnode,1187470080331358621040493926581979953470445191168},undefined},{{riak_pipe_vnode,1210306043414653979137426502093171875652569137152},undefined},{{riak_pipe_vnode,1233142006497949337234359077604363797834693083136},undefined},{{riak_pipe_vnode,1255977969581244695331291653115555720016817029120},undefined},{{riak_pipe_vnode,1278813932664540053428224228626747642198940975104},undefined},{{riak_pipe_vnode,1301649895747835411525156804137939564381064921088},undefined},{{riak_pipe_vnode,1324485858831130769622089379649131486563188867072},undefined},{{riak_pipe_vnode,1347321821914426127719021955160323408745312813056},undefined},{{riak_pipe_vnode,1370157784997721485815954530671515330927436759040},undefined},{{riak_pipe_vnode,1392993748081016843912887106182707253109560705024},undefined},{{riak_pipe_vnode,1415829711164312202009819681693899175291684651008},undefined},{{riak_pipe_vnode,1438665674247607560106752257205091097473808596992},undefined}],[{{riak_kv_vnode,1324485858831130769622089379649131486563188867072},'[email protected]'}]} > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
