Hey Godefroy, What are you using to monitor disk space/usage?
Mark On Mon, Mar 25, 2013 at 5:33 AM, Godefroy de Compreignac <[email protected]> wrote: > Sorry I talked too soon... > My problem still exist: this morning 2 of my 4 nodes were down. No more disk > space. It happens 2 times a week, and each time I'm forced to move a vnode > on a separate disk. But I won't be able to do that much longer (no more disk > space on the other disks). > I really need your help to save the cluster. Any loss is serious. > > Here are the last errors I have. I didn't included all the "busy_dist_port" > info. Thank you in advance for your help! > > $ sudo grep -F "[error]" /var/log/riak/console.log | tail -20 > 2013-03-25 10:25:13.166 [error] <0.4173.584> gen_server <0.4173.584> > terminated with reason: no function clause matching > riak_core_pb:encode({ts,{1363,975864,612168}}, > {{ts,{1363,975864,612168}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 > 2013-03-25 10:25:13.166 [error] > <0.21174.591>@riak_core_handoff_sender:start_fold:226 hinted_handoff > transfer of riak_kv_vnode from '[email protected]' > 411047335499316445744786359201454599278231027712 to '[email protected]' > 411047335499316445744786359201454599278231027712 failed because of > error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1363,975864,612168}},{{ts,{1363,975864,612168}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,108,97,98,108,111,103,45,99,97,99,104,101,109,0,0,0,35,45,86,54,67,117,116,115,122,74,75,53,97,50,74,45,79,102,73,53,119,95,73,80,109,112,71,99,64,50,53,48,120,49,54,55,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,6,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,76,105,110,107,115,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,105,97,109,97,97,97,103,97,101,97,47,97,106,97,112,97,101,97,103,106,108,0,0,0,22,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,104,97,113,97,87,97,116,97,53,97,76,97,72,97,108,97,74,97,52,97,121,97,86,97,83,97,83,97,104,97,57,97,75,97,116,97,67,97,50,97,85,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,83,98,0,14,227,248,98,0,9,86,237,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,11,88,45,82,105,97,107,45,77,101,116,97,106,106,109,0,0,93,140,255,216,255,224,0,16,74,70,73,70,0,1,1,1,0,72,0,72,0,0,255,219,0,67,0,3,2,2,3,2,2,3,3,2,3,3,3,3,3,4,7,5,4,4,4,4,9,6,7,5,7,10,9,11,11,10,9,10,10,12,13,17,14,12,12,16,12,10,10,14,20,15,16,17,18,19,19,19,11,14,20,22,20,18,22,17,18,19,18,255,219,0,67,1,3,3,3,4,4,4,8,5,5,8,18,12,10,12,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,18,255,192,0,17,8,0,167,0,250,3,1,17,0,2,17,1,3,17,1,255,196,0,29,0,0,1,4,3,1,1,0,0,0,0,0,0,0,0,0,0,6,3,4,5,7,1,2,8,0,9,255,196,...>>}],...},...]},...}}} > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] > 2013-03-25 10:25:13.167 [error] <0.4173.584> CRASH REPORT Process > <0.4173.584> with 0 neighbours exited with reason: no function clause > matching riak_core_pb:encode({ts,{1363,975864,612168}}, > {{ts,{1363,975864,612168}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in gen_server:terminate/6 line 747 > 2013-03-25 10:25:13.170 [error] <0.25119.4> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > {riak_core_vnode_worker,start_link,undefined} at <0.4173.584> exit with > reason no function clause matching > riak_core_pb:encode({ts,{1363,975864,612168}}, > {{ts,{1363,975864,612168}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in context child_terminated > 2013-03-25 10:25:17.128 [error] > <0.21808.591>@riak_core_handoff_sender:start_fold:226 hinted_handoff > transfer of riak_kv_vnode from '[email protected]' > 913438523331814323877303020447676887284957839360 to '[email protected]' > 913438523331814323877303020447676887284957839360 failed because of > error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1363,975931,872486}},{{ts,{1363,975931,872486}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,108,97,98,108,111,103,45,99,97,99,104,101,109,0,0,0,35,49,114,71,115,114,67,120,88,81,104,88,102,85,78,67,104,72,48,49,115,111,116,100,49,114,51,111,64,53,48,48,120,52,48,52,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,6,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,76,105,110,107,115,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,105,97,109,97,97,97,103,97,101,97,47,97,106,97,112,97,101,97,103,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,51,97,106,97,100,97,56,97,88,97,85,97,77,97,66,97,87,97,106,97,104,97,83,97,117,97,50,97,122,97,79,97,69,97,88,97,104,97,102,97,107,97,117,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,83,98,0,14,228,59,98,0,13,78,101,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,11,88,45,82,105,97,107,45,77,101,116,97,106,106,109,0,0,115,82,255,216,255,224,0,16,74,70,73,70,0,1,1,1,0,96,0,96,0,0,255,219,0,67,0,7,5,5,6,5,4,7,6,5,6,8,7,7,8,10,17,11,10,9,9,10,21,15,16,12,17,24,21,26,25,24,21,24,23,27,30,39,33,27,29,37,29,23,24,34,46,34,37,40,41,43,44,43,26,32,47,51,47,42,50,39,42,43,42,255,219,0,67,1,7,8,8,10,9,10,20,11,11,20,42,28,24,28,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,255,192,0,17,8,1,148,1,244,3,1,34,0,2,17,1,3,17,1,255,196,0,28,0,1,0,2,3,1,1,1,0,0,0,0,0,0,0,...>>}],...},...]},...}}} > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] > 2013-03-25 10:25:17.128 [error] <0.5408.584> gen_server <0.5408.584> > terminated with reason: no function clause matching > riak_core_pb:encode({ts,{1363,975931,872486}}, > {{ts,{1363,975931,872486}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 > 2013-03-25 10:25:17.131 [error] <0.5408.584> CRASH REPORT Process > <0.5408.584> with 0 neighbours exited with reason: no function clause > matching riak_core_pb:encode({ts,{1363,975931,872486}}, > {{ts,{1363,975931,872486}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in gen_server:terminate/6 line 747 > 2013-03-25 10:25:17.161 [error] <0.9758.5> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > {riak_core_vnode_worker,start_link,undefined} at <0.5408.584> exit with > reason no function clause matching > riak_core_pb:encode({ts,{1363,975931,872486}}, > {{ts,{1363,975931,872486}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in context child_terminated > 2013-03-25 10:25:19.966 [error] <0.5968.584> gen_server <0.5968.584> > terminated with reason: no function clause matching > riak_core_pb:encode({ts,{1363,975886,897088}}, > {{ts,{1363,975886,897088}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 > 2013-03-25 10:25:19.966 [error] > <0.22140.591>@riak_core_handoff_sender:start_fold:226 hinted_handoff > transfer of riak_kv_vnode from '[email protected]' > 91343852333181432387730302044767688728495783936 to '[email protected]' > 91343852333181432387730302044767688728495783936 failed because of > error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1363,975886,897088}},{{ts,{1363,975886,897088}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,108,97,98,108,111,103,45,99,97,99,104,101,109,0,0,0,35,51,81,51,112,78,86,88,79,84,106,57,66,110,90,49,97,65,103,122,45,114,83,67,99,110,53,52,64,50,53,48,120,51,54,48,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,6,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,76,105,110,107,115,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,105,97,109,97,97,97,103,97,101,97,47,97,106,97,112,97,101,97,103,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,52,97,112,97,110,97,66,97,87,97,104,97,50,97,72,97,48,97,48,97,101,97,84,97,75,97,102,97,50,97,100,97,98,97,75,97,100,97,72,97,88,97,90,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,83,98,0,14,228,14,98,0,13,175,214,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,11,88,45,82,105,97,107,45,77,101,116,97,106,106,109,0,0,113,109,255,216,255,224,0,16,74,70,73,70,0,1,1,1,0,100,0,100,0,0,255,219,0,67,0,2,1,1,1,1,1,2,1,1,1,2,2,2,2,2,4,3,2,2,2,2,5,4,4,3,4,6,5,6,6,6,5,6,6,6,7,9,8,6,7,9,7,6,6,8,11,8,9,10,10,10,10,10,6,8,11,12,11,10,12,9,10,10,10,255,219,0,67,1,2,2,2,2,2,2,5,3,3,5,10,7,6,7,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,255,192,0,17,8,1,104,0,250,3,1,34,0,2,17,1,3,17,1,255,196,0,30,0,0,2,2,3,1,1,1,1,0,0,0,0,0,0,0,0,4,5,3,6,2,7,8,0,1,9,10,255,196,0,84,16,0,1,3,...>>}],...},...]},...}}} > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] > 2013-03-25 10:25:19.968 [error] <0.5968.584> CRASH REPORT Process > <0.5968.584> with 0 neighbours exited with reason: no function clause > matching riak_core_pb:encode({ts,{1363,975886,897088}}, > {{ts,{1363,975886,897088}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in gen_server:terminate/6 line 747 > 2013-03-25 10:25:19.972 [error] <0.27190.4> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > {riak_core_vnode_worker,start_link,undefined} at <0.5968.584> exit with > reason no function clause matching > riak_core_pb:encode({ts,{1363,975886,897088}}, > {{ts,{1363,975886,897088}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in context child_terminated > 2013-03-25 10:25:27.003 [error] > <0.23033.591>@riak_core_handoff_sender:start_fold:226 hinted_handoff > transfer of riak_kv_vnode from '[email protected]' > 228359630832953580969325755111919221821239459840 to '[email protected]' > 228359630832953580969325755111919221821239459840 failed because of > error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1363,976020,73012}},{{ts,{1363,976020,73012}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,108,97,98,108,111,103,45,99,97,99,104,101,109,0,0,0,35,51,108,97,88,112,68,87,54,51,49,99,49,51,97,48,50,103,106,115,70,50,56,72,109,54,74,77,64,49,53,48,120,50,54,54,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,6,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,76,105,110,107,115,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,105,97,109,97,97,97,103,97,101,97,47,97,106,97,112,97,101,97,103,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,54,97,56,97,77,97,109,97,122,97,115,97,83,97,76,97,114,97,105,97,98,97,82,97,71,97,71,97,68,97,120,97,117,97,75,97,107,97,57,97,56,97,104,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,83,98,0,14,228,148,98,0,1,28,212,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,11,88,45,82,105,97,107,45,77,101,116,97,106,106,109,0,0,24,216,255,216,255,224,0,16,74,70,73,70,0,1,1,1,0,72,0,72,0,0,255,226,2,28,73,67,67,95,80,82,79,70,73,76,69,0,1,1,0,0,2,12,108,99,109,115,2,16,0,0,109,110,116,114,82,71,66,32,88,89,90,32,7,220,0,1,0,25,0,3,0,41,0,57,97,99,115,112,65,80,80,76,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,246,214,0,1,0,0,0,0,211,45,108,99,109,115,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,100,101,115,99,0,0,0,252,0,0,0,94,99,112,114,116,0,0,1,92,0,0,0,11,119,116,112,116,0,0,1,104,0,0,0,20,98,107,112,116,0,0,1,124,...>>}],...},...]},...}}} > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] > 2013-03-25 10:25:27.003 [error] <0.7134.584> gen_server <0.7134.584> > terminated with reason: no function clause matching > riak_core_pb:encode({ts,{1363,976020,73012}}, > {{ts,{1363,976020,73012}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 > 2013-03-25 10:25:27.018 [error] <0.7134.584> CRASH REPORT Process > <0.7134.584> with 0 neighbours exited with reason: no function clause > matching riak_core_pb:encode({ts,{1363,976020,73012}}, > {{ts,{1363,976020,73012}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in gen_server:terminate/6 line 747 > 2013-03-25 10:25:27.029 [error] <0.13194.5> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > {riak_core_vnode_worker,start_link,undefined} at <0.7134.584> exit with > reason no function clause matching > riak_core_pb:encode({ts,{1363,976020,73012}}, > {{ts,{1363,976020,73012}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in context child_terminated > 2013-03-25 10:25:28.767 [error] > <0.23312.591>@riak_core_handoff_sender:start_fold:226 hinted_handoff > transfer of riak_kv_vnode from '[email protected]' > 548063113999088594326381812268606132370974703616 to '[email protected]' > 548063113999088594326381812268606132370974703616 failed because of > error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1363,975886,899793}},{{ts,{1363,975886,899793}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,108,97,98,108,111,103,45,99,97,99,104,101,109,0,0,0,35,54,77,54,103,67,78,84,75,55,109,80,117,101,81,119,52,65,104,111,48,52,77,117,69,113,111,119,64,53,48,48,120,53,48,48,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,6,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,76,105,110,107,115,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,105,97,109,97,97,97,103,97,101,97,47,97,106,97,112,97,101,97,103,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,51,97,89,97,121,97,90,97,107,97,116,97,118,97,88,97,122,97,101,97,103,97,112,97,69,97,117,97,55,97,55,97,75,97,71,97,88,97,108,97,48,97,52,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,83,98,0,14,228,14,98,0,13,186,58,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,11,88,45,82,105,97,107,45,77,101,116,97,106,106,109,0,0,249,56,255,216,255,224,0,16,74,70,73,70,0,1,1,1,0,72,0,72,0,0,255,219,0,67,0,3,2,2,3,2,2,3,3,3,3,4,3,3,4,5,8,5,5,4,4,5,10,7,7,6,8,12,10,12,12,11,10,11,11,13,14,18,16,13,14,17,14,11,11,16,22,16,17,19,20,21,21,21,12,15,23,24,22,20,24,18,20,21,20,255,219,0,67,1,3,4,4,5,4,5,9,5,5,9,20,13,11,13,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,255,192,0,17,8,1,244,1,244,3,1,34,0,2,17,1,3,17,1,255,196,0,29,0,0,2,1,5,1,1,0,0,0,0,0,0,0,0,0,0,0,1,6,2,3,4,5,...>>}],...},...]},...}}} > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] > 2013-03-25 10:25:28.767 [error] <0.7652.584> gen_server <0.7652.584> > terminated with reason: no function clause matching > riak_core_pb:encode({ts,{1363,975886,899793}}, > {{ts,{1363,975886,899793}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 > 2013-03-25 10:25:28.770 [error] <0.7652.584> CRASH REPORT Process > <0.7652.584> with 0 neighbours exited with reason: no function clause > matching riak_core_pb:encode({ts,{1363,975886,899793}}, > {{ts,{1363,975886,899793}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in gen_server:terminate/6 line 747 > 2013-03-25 10:25:28.770 [error] <0.29412.4> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > {riak_core_vnode_worker,start_link,undefined} at <0.7652.584> exit with > reason no function clause matching > riak_core_pb:encode({ts,{1363,975886,899793}}, > {{ts,{1363,975886,899793}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in context child_terminated > > > $ sudo grep -F "[error]" /var/log/riak/console.log | tail -20 > 2013-03-25 05:44:48.666 [error] <0.9721.141> gen_fsm <0.9721.141> in state > ready terminated with reason: no case clause matching > {error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{state,9404519,9396330,9388137,10737418240,2598492,43200}},{<<"storage-kazeo">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98669>,"114179815416476790484662877555959610910619729920",[{data_root,"/data/riak/bitcask/storage-kazeo"},{read_write,true}],114179815416476790484662877555959610910619729920,"/data/riak/bitcask/storage-kazeo"}},{<<"storage-laprovence">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98622>,"114179815416476790484662...",...}},...],...}} > in riak_kv_vnode:do_diffobj_put/3 line 1154 > 2013-03-25 05:44:48.667 [error] <0.9721.141> CRASH REPORT Process > <0.9721.141> with 10 neighbours exited with reason: no case clause matching > {error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{state,9404519,9396330,9388137,10737418240,2598492,43200}},{<<"storage-kazeo">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98669>,"114179815416476790484662877555959610910619729920",[{data_root,"/data/riak/bitcask/storage-kazeo"},{read_write,true}],114179815416476790484662877555959610910619729920,"/data/riak/bitcask/storage-kazeo"}},{<<"storage-laprovence">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98622>,"114179815416476790484662...",...}},...],...}} > in riak_kv_vnode:do_diffobj_put/3 line 1154 in gen_fsm:terminate/7 line 611 > 2013-03-25 05:44:48.668 [error] <0.9722.141> Supervisor poolboy_sup had > child riak_core_vnode_worker started with > riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[114179815416476790484662877555959610910619729920,...]},...]) > at undefined exit with reason no case clause matching > {error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{state,9404519,9396330,9388137,10737418240,2598492,43200}},{<<"storage-kazeo">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98669>,"114179815416476790484662877555959610910619729920",[{data_root,"/data/riak/bitcask/storage-kazeo"},{read_write,true}],114179815416476790484662877555959610910619729920,"/data/riak/bitcask/storage-kazeo"}},{<<"storage-laprovence">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98622>,"114179815416476790484662...",...}},...],...}} > in riak_kv_vnode:do_diffobj_put/3 line 1154 in context shutdown_error > 2013-03-25 05:44:48.669 [error] <0.9722.141> gen_server <0.9722.141> > terminated with reason: no case clause matching > {error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{state,9404519,9396330,9388137,10737418240,2598492,43200}},{<<"storage-kazeo">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98669>,"114179815416476790484662877555959610910619729920",[{data_root,"/data/riak/bitcask/storage-kazeo"},{read_write,true}],114179815416476790484662877555959610910619729920,"/data/riak/bitcask/storage-kazeo"}},{<<"storage-laprovence">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98622>,"114179815416476790484662...",...}},...],...}} > in riak_kv_vnode:do_diffobj_put/3 line 1154 > 2013-03-25 05:44:48.670 [error] <0.9722.141> CRASH REPORT Process > <0.9722.141> with 0 neighbours exited with reason: no case clause matching > {error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{state,9404519,9396330,9388137,10737418240,2598492,43200}},{<<"storage-kazeo">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98669>,"114179815416476790484662877555959610910619729920",[{data_root,"/data/riak/bitcask/storage-kazeo"},{read_write,true}],114179815416476790484662877555959610910619729920,"/data/riak/bitcask/storage-kazeo"}},{<<"storage-laprovence">>,riak_kv_bitcask_backend,{state,#Ref<0.0.202.98622>,"114179815416476790484662...",...}},...],...}} > in riak_kv_vnode:do_diffobj_put/3 line 1154 in gen_server:terminate/6 line > 747 > 2013-03-25 05:47:02.413 [error] <0.2635.98> gen_server <0.2635.98> > terminated with reason: no function clause matching > riak_core_pb:encode({ts,{1364,132523,595762}}, > {{ts,{1364,132523,595762}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 > 2013-03-25 05:47:02.413 [error] > <0.21086.140>@riak_core_handoff_sender:start_fold:226 ownership_handoff > transfer of riak_kv_vnode from '[email protected]' > 924856504873462002925769308203272848376019812352 to '[email protected]' > 924856504873462002925769308203272848376019812352 failed because of > error:{badmatch,{error,{worker_crash,{function_clause,[{riak_core_pb,encode,[{ts,{1364,132523,595762}},{{ts,{1364,132523,595762}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,108,97,98,108,111,103,45,99,97,99,104,101,109,0,0,0,35,45,45,116,65,57,116,67,100,108,119,82,114,95,104,101,110,115,78,100,77,68,104,99,45,74,114,77,64,53,48,48,120,51,55,53,108,0,0,0,1,104,3,100,0,9,114,95,99,111,110,116,101,110,116,104,9,100,0,4,100,105,99,116,97,6,97,16,97,16,97,8,97,80,97,48,104,16,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,106,104,1,104,16,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,76,105,110,107,115,106,106,106,106,106,106,106,106,106,108,0,0,0,2,108,0,0,0,11,109,0,0,0,12,99,111,110,116,101,110,116,45,116,121,112,101,97,105,97,109,97,97,97,103,97,101,97,47,97,106,97,112,97,101,97,103,106,108,0,0,0,23,109,0,0,0,11,88,45,82,105,97,107,45,86,84,97,103,97,52,97,110,97,48,97,106,97,122,97,83,97,74,97,76,97,81,97,68,97,72,97,100,97,52,97,100,97,99,97,72,97,57,97,54,97,104,97,120,97,76,97,65,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,5,105,110,100,101,120,106,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,20,88,45,82,105,97,107,45,76,97,115,116,45,77,111,100,105,102,105,101,100,104,3,98,0,0,5,84,98,0,2,5,171,98,0,9,22,117,106,106,108,0,0,0,1,108,0,0,0,1,109,0,0,0,11,88,45,82,105,97,107,45,77,101,116,97,106,106,109,0,1,34,141,255,216,255,224,0,16,74,70,73,70,0,1,1,1,0,72,0,72,0,0,255,219,0,67,0,2,1,1,1,1,1,2,1,1,1,2,2,2,2,2,4,3,2,2,2,2,5,4,4,3,4,6,5,6,6,6,5,6,6,6,7,9,8,6,7,9,7,6,6,8,11,8,9,10,10,10,10,10,6,8,11,12,11,10,12,9,10,10,10,255,219,0,67,1,2,2,2,2,2,2,5,3,3,5,10,7,6,7,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,255,192,0,17,8,1,119,1,244,3,1,34,0,2,17,1,3,17,1,255,196,0,30,0,0,1,4,3,1,1,1,0,0,0,0,0,0,0,0,0,6,4,5,7,8,0,3,9,2,1,10,255,196,0,76,16,0,1,3,...>>}],...},...]},...}}} > [{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,170}]}] > 2013-03-25 05:47:02.414 [error] <0.2635.98> CRASH REPORT Process <0.2635.98> > with 0 neighbours exited with reason: no function clause matching > riak_core_pb:encode({ts,{1364,132523,595762}}, > {{ts,{1364,132523,595762}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in gen_server:terminate/6 line 747 > 2013-03-25 05:47:02.416 [error] <0.7449.0> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > {riak_core_vnode_worker,start_link,undefined} at <0.2635.98> exit with > reason no function clause matching > riak_core_pb:encode({ts,{1364,132523,595762}}, > {{ts,{1364,132523,595762}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,107,...>>}) > line 40 in context child_terminated > 2013-03-25 05:49:22.321 [error] <0.17183.142> Bad datafile entry, > discarding(3255/26661 bytes) > 2013-03-25 05:49:44.484 [error] <0.17183.142> Failed to merge > ["/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920",[{data_root,"/data/riak/bitcask/storage"},{read_write,true}],["/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/53.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/52.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/51.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/50.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/49.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/48.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/39.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/38.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/24.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/20.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/18.bitcask.data","/data/riak/bitcask/storage/114179815416476790484662877555959610910619729920/17.bitcask.data"]]: > {{badmatch,{error,enospc}},[{bitcask_fileops,write,4,[{file,"src/bitcask_fileops.erl"},{line,154}]},{bitcask,inner_merge_write,6,[{file,"src/bitcask.erl"},{line,1064}]},{bitcask_fileops,fold_loop,8,[{file,"src/bitcask_fileops.erl"},{line,344}]},{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,987}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,584}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,133}]}]} > 2013-03-25 05:49:59.144 [error] <0.16671.3> gen_fsm <0.16671.3> in state > active terminated with reason: no match of right hand value {error,enospc} > in bitcask_fileops:write/4 line 154 > 2013-03-25 05:49:59.150 [error] <0.16671.3> CRASH REPORT Process <0.16671.3> > with 1 neighbours exited with reason: no match of right hand value > {error,enospc} in bitcask_fileops:write/4 line 154 in gen_fsm:terminate/7 > line 611 > 2013-03-25 05:49:59.152 [error] <0.137.0> Supervisor riak_core_vnode_sup had > child undefined started with {riak_core_vnode,start_link,undefined} at > <0.16671.3> exit with reason no match of right hand value {error,enospc} in > bitcask_fileops:write/4 line 154 in context child_terminated > 2013-03-25 05:49:59.153 [error] <0.18047.3> gen_fsm <0.18047.3> in state > ready terminated with reason: no match of right hand value {error,enospc} in > bitcask_fileops:write/4 line 154 > 2013-03-25 05:49:59.153 [error] <0.18047.3> CRASH REPORT Process <0.18047.3> > with 10 neighbours exited with reason: no match of right hand value > {error,enospc} in bitcask_fileops:write/4 line 154 in gen_fsm:terminate/7 > line 611 > 2013-03-25 05:49:59.154 [error] <0.18048.3> Supervisor poolboy_sup had child > riak_core_vnode_worker started with > riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[1107544209539824867701229912292808225833011380224,...]},...]) > at undefined exit with reason no match of right hand value {error,enospc} in > bitcask_fileops:write/4 line 154 in context shutdown_error > 2013-03-25 05:49:59.156 [error] <0.18048.3> gen_server <0.18048.3> > terminated with reason: no match of right hand value {error,enospc} in > bitcask_fileops:write/4 line 154 > 2013-03-25 05:49:59.157 [error] <0.18048.3> CRASH REPORT Process <0.18048.3> > with 0 neighbours exited with reason: no match of right hand value > {error,enospc} in bitcask_fileops:write/4 line 154 in gen_server:terminate/6 > line 747 > 2013-03-25 10:29:05.078 [error] <0.32028.0> Bad datafile entry, > discarding(3255/26661 bytes) > > > > > 2013/3/22 Evan Vigil-McClanahan <[email protected]> >> >> Glad to have been able to help. >> >> On Fri, Mar 22, 2013 at 12:11 PM, Godefroy de Compreignac >> <[email protected]> wrote: >> > Thank you a lot for your help! >> > >> > I guess I forgot to restart one node after the configuration of a new >> > memory >> > backend. >> > Anyway, I upgraded all my nodes to Riak 1.3.1, and everything seems to >> > be >> > working great! >> > >> > Godefroy >> > >> > >> > 2013/3/22 Evan Vigil-McClanahan <[email protected]> >> >> >> >> OK, so there's a lot going on there. >> >> >> >> > 2013-03-22 12:02:18.719 [error] <0.16959.2526> gen_server >> >> > <0.16959.2526> >> >> > terminated with reason: no function clause matching >> >> > riak_core_pb:encode({ts,{1363,205559,674898}}, >> >> > >> >> > >> >> > {{ts,{1363,205559,674898}},<<131,104,7,100,0,8,114,95,111,98,106,101,99,116,109,0,0,0,14,101,...>>}) >> >> > line 40 >> >> >> >> This one is usually because you have different settings on the memory >> >> backend on two nodes (one with a ttl, the other without). Typically >> >> the solution here is to make sure that the settings are the same >> >> across the cluster and then do a rolling restart of the cluster. >> >> >> >> >> >> > 2013-03-22 11:28:53.379 [error] <0.28559.1189> gen_fsm <0.28559.1189> >> >> > in >> >> > state active terminated with reason: no case clause matching >> >> > >> >> > >> >> > {error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{state,57458783,57450590,57442397,10737418240,31415183,43200}},{<<"storage-kazeo">>,riak_kv_bitcask_backend,{state,#Ref<0.0.461.178788>,"79925870791533753339264014289171727637433810944",[{data_root,"/data/riak/bitcask/storage-kazeo"},{read_write,true}],79925870791533753339264014289171727637433810944,"/data/riak/bitcask/storage-kazeo"}},{<<"storage-laprovence">>,riak_kv_bitcask_backend,{state,#Ref<0.0.461.178779>,"79925870791533753339...",...}},...],...}} >> >> > in riak_kv_vnode:do_diffobj_put/3 line 1059 >> >> > 2013-03-22 11:28:53.379 [error] >> >> > <0.15853.1198>@riak_core_handoff_receiver:handle_info:80 Handoff >> >> > receiver >> >> > for partition 79925870791533753339264014289171727637433810944 exited >> >> > abnormally after processing 5337 objects: >> >> > >> >> > >> >> > {{{case_clause,{error,bad_crc,{state,[{<<"cache">>,riak_kv_memory_backend,{ >> >> >> >> This one looks like it might be because of some bad data left over >> >> from your filling up of the disk. It's possible that upgrading to 1.3 >> >> will help here, it has better handling of corrupted data in bitcask. >> >> >> >> If that isn't an option right now, let me know, there may be some >> >> other ways to get the corrupted values out of the cluster. >> > >> > > > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
