Hi Charles,
This is likely the result of heavy load on Riak; how much have you tuned
Riak & your OS? Please make sure to review our tuning documentation.
Thought it's not clear in our CS documentation, make sure you have the
leveldb backend properly tuned:
LevelDB backend:
http://docs.basho.com/riak/latest/ops/advanced/backends/leveldb/
You might want to try increasing the request_pool and bucket_list_pool
values in your cs app.config, making sure to keep the value of
pb_backlog in Riak greater than the sum of the two pool values. The
recommended pb_backlog value is 256.
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak-CS/
If you go to http://docs.basho.com/riak/latest/ you can find an entire
section on Tuning under "Riak for Operators" in the left navigation that
might help you tune your cluster.
Regards,
~ Zack ~
On Tue Jul 22 08:48:02 2014, Charles Bijon wrote:
Since 4 days we have an issue on our cluster and we don't know how
correct this problem.
We have this problem after migrating 38 old node to 45 new nodes cluster.
Someone have an idea about this ?
2014-07-22 14:41:11 =CRASH REPORT====
crasher:
initial call: mochiweb_acceptor:init/3
pid: <0.1957.2>
registered_name: []
exception exit:
{{normal,{gen_fsm,sync_send_event,[<0.1998.2>,get_next_chunk,infinity]}},[{gen_fsm,sync_send_event,3,[{file,"gen_fsm.erl"},{line,214}]},{riak_cs_wm_utils,streaming_get,4,[{file,"src/riak_cs_wm_utils.erl"},{line,272}]},{webmachine_decision_core,'-make_encoder_str
eam/3-fun-0-',3,[{file,"src/webmachine_decision_core.erl"},{line,667}]},{webmachine_request,send_stream_body_no_chunk,2,[{file,"src/webmachine_request.erl"},{line,334}]},{webmachine_request,send_response,3,[{file,"src/webmachine_request.erl"},{line,398}]},{webmachine_request,call,2
,[{file,"src/webmachine_request.erl"},{line,251}]},{webmachine_decision_core,wrcall,1,[{file,"src/webmachine_decision_core.erl"},{line,42}]},{webmachine_decision_core,finish_response,3,[{file,"src/webmachine_decision_core.erl"},{line,92}]}]}
ancestors: [object_web_mochiweb,riak_cs_sup,<0.141.0>]
.....
,{moss_bucket_v1,"les-publications-conde-nast-sa",created,"2013-11-05T15:46:29.000Z",{1383,666389,334639},undefined},{moss_bucket_v1,"les-quatre-chemins",created,"2014-03-10T22:30:54.000Z",{1394,490654,643513},undefined},{moss_bucket_v1,...},...],...},...},...}}]
trap_exit: false
status: running
heap_size: 317811
stack_size: 24
reductions: 472810
neighbours:
2014-07-22 14:41:43 =CRASH REPORT====
2014-07-22 14:45:31.290 [error]
<0.6203.2>@riak_cs_get_fsm:waiting_chunks:311 riak_cs_get_fsm: Cannot
get S3 <<"daily-news">>
<<"daily-news/bw/2014-07-22/release/x_ml_pdf/merged/download.complete">>
block# {<<0,46,70,239,21,65,71,20,134,105,153,120,80,55,165,110>>,0}:
{error,notfound}
2014-07-22 14:45:31.292 [error] <0.6182.2> CRASH REPORT Process
<0.6182.2> with 0 neighbours exited with reason:
{normal,{gen_fsm,sync_send_event,[<0.6203.2>,get_next_chunk,infinity]}} in
gen_fsm:sync_send_event/3 line 214
2014-07-22 14:45:37.557 [error]
<0.6270.2>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
record for s3 failed. Reason: no_user_key
2014-07-22 14:45:42.327 [error]
<0.6280.2>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
record for s3 failed. Reason: no_user_key
2014-07-22 14:45:43.948 [error]
<0.6367.2>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
record for s3 failed. Reason: no_user_key
2014-07-22 14:45:43.971 [error]
<0.6395.2>@riak_cs_get_fsm:waiting_chunks:311 riak_cs_get_fsm: Cannot
get S3 <<"groupe-nice-matin">>
<<"corse-matin/corse-matin/2014-07-22/release/reader/pages/jpeg/ld/0016.d04e5b20-706e-4f6c-a0fd-469447d5ac33.jpeg">>
block#
{<<5,216,166,27,75,67,78,254,180,245,135,148,87,243,135,231>>,0}:
{error,notfound}
2014-07-22 14:45:43.973 [error] <0.6367.2> CRASH REPORT Process
<0.6367.2> with 0 neighbours exited with reason:
{normal,{gen_fsm,sync_send_event,[<0.6395.2>,get_next_chunk,infinity]}} in
gen_fsm:sync_send_event/3 line 214
Regards,
Charles
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
