Hello! I have 8x cluster of riak+riak-cs on debian. Config templates attached Versions: ii riak 1.4.8-1 amd64 Riak is a distributed data store ii riak-cs 1.4.5-1 amd64 Riak CS
Every riak-cs connect to local node. Between clients and riak-cs exist
frontend (Tengine version: Tengine/1.5.1 (nginx/1.2.9)), config
attached
Clients - s3cmd + some numbers of php (read-only)
When 1-3 clients wants write to riak-cs, write speed is near 3-4MB/sec.
If 30-40 clients wants write, write speed slow down to lower than 100kB/sec.
In riak-cs crash.log:
2014-04-02 03:52:11 =ERROR REPORT====
webmachine error:
path="/buckets/test/objects/win.img/uploads/PuqEyz0BRCCk6rDxtH7tRQ=="
{error,{error,{badmatch,{error,closed}},[{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]}}
[{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]
After this event s3cmd makes throttling to slower speed:
$ s3cmd put win.img s3://test/
win.img -> s3://test/win.img [part 1 of 1366, 15MB]
184320 of 15728640 1% in 0s 2.16 MB/s failed
WARNING: Upload failed:
/win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
win.img -> s3://test/win.img [part 1 of 1366, 15MB]
13799424 of 15728640 87% in 2s 5.18 MB/s failed
WARNING: Upload failed:
/win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
win.img -> s3://test/win.img [part 1 of 1366, 15MB]
167936 of 15728640 1% in 0s 249.46 kB/s failed
WARNING: Upload failed:
/win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
win.img -> s3://test/win.img [part 1 of 1366, 15MB]
6225920 of 15728640 39% in 76s 79.51 kB/s failed
WARNING: Upload failed:
/win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
win.img -> s3://test/win.img [part 1 of 1366, 15MB]
15728640 of 15728640 100% in 962s 15.96 kB/s done
I think, even on 1Gbit network betwen nodes, write speed should be
higher, but i don't understand where the bottleneck.
--
Stanislav
app.config.template
Description: Binary data
vm.args.template
Description: Binary data
riak-cs-nginx
Description: Binary data
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
