Stanislov, Could you also include your riak app.config and vm.args. It seems like you're load balancing Riak CS but I'm curious how the underlying Riak topology looks as well since that will likely be where the performance bottlenecks are uncovered.
On Wed, Apr 2, 2014 at 6:38 AM, Luke Bakken <[email protected]> wrote: > Hi Stanislav, > > In your Riak /etc/riak/app.config files, please use the following value: > > {pb_backlog, 256}, > > After changing this, you will have to restart Riak in a rolling fashion. > > Could you please run riak-debug on one node in your cluster and make the > generated archive available? (dropbox, for instance). Also, could you run tar > -czf /tmp/riak-cs-$(hostname).tgz and make the archive available? > > Thanks > -- > Luke Bakken > CSE > [email protected] > > > On Tue, Apr 1, 2014 at 9:32 PM, Stanislav Vlasov > <[email protected]>wrote: > >> Hello! >> >> I have 8x cluster of riak+riak-cs on debian. Config templates attached >> Versions: >> ii riak 1.4.8-1 >> amd64 Riak is a distributed data store >> ii riak-cs 1.4.5-1 >> amd64 Riak CS >> >> Every riak-cs connect to local node. Between clients and riak-cs exist >> frontend (Tengine version: Tengine/1.5.1 (nginx/1.2.9)), config >> attached >> Clients - s3cmd + some numbers of php (read-only) >> >> When 1-3 clients wants write to riak-cs, write speed is near 3-4MB/sec. >> If 30-40 clients wants write, write speed slow down to lower than >> 100kB/sec. >> >> In riak-cs crash.log: >> >> 2014-04-02 03:52:11 =ERROR REPORT==== >> webmachine error: >> path="/buckets/test/objects/win.img/uploads/PuqEyz0BRCCk6rDxtH7tRQ==" >> >> {error,{error,{badmatch,{error,closed}},[{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]}} >> >> [{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}] >> >> After this event s3cmd makes throttling to slower speed: >> >> $ s3cmd put win.img s3://test/ >> win.img -> s3://test/win.img [part 1 of 1366, 15MB] >> 184320 of 15728640 1% in 0s 2.16 MB/s failed >> WARNING: Upload failed: >> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.00) >> WARNING: Waiting 3 sec... >> win.img -> s3://test/win.img [part 1 of 1366, 15MB] >> 13799424 of 15728640 87% in 2s 5.18 MB/s failed >> WARNING: Upload failed: >> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.01) >> WARNING: Waiting 6 sec... >> win.img -> s3://test/win.img [part 1 of 1366, 15MB] >> 167936 of 15728640 1% in 0s 249.46 kB/s failed >> WARNING: Upload failed: >> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.05) >> WARNING: Waiting 9 sec... >> win.img -> s3://test/win.img [part 1 of 1366, 15MB] >> 6225920 of 15728640 39% in 76s 79.51 kB/s failed >> WARNING: Upload failed: >> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.25) >> WARNING: Waiting 12 sec... >> win.img -> s3://test/win.img [part 1 of 1366, 15MB] >> 15728640 of 15728640 100% in 962s 15.96 kB/s done >> >> I think, even on 1Gbit network betwen nodes, write speed should be >> higher, but i don't understand where the bottleneck. >> >> -- >> Stanislav >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
