Hi Stanislav,
Could you configure your clients to connect directly to Riak CS instead of
through your proxy? A colleague of mine suggested that the
{error,closed}and 104 error code in s3cmd appear to be caused by the
connection between
the client and Riak CS being closed or terminated unexpectedly.
I would still recommend keeping {pb_backlog, 256} as that will allow for
more simultaneous connections to be handled going forward.
--
Luke Bakken
CSE
[email protected]
On Tue, Apr 1, 2014 at 9:32 PM, Stanislav Vlasov <[email protected]>wrote:
> Hello!
>
> I have 8x cluster of riak+riak-cs on debian. Config templates attached
> Versions:
> ii riak 1.4.8-1
> amd64 Riak is a distributed data store
> ii riak-cs 1.4.5-1
> amd64 Riak CS
>
> Every riak-cs connect to local node. Between clients and riak-cs exist
> frontend (Tengine version: Tengine/1.5.1 (nginx/1.2.9)), config
> attached
> Clients - s3cmd + some numbers of php (read-only)
>
> When 1-3 clients wants write to riak-cs, write speed is near 3-4MB/sec.
> If 30-40 clients wants write, write speed slow down to lower than
> 100kB/sec.
>
> In riak-cs crash.log:
>
> 2014-04-02 03:52:11 =ERROR REPORT====
> webmachine error:
> path="/buckets/test/objects/win.img/uploads/PuqEyz0BRCCk6rDxtH7tRQ=="
>
> {error,{error,{badmatch,{error,closed}},[{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]}}
>
> [{webmachine_request,recv_unchunked_body,3,[{file,"src/webmachine_request.erl"},{line,474}]},{webmachine_request,call,2,[{file,"src/webmachine_request.erl"},{line,193}]},{wrq,stream_req_body,2,[{file,"src/wrq.erl"},{line,121}]},{riak_cs_wm_object_upload_part,accept_body,2,[{file,"src/riak_cs_wm_object_upload_part.erl"},{line,235}]},{riak_cs_wm_common,accept_body,2,[{file,"src/riak_cs_wm_common.erl"},{line,337}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]
>
> After this event s3cmd makes throttling to slower speed:
>
> $ s3cmd put win.img s3://test/
> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
> 184320 of 15728640 1% in 0s 2.16 MB/s failed
> WARNING: Upload failed:
> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
> Connection reset by peer)
> WARNING: Retrying on lower speed (throttle=0.00)
> WARNING: Waiting 3 sec...
> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
> 13799424 of 15728640 87% in 2s 5.18 MB/s failed
> WARNING: Upload failed:
> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
> Connection reset by peer)
> WARNING: Retrying on lower speed (throttle=0.01)
> WARNING: Waiting 6 sec...
> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
> 167936 of 15728640 1% in 0s 249.46 kB/s failed
> WARNING: Upload failed:
> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
> Connection reset by peer)
> WARNING: Retrying on lower speed (throttle=0.05)
> WARNING: Waiting 9 sec...
> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
> 6225920 of 15728640 39% in 76s 79.51 kB/s failed
> WARNING: Upload failed:
> /win.img?partNumber=1&uploadId=PuqEyz0BRCCk6rDxtH7tRQ== ([Errno 104]
> Connection reset by peer)
> WARNING: Retrying on lower speed (throttle=0.25)
> WARNING: Waiting 12 sec...
> win.img -> s3://test/win.img [part 1 of 1366, 15MB]
> 15728640 of 15728640 100% in 962s 15.96 kB/s done
>
> I think, even on 1Gbit network betwen nodes, write speed should be
> higher, but i don't understand where the bottleneck.
>
> --
> Stanislav
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
