[
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876925#comment-15876925
]
ASF GitHub Bot commented on COUCHDB-3302:
-----------------------------------------
GitHub user nickva opened a pull request:
https://github.com/apache/couchdb-fabric/pull/89
Prevent attachment upload from timing out during update_docs fabric call
Currently if an attachment was large enough or the connection was slow
enough
such that it took more than fabric.request_timeout = 60000 milliseconds, the
fabric request would time out during attachment data transfer from
coordinator
node to other nodes and the whole request would fail.
This was most evident when replicating database with large attachments.
The fix is to periodically send `attachment_chunk_received` to coordinator
to
prevent the timeout.
COUCHDB-3302
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloudant/couchdb-fabric couchdb-3302
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/couchdb-fabric/pull/89.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #89
----
commit 6e9074bc8778e00471d96191319ac67d6c78c05a
Author: Nick Vatamaniuc <[email protected]>
Date: 2017-02-21T22:46:57Z
Prevent attachment upload from timing out during update_docs fabric call
Currently if an attachment was large enough or the connection was slow
enough
such that it took more than fabric.request_timeout = 60000 milliseconds, the
fabric request would time out during attachment data transfer from
coordinator
node to other nodes and the whole request would fail.
This was most evident when replicating database with large attachments.
The fix is to periodically send `attachment_chunk_received` to coordinator
to
prevent the timeout.
COUCHDB-3302
----
> Attachment replication over low bandwidth network connections
> -------------------------------------------------------------
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log,
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit
> network connection (simulated locally with traffic shaping, see way below for
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: [email protected] / [email protected]
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type:
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z [email protected] emulator --------
> Error in process <0.15811.0> on node '[email protected]' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z [email protected] <0.8721.0> --------
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false";
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
> [{couch_att,'-foldl/4-fun-0-',3,
> [{file,"src/couch_att.erl"},{line,591}]},
> {couch_att,fold_streamed_data,4,
> [{file,"src/couch_att.erl"},{line,642}]},
> {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
> {couch_httpd_multipart,atts_to_mp,4,
> [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
> {gen_server,call,
> [<0.15778.0>,
> {send_req,
> {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false";,
> "127.0.0.1",5983,undefined,undefined,
> "/t/doc1?new_edits=false",http,ipv4_address},
> [{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
> "multipart/related;
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
> put,
> {#Fun<couch_replicator_api_wrap.11.59841038>,
>
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
> [{att,<<"att_64">>,<<"application/octet-stream">>,
> 33193656,33193656,
> <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
> 1,
> {follows,<0.8720.0>,#Ref<0.0.1.23804>},
> identity}],
> <<"0dea87076009b928b191e0b456375c93">>,33194202}},
> [{response_format,binary},
> {inactivity_timeout,30000},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
> infinity}},
> infinity]}}}}
> {noformat}
> Expected Behaviour:
> Replication eventually succeeds.
> Appendix:
> Set up Traffic Shaping on a Mac:
> {noformat}
> (cat /etc/pf.conf && echo "dummynet-anchor \"reptest\"" && echo "anchor
> \"reptest\"") | sudo pfctl -f -
> echo "dummynet in quick proto tcp from any to any port 5983 pipe 3" | sudo
> pfctl -a reptest -f -
> echo "dummynet out quick proto tcp from any to any port 5983 pipe 4" | sudo
> pfctl -a reptest -f -
> sudo dnctl pipe 3 config bw 2Mbit/s
> sudo dnctl pipe 4 config bw 2Mbit/s
> sudo pfctl -E
> {noformat}
> Reset with:
> {noformat}
> sudo pfctl -f /etc/pf.conf
> sudo dnctl flush
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
