[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-04-04 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955849#comment-15955849
 ] 

ASF subversion and git services commented on COUCHDB-3302:
--

Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch 
refs/heads/COUCHDB-3287-pluggable-storage-engines from [~vatamane]
[ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

Prevent attachment upload from timing out during update_docs fabric call

Currently if an attachment was large enough or the connection was slow enough
such that it took more than fabric.request_timeout = 6 milliseconds, the
fabric request would time out during attachment data transfer from coordinator
node to other nodes and the whole request would fail.

This was most evident when replicating database with large attachments.

The fix is to periodically send `attachment_chunk_received` to coordinator to
prevent the timeout.

COUCHDB-3302


> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-04-04 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955842#comment-15955842
 ] 

ASF subversion and git services commented on COUCHDB-3302:
--

Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch 
refs/heads/COUCHDB-3288-remove-public-db-record from [~vatamane]
[ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

Prevent attachment upload from timing out during update_docs fabric call

Currently if an attachment was large enough or the connection was slow enough
such that it took more than fabric.request_timeout = 6 milliseconds, the
fabric request would time out during attachment data transfer from coordinator
node to other nodes and the whole request would fail.

This was most evident when replicating database with large attachments.

The fix is to periodically send `attachment_chunk_received` to coordinator to
prevent the timeout.

COUCHDB-3302


> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-21 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877308#comment-15877308
 ] 

ASF subversion and git services commented on COUCHDB-3302:
--

Commit 078d46c5a18864edd2f4bc16c89f12178ce13076 in couchdb's branch 
refs/heads/master from [~vatamane]
[ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=078d46c ]

Bump fabric : fix attachment uploading

COUCHDB-3302


> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>  [{att,<<"att_64">>,<<"application/octet-stream">>,
>   33193656,33193656,
>   <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
>   1,
>   {follows,<0.8720.0>,#Ref<0.0.1.23804>},
>   identity}],
>  <<"0dea87076009b928b191e0b456375c93">>,33194202}},
>[{response_format,binary},
> {inactivity_timeout,3},
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-21 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877305#comment-15877305
 ] 

ASF GitHub Bot commented on COUCHDB-3302:
-

Github user asfgit closed the pull request at:

https://github.com/apache/couchdb-fabric/pull/89


> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>  [{att,<<"att_64">>,<<"application/octet-stream">>,
>   33193656,33193656,
>   <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
>   1,
>   {follows,<0.8720.0>,#Ref<0.0.1.23804>},
>   identity}],
>  <<"0dea87076009b928b191e0b456375c93">>,33194202}},
>[{response_format,binary},
> {inactivity_timeout,3},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
>infinity}},
>   infinity]
> {noformat}
> Expected Behaviour:
> Replication eventually succeeds.
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-21 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877303#comment-15877303
 ] 

ASF subversion and git services commented on COUCHDB-3302:
--

Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch 
refs/heads/master from [~vatamane]
[ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

Prevent attachment upload from timing out during update_docs fabric call

Currently if an attachment was large enough or the connection was slow enough
such that it took more than fabric.request_timeout = 6 milliseconds, the
fabric request would time out during attachment data transfer from coordinator
node to other nodes and the whole request would fail.

This was most evident when replicating database with large attachments.

The fix is to periodically send `attachment_chunk_received` to coordinator to
prevent the timeout.

COUCHDB-3302


> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>  [{att,<<"att_64">>,<<"application/octet-stream">>,
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-21 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876925#comment-15876925
 ] 

ASF GitHub Bot commented on COUCHDB-3302:
-

GitHub user nickva opened a pull request:

https://github.com/apache/couchdb-fabric/pull/89

Prevent attachment upload from timing out during update_docs fabric call

Currently if an attachment was large enough or the connection was slow 
enough
such that it took more than fabric.request_timeout = 6 milliseconds, the
fabric request would time out during attachment data transfer from 
coordinator
node to other nodes and the whole request would fail.

This was most evident when replicating database with large attachments.

The fix is to periodically send `attachment_chunk_received` to coordinator 
to
prevent the timeout.

COUCHDB-3302

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloudant/couchdb-fabric couchdb-3302

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/couchdb-fabric/pull/89.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #89


commit 6e9074bc8778e00471d96191319ac67d6c78c05a
Author: Nick Vatamaniuc 
Date:   2017-02-21T22:46:57Z

Prevent attachment upload from timing out during update_docs fabric call

Currently if an attachment was large enough or the connection was slow 
enough
such that it took more than fabric.request_timeout = 6 milliseconds, the
fabric request would time out during attachment data transfer from 
coordinator
node to other nodes and the whole request would fail.

This was most evident when replicating database with large attachments.

The fix is to periodically send `attachment_chunk_received` to coordinator 
to
prevent the timeout.

COUCHDB-3302




> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-21 Thread Nick Vatamaniuc (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876250#comment-15876250
 ] 

Nick Vatamaniuc commented on COUCHDB-3302:
--

fabric_doc_attachments used when PUT-ing individual attachments, I was looking 
at a doc PUT with attachments in mulitpart-related format

> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>  [{att,<<"att_64">>,<<"application/octet-stream">>,
>   33193656,33193656,
>   <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
>   1,
>   {follows,<0.8720.0>,#Ref<0.0.1.23804>},
>   identity}],
>  <<"0dea87076009b928b191e0b456375c93">>,33194202}},
>[{response_format,binary},
> {inactivity_timeout,3},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
>infinity}},
>   infinity]
> {noformat}
> Expected Behaviour:
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-21 Thread Paul Joseph Davis (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876234#comment-15876234
 ] 

Paul Joseph Davis commented on COUCHDB-3302:


You logged that nothing sends that message? There's a call to send it in two 
places:

https://github.com/apache/couchdb-fabric/blob/master/src/fabric_doc_attachments.erl#L40
https://github.com/apache/couchdb-fabric/blob/master/src/fabric_doc_attachments.erl#L71

This code is gnarly enough that its quite possible there's something broken 
with those calls obviously but given how rexi:reply/1 should throw a badmatch 
if the rexi_from pdict entry isn't set I'm not sure what it'd be.

> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>  [{att,<<"att_64">>,<<"application/octet-stream">>,
>   33193656,33193656,
>   <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
>   1,
>   {follows,<0.8720.0>,#Ref<0.0.1.23804>},
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-20 Thread Nick Vatamaniuc (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875076#comment-15875076
 ] 

Nick Vatamaniuc commented on COUCHDB-3302:
--

Confirmed that setting fabric request_timeout to a higher value like 9 
helps with this.

At least {{./attach_large.py --size=10 --mintime=80}}

Successfully finishes while it doesn't with the default value of 6

> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>  [{att,<<"att_64">>,<<"application/octet-stream">>,
>   33193656,33193656,
>   <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
> 161,242>>,
>   1,
>   {follows,<0.8720.0>,#Ref<0.0.1.23804>},
>   identity}],
>  <<"0dea87076009b928b191e0b456375c93">>,33194202}},
>[{response_format,binary},
> {inactivity_timeout,3},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
>

[jira] [Commented] (COUCHDB-3302) Attachment replication over low bandwidth network connections

2017-02-20 Thread Nick Vatamaniuc (JIRA) 

[ 
https://issues.apache.org/jira/browse/COUCHDB-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875031#comment-15875031
 ] 

Nick Vatamaniuc commented on COUCHDB-3302:
--

>From investigating it seems  to be related to how long it takes for the 
>request to complete. 

I created a "paced" python multi-part sender which PUTs an attachment over  
period of time. It splits it into chunks then sends those with sleep in between.

Attached script as attach_large.py. Can run it with {[./attach_large.py 
--size=10 --mintime=80}} that will put an attachment of size 10 bytes 
over at least 80 seconds.

With that code I was able to get a 500 error and I get:

{code}
HTTP/1.1 500 Internal Server Error
Cache-Control: must-revalidate
Content-Length: 47
Content-Type: application/json
Date: Mon, 20 Feb 2017 19:27:30 GMT
Server: CouchDB/2.0.0 (Erlang OTP/18)
X-Couch-Request-ID: 80a6cfd301
X-CouchDB-Body-Time: 0

{"error":"unknown_error","reason":"undefined"}
{code}

> Attachment replication over low bandwidth network connections
> -
>
> Key: COUCHDB-3302
> URL: https://issues.apache.org/jira/browse/COUCHDB-3302
> Project: CouchDB
>  Issue Type: Bug
>  Components: Replication
>Reporter: Jan Lehnardt
> Attachments: attach_large.py, replication-failure.log, 
> replication-failure-target.log
>
>
> Setup:
> Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit 
> network connection (simulated locally with traffic shaping, see way below for 
> an example).
> {noformat}
> git clone https://github.com/apache/couchdb.git
> cd couchdb
> ./configure --disable-docs --disable-fauxton
> make release
> cd ..
> cp -r couchdb/rel/couchdb source
> cp -r couchdb/rel/couchdb target
> # set up local ini: chttpd / port: 5981 / 5983
> # set up vm.args: source@hostname.local / target@hostname.local
> # no admins
> Start both CouchDB in their own terminal windows: ./bin/couchdb
> # create all required databases, and our `t` test database
> curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
> # create 64MB attachments
> dd if=/dev/urandom of=att-64 bs=1024 count=65536
> # create doc on source
> curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: 
> application/octet-stream' -d @att-64
> # replicate to target
> curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json 
> -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
> {noformat}
> With the traffic shaping in place, the replication call doesn’t return, and 
> eventually CouchDB fails with:
> {noformat}
> [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator  
> Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
> [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0>  
> Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false; 
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
>   [{couch_att,'-foldl/4-fun-0-',3,
>[{file,"src/couch_att.erl"},{line,591}]},
>{couch_att,fold_streamed_data,4,
>[{file,"src/couch_att.erl"},{line,642}]},
>{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
>{couch_httpd_multipart,atts_to_mp,4,
>[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
>  {gen_server,call,
>  [<0.15778.0>,
>   {send_req,
>   {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false;,
>"127.0.0.1",5983,undefined,undefined,
>"/t/doc1?new_edits=false",http,ipv4_address},
>[{"Accept","application/json"},
> {"Content-Length",33194202},
> {"Content-Type",
>  "multipart/related; 
> boundary=\"0dea87076009b928b191e0b456375c93\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
>put,
>{#Fun,
> 
> {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
>