[
https://issues.apache.org/jira/browse/COUCHDB-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848556#action_12848556
]
Jan Lehnardt commented on COUCHDB-679:
--------------------------------------
Failing in trunk.
I'm on Erlang R13B01 (erts-5.7.2) [source] [64-bit] [smp:2:2] [rq:2]
[async-threads:0] [kernel-poll:false]
This is my log:
> ./test/etap/run test/etap/114-replication-big-attachments.t
# Current time local 2010-03-22 23:06:06
# Using etap version "0.3.4"
1..26
Apache CouchDB 0.0.0 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.1.0>] Apache CouchDB has started on http://127.0.0.1:5984/
[info] [<0.57.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_a/doc_2097152/att_2097152.dat 201
ok 1 - Created attachment based on file
/Users/jan/Work/couchdb-git/tmp/att_2097152.dat
[info] [<0.85.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_a/ 200
[info] [<0.88.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_a/ 200
[info] [<0.90.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/_local%2Ff3f357dd7528fc1983437707dd97780e 404
[info] [<0.92.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/_changes?style=all_docs&heartbeat=10000&since=0&feed=normal
200
[info] [<0.95.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/doc_2097152?open_revs=["1-bda0d041cdceb8ee3c97564c569cea0d"]&revs=true&latest=true&att_gzip_length=true
200
[info] [<0.75.0>] starting new replication "f3f357dd7528fc1983437707dd97780e"
at <0.86.0>
[info] [<0.102.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/doc_2097152/att_2097152.dat?rev=1-bda0d041cdceb8ee3c97564c569cea0d
200
[info] [<0.106.0>] 127.0.0.1 - - 'POST'
/couch_test_rep_big_att_a/_ensure_full_commit?seq=1 201
[info] [<0.86.0>] recording a checkpoint for
http://127.0.0.1:5984/couch_test_rep_big_att_a/ -> couch_test_rep_big_att_b at
source update_seq 1
[info] [<0.111.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_a/_local%2Ff3f357dd7528fc1983437707dd97780e 201
[info] [<0.75.0>] 127.0.0.1 - - 'POST' /_replicate 200
ok 2 - Pull replication successfully triggered
ok 3 - Pull replication completed with success
[info] [<0.113.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_b/doc_2097152/att_2097152.dat 200
ok 4 - Request for streaming attachment successful
ok 5 - Saved attachment to file
/Users/jan/Work/couchdb-git/tmp/att_2097152.dat.copy
ok 6 - Downloaded attachment has correct size
ok 7 - Downloaded attachment has valid content
[info] [<0.127.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_b/ 200
[info] [<0.57.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_b/ 200
[info] [<0.57.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_b/_local%2F0ecaeff6b697d5aa09320a6b28506b87 404
[info] [<0.115.0>] starting new replication "0ecaeff6b697d5aa09320a6b28506b87"
at <0.128.0>
[info] [<0.57.0>] 127.0.0.1 - - 'POST' /couch_test_rep_big_att_b/_missing_revs
200
[info] [<0.57.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_b/doc_2097152?new_edits=false 201
[info] [<0.75.0>] 127.0.0.1 - - 'POST'
/couch_test_rep_big_att_b/_ensure_full_commit 201
[info] [<0.128.0>] recording a checkpoint for couch_test_rep_big_att_a ->
http://127.0.0.1:5984/couch_test_rep_big_att_b/ at source update_seq 1
[info] [<0.130.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_b/_local%2F0ecaeff6b697d5aa09320a6b28506b87 201
[info] [<0.115.0>] 127.0.0.1 - - 'POST' /_replicate 200
ok 8 - Push replication successfully triggered
ok 9 - Push replication completed with success
[info] [<0.145.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_b/doc_2097152/att_2097152.dat 200
ok 10 - Request for streaming attachment successful
ok 11 - Saved attachment to file
/Users/jan/Work/couchdb-git/tmp/att_2097152.dat.copy
not ok 12 - Downloaded attachment has correct size
---
description: "Downloaded attachment has correct size"
found: 2095939
wanted: 2097152
...
not ok 13 - Downloaded attachment has valid content
---
description: "Downloaded attachment has valid content"
found: false
wanted: true
...
[info] [<0.148.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_a/doc_4194304/att_4194304.dat 201
ok 14 - Created attachment based on file
/Users/jan/Work/couchdb-git/tmp/att_4194304.dat
[info] [<0.173.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_a/ 200
[info] [<0.85.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_a/ 200
[info] [<0.85.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/_local%2Ff3f357dd7528fc1983437707dd97780e 404
[info] [<0.176.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/_changes?style=all_docs&heartbeat=10000&since=0&feed=normal
200
[info] [<0.162.0>] starting replication "f3f357dd7528fc1983437707dd97780e" at
<0.174.0>
[info] [<0.85.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/doc_4194304?open_revs=["1-a917d129b41433090d9f164918045a41"]&revs=true&latest=true&att_gzip_length=true
200
[info] [<0.85.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_a/doc_4194304/att_4194304.dat?rev=1-a917d129b41433090d9f164918045a41
200
[info] [<0.85.0>] 127.0.0.1 - - 'POST'
/couch_test_rep_big_att_a/_ensure_full_commit?seq=1 201
[info] [<0.174.0>] recording a checkpoint for
http://127.0.0.1:5984/couch_test_rep_big_att_a/ -> couch_test_rep_big_att_b at
source update_seq 1
[info] [<0.88.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_a/_local%2Ff3f357dd7528fc1983437707dd97780e 201
[info] [<0.162.0>] 127.0.0.1 - - 'POST' /_replicate 200
ok 15 - Pull replication successfully triggered
ok 16 - Pull replication completed with success
[info] [<0.179.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_b/doc_4194304/att_4194304.dat 200
ok 17 - Request for streaming attachment successful
ok 18 - Saved attachment to file
/Users/jan/Work/couchdb-git/tmp/att_4194304.dat.copy
ok 19 - Downloaded attachment has correct size
ok 20 - Downloaded attachment has valid content
[info] [<0.205.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_b/ 200
[info] [<0.88.0>] 127.0.0.1 - - 'GET' /couch_test_rep_big_att_b/ 200
[info] [<0.88.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_b/_local%2F0ecaeff6b697d5aa09320a6b28506b87 404
[info] [<0.192.0>] starting replication "0ecaeff6b697d5aa09320a6b28506b87" at
<0.206.0>
[info] [<0.88.0>] 127.0.0.1 - - 'POST' /couch_test_rep_big_att_b/_missing_revs
200
[info] [<0.88.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_b/doc_4194304?new_edits=false 201
[info] [<0.90.0>] 127.0.0.1 - - 'POST'
/couch_test_rep_big_att_b/_ensure_full_commit 201
[info] [<0.206.0>] recording a checkpoint for couch_test_rep_big_att_a ->
http://127.0.0.1:5984/couch_test_rep_big_att_b/ at source update_seq 1
[info] [<0.95.0>] 127.0.0.1 - - 'PUT'
/couch_test_rep_big_att_b/_local%2F0ecaeff6b697d5aa09320a6b28506b87 201
[info] [<0.192.0>] 127.0.0.1 - - 'POST' /_replicate 200
ok 21 - Push replication successfully triggered
ok 22 - Push replication completed with success
[info] [<0.208.0>] 127.0.0.1 - - 'GET'
/couch_test_rep_big_att_b/doc_4194304/att_4194304.dat 200
ok 23 - Request for streaming attachment successful
ok 24 - Saved attachment to file
/Users/jan/Work/couchdb-git/tmp/att_4194304.dat.copy
ok 25 - Downloaded attachment has correct size
not ok 26 - Downloaded attachment has valid content
---
description: "Downloaded attachment has valid content"
found: false
wanted: true
...
> Pull replication no longer works when a source doc has a large attachment
> -------------------------------------------------------------------------
>
> Key: COUCHDB-679
> URL: https://issues.apache.org/jira/browse/COUCHDB-679
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Environment: trunk and 0.11
> Reporter: Filipe Manana
> Priority: Blocker
> Fix For: 0.11
>
> Attachments: CouchDB-679-etap_test-trunk-2.patch,
> CouchDB-679-etap_test-trunk-3.patch, CouchDB-679-etap_test-trunk.patch
>
>
> After rev r916868 (fix for CouchDB-597), doing a pull replication of a doc
> that has a large attachment no longer works.
> The problem is in couch_rep_att.erl:
> convert_stub(#att{data=stub, name=Name} = Attachment,
> {#http_db{} = Db, Id, Rev}) ->
> {Pos, [RevId|_]} = Rev,
> Request = Db#http_db{
> resource = lists:flatten([couch_util:url_encode(Id), "/",
> couch_util:url_encode(Name)]),
> qs = [{rev, couch_doc:rev_to_str({Pos,RevId})}]
> },
> Ref = make_ref(),
> RcvFun = fun() ->
> Bin = attachment_receiver(Ref, Request),
> cleanup(),
> Bin
> end,
> Attachment#att{data=RcvFun}.
> The cleanup/0 function can not be called there, since when the attachment is
> received in multiple chunks, after receiving the first chunk the subsequent
> ones are silently discarded by cleanup/0:
> cleanup() ->
> receive
> {ibrowse_async_response, _, _} ->
> %% TODO maybe log, didn't expect to have data here
> cleanup();
> {ibrowse_async_response_end, _} ->
> cleanup();
> {ibrowse_async_headers, _, _, _} ->
> cleanup()
> after 0 ->
> erase(),
> ok
> end.
> If you look into couch_db.erl, you'll see that the attachment receiver
> function maybe called multiple times:
> flush_att(Fd, #att{data=Fun,att_len=AttLen}=Att) when is_function(Fun) ->
> with_stream(Fd, Att, fun(OutputStream) ->
> write_streamed_attachment(OutputStream, Fun, AttLen)
> end).
> write_streamed_attachment(_Stream, _F, 0) ->
> ok;
> write_streamed_attachment(Stream, F, LenLeft) ->
> Bin = F(),
> ok = couch_stream:write(Stream, Bin),
> write_streamed_attachment(Stream, F, LenLeft - size(Bin)).
> This is serious :(
> However, removing that call to cleanup/0 may cause the return of CouchDB-597,
> therefore I don't supply a patch here.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
