[
https://issues.apache.org/jira/browse/COUCHDB-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Filipe Manana updated COUCHDB-639:
----------------------------------
Attachment: rep-att-comp-and-multipart-trunk-2.patch
Ok, now I'm no longer using a replication specific URI API. Definitely, it was
a bad idea.
For multipart docs, I simply use now a PUT /somedb/docId?new_edits=false. This
is an API that exists already. Dunno why, but previously I was associating
new_edits to _bulk_docs only.
So, for docs without attachments, I upload them to the remote target DB using
_bulk_docs, exactly like before. For docs with attachments, I upload them using
PUT /somedb/docId?new_edits=false and sending the doc as a multipart stream.
Simple enough imho.
> Make replication profit of attachment compression and improve push
> replication for large attachments
> ----------------------------------------------------------------------------------------------------
>
> Key: COUCHDB-639
> URL: https://issues.apache.org/jira/browse/COUCHDB-639
> Project: CouchDB
> Issue Type: Improvement
> Components: Replication
> Affects Versions: 0.11
> Environment: trunk
> Reporter: Filipe Manana
> Attachments: rep-att-comp-and-multipart-trunk-2.patch,
> rep-att-comp-and-multipart-trunk.patch
>
>
> At the moment, for compressed attachments, the replication uncompresses and
> then compresses again the attachments. Therefore, a waste of CPU time.
> The push replication is also not reliable for very large attachments (500mb +
> for example). Currently it sends the attachments in-lined in the respective
> JSON doc. Not only this requires too much ram memory, it also wastes too much
> CPU time doing the base64 encoding of the attachment (and also a
> decompression if the attachment is compressed).
> The following patch (rep-att-comp-and-multipart-trunk*.patch) addresses both
> issues. Docs containing attachments are now streamed to the target remote DB
> using the multipart doc streaming feature provided by couch_doc.erl, and
> compressed attachments are not uncompressed and re-compressed during the
> replication
> JavaScript tests included.
> Previously doing a replication of a DB containing 2 docs with attachments of
> 100mb and 500mb caused the Erlang VM to consume near 1.2GB of ram memory in
> my system. With that patch applied, it uses about 130Mb of ram memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
