empeje commented on issue #1475: Replication Problem when using SSL with Nginx frontend and native SSL CouchDB URL: https://github.com/apache/couchdb/issues/1475#issuecomment-409654163 Thanks for your answer @nickva. So many things to address here. >Regarding the 3Gb files, are those sizes for single document bodies? Are those attachments? Individual or total? Usually storing large individual documents or attachments in that range is considered an anti-pattern for CouchDB. Is there any way to break those up into smaller documents? It is mostly an attachment. We are distributing learning resource data via CouchDB. I know it is anti-pattern, but we take this risk because it is one of the simplest way to create federated system with low human resource in a social benefit organization like us. Our goal is at least able to replicate 1GB of attachment. >Is it a single replication or are there others (many) running at the same time? Our server may receives multiple replication request at a time. >Noticed that max http request size and max client body size are set to 1Gb and CouchDB's max document size is set to 4Gb. Perhaps adjust the request size in both Nginx and CouchDb to a large value. For max request size, you'd want it greater than the sum of all the document's revisions and the total size of its attachments. Thank you for catching this, but we also got a problem when we replicate data under 1 GB. What I say 2.7 GB here is basically a complete database consists of several record with its attachment. >Regarding Experiment 1, with 2.1.2 release, should apply some of those max size configs, as opposed to using the defaults. In that release max request size was 64Mb so the default values won't work, especially if you have individual doc revisions + attachment sizes that exceed that. Thats why we led to Experiment 2, we still got a problem when we increase the max request size. >Do you know if smaller sizes work, what are the limits that work (1Gb, 100Mb,...)? For sure we have success replicating 121 MB record with 120 MB portion in form of attachment. >Perhaps try replicating on a local network without SSL or Nginx as a debugging experiment... We are on the way to debug this, will update the result later. >Inspect the logs on source and target. Do you see 413 http errors, timeouts? Especially see if you can notice the part when errors start happening. A 413 error, either sent by CouchDB or Nginx, might indicate that some of the max size limits might be applied. We haven't notice the `413` error but good to know this, so I can debug further. >use_checkpoints = true is the default for the replicator, no need to set it explicitly. Thanks, good to know this. We're little bit desperate with this result so we try various way hoping some of them work. > There is also http://docs.couchdb.org/en/stable/config/replicator.html#replicator/connection_timeout setting, maybe adjusting that up a bit might help. Thanks > Checkpoints are usually not that expensive, just an update of a local document on target and source. Try not to delay it too much. I would reduce that to something closer to the default. I tried, but wonder if checkpoint is also working with attachment? *** Thanks for the support. We are currently experimenting with plain no SSL setup in local and with HAProxy, and also will try some of your suggestion. Hope some of them work and then I able to share here.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
