[GitHub] davisp commented on a change in pull request #1201: This fixes couch_bt_engine:fold_local_docs/4
davisp commented on a change in pull request #1201: This fixes
couch_bt_engine:fold_local_docs/4
URL: https://github.com/apache/couchdb/pull/1201#discussion_r172037448
##
File path: src/couch/src/couch_bt_engine.erl
##
@@ -464,7 +464,10 @@ fold_docs(St, UserFun, UserAcc, Options) ->
fold_local_docs(St, UserFun, UserAcc, Options) ->
-fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options).
+case fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options) of
+{ok, _Reds, FinalAcc} -> {ok, null, FinalAcc};
Review comment:
Good call. Its actually higher up when its expected to be null in fabric.
What happens is that depending on what shards have actual docs you get a
combination of null and 0 which leads to badarith exceptions when trying to do
`0 + null` basically.
This is definitely a bit of a sharp corner on PSE vs features that assume
legacy engine behavior. But for v1 its mostly a game of trying to define
existing behavior so that we can refine what's expected/acceptable for future
engines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tonysun83 commented on a change in pull request #1201: This fixes couch_bt_engine:fold_local_docs/4
tonysun83 commented on a change in pull request #1201: This fixes
couch_bt_engine:fold_local_docs/4
URL: https://github.com/apache/couchdb/pull/1201#discussion_r172035572
##
File path: src/couch/src/couch_bt_engine.erl
##
@@ -464,7 +464,10 @@ fold_docs(St, UserFun, UserAcc, Options) ->
fold_local_docs(St, UserFun, UserAcc, Options) ->
-fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options).
+case fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options) of
+{ok, _Reds, FinalAcc} -> {ok, null, FinalAcc};
Review comment:
In ```fold_docs_int```, when it's a ```local_tree``` we return ```{ok, 0,
FinalUserAcc};```. What's the reasoning behind changing it to 0?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tonysun83 commented on a change in pull request #1201: This fixes couch_bt_engine:fold_local_docs/4
tonysun83 commented on a change in pull request #1201: This fixes
couch_bt_engine:fold_local_docs/4
URL: https://github.com/apache/couchdb/pull/1201#discussion_r172035572
##
File path: src/couch/src/couch_bt_engine.erl
##
@@ -464,7 +464,10 @@ fold_docs(St, UserFun, UserAcc, Options) ->
fold_local_docs(St, UserFun, UserAcc, Options) ->
-fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options).
+case fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options) of
+{ok, _Reds, FinalAcc} -> {ok, null, FinalAcc};
Review comment:
In ```fold_docs_int```, when it's a ```local_tree``` we return ```{ok, 0,
FinalUserAcc};```. What's the reasoning behind changing it to ```null```?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tonysun83 commented on a change in pull request #1201: This fixes couch_bt_engine:fold_local_docs/4
tonysun83 commented on a change in pull request #1201: This fixes
couch_bt_engine:fold_local_docs/4
URL: https://github.com/apache/couchdb/pull/1201#discussion_r172035572
##
File path: src/couch/src/couch_bt_engine.erl
##
@@ -464,7 +464,10 @@ fold_docs(St, UserFun, UserAcc, Options) ->
fold_local_docs(St, UserFun, UserAcc, Options) ->
-fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options).
+case fold_docs_int(St, St#st.local_tree, UserFun, UserAcc, Options) of
+{ok, _Reds, FinalAcc} -> {ok, null, FinalAcc};
Review comment:
In fold_docs_int, when it's a ```local_tree``` we return ```{ok, 0,
FinalUserAcc};```. What's the reasoning behind changing it to 0?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
URL: https://github.com/apache/couchdb/issues/796#issuecomment-370187107
Hi,
I have the same issue with 60 dbs, running on a local computer, each db has
4 views, up to 2 millions docs for every DB. only the last db is being updated.
I am getting emfile error, and shard db error, and errors are changing
according to the query range.
Sometimes only one db returns error, sometimes multiple.
https://gist.github.com/adyshimony/d5b9fec95a7ec7544095cef398960c5c
I set up the config exactly as written here
http://docs.couchdb.org/en/stable/maintenance/performance.html#system-resource-limits
Checking ulimit -n returns 64000
Also set max_dbs_open is 15k
Also added
[couchdb]
update_lru_on_read = false
But when checking for service limits:
??beam.smp(1532)???couchjs(3546)???{couchjs}(3554)
???couchjs(3547)???{couchjs}(3555)
???couchjs(3548)???{couchjs}(3553)
cat /proc/3546/limits
I am getting Max open files 1024
For every process in the list, even that I configured this:
/etc/systemd/system/couchdb.d/override.conf
[Service]
LimitNOFILE=64000.
**OK solutions found, I am not delete this cause maybe it can help someone.**
Add
/etc/systemd/system/couchdb.service.d/limits.conf
[Service]
LimitNOFILE=64000
Fixed the issue with all the other things I mention above.
/etc/systemd/system/couchdb.d/override.conf is not enough if working at all.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
URL: https://github.com/apache/couchdb/issues/796#issuecomment-370187107
Hi,
I have the same issue with 60 dbs, running on a local computer, each db has
4 views, up to 2 millions docs for every DB. only the last db is being updated.
I am getting emfile error, and shard db error, and errors are changing
according to the query range.
Sometimes only one db returns error, sometimes multiple.
https://gist.github.com/adyshimony/d5b9fec95a7ec7544095cef398960c5c
I set up the config exactly as written here
http://docs.couchdb.org/en/stable/maintenance/performance.html#system-resource-limits
Checking ulimit -n returns 64000
Also set max_dbs_open is 15k
Also added
[couchdb]
update_lru_on_read = false
Checking for service limits:
??beam.smp(1532)???couchjs(3546)???{couchjs}(3554)
???couchjs(3547)???{couchjs}(3555)
???couchjs(3548)???{couchjs}(3553)
???couchjs(3549)???{couchjs}(3560)
???couchjs(3550)???{couchjs}(3552)
???couchjs(3551)???{couchjs}(3568)
???couchjs(3556)???{couchjs}(3566)
???couchjs(3557)???{couchjs}(3573)
???couchjs(3558)???{couchjs}(3570)
???couchjs(3559)???{couchjs}(3572)
???couchjs(3561)???{couchjs}(3575)
???couchjs(3562)???{couchjs}(3574)
???couchjs(3563)???{couchjs}(3571)
???couchjs(3564)???{couchjs}(3569)
???couchjs(3565)???{couchjs}(3576)
???couchjs(3567)???{couchjs}(3584)
???couchjs(3577)???{couchjs}(3582)
???couchjs(3578)???{couchjs}(3585)
???couchjs(3579)???{couchjs}(3586)
???couchjs(3580)???{couchjs}(3590)
???couchjs(3581)???{couchjs}(3588)
???couchjs(3583)???{couchjs}(3593)
???couchjs(3587)???{couchjs}(3595)
???couchjs(3589)???{couchjs}(3592)
???couchjs(3591)???{couchjs}(3599)
???couchjs(3594)???{couchjs}(3600)
???couchjs(3597)???{couchjs}(3601)
???couchjs(3598)???{couchjs}(3603)
???couchjs(3602)???{couchjs}(3606)
???couchjs(3604)???{couchjs}(3608)
???couchjs(3605)???{couchjs}(3610)
???couchjs(3607)???{couchjs}(3609)
???cpu_sup(1690)
???memsup(1689)
Like this
cat /proc/3546/limits
I am getting Max open files 1024
For every process in the list, even that I configure this:
/etc/systemd/system/couchdb.d/override.conf
[Service]
LimitNOFILE=64000.
As mentioned here.
Maybe this is the problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened URL: https://github.com/apache/couchdb/issues/796#issuecomment-370187107 Hi, I have the same issue with 60 dbs, running on a local computer, each db has 4 views, up to 2 millions docs for every DB. only the last db is being updated. I am getting emfile error, and shard db error, and errors are changing according to the query range. Sometimes only one db returns error, sometimes multiple. https://gist.github.com/adyshimony/d5b9fec95a7ec7544095cef398960c5c I set up the config exactly as written here http://docs.couchdb.org/en/stable/maintenance/performance.html#system-resource-limits Checking ulimit -n returns 64000 Also set max_dbs_open is 15k Also added [couchdb] update_lru_on_read = false Any clew can help me out here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened URL: https://github.com/apache/couchdb/issues/796#issuecomment-370187107 Hi, I have the same issue with 60 dbs, running on a local computer, each db has 4 views, up to 2 millions docs for every DB. I am getting emfile error, and shard db error, and errors are changing according to the query range. Sometimes only one db returns error, sometimes multiple. https://gist.github.com/adyshimony/d5b9fec95a7ec7544095cef398960c5c I set up the config exactly as written here http://docs.couchdb.org/en/stable/maintenance/performance.html#system-resource-limits Checking ulimit -n returns 64000 Also set max_dbs_open is 15k Also added [couchdb] update_lru_on_read = false Any clew can help me out here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened
adyshimony commented on issue #796: CouchDB Error - No DB shards could be opened URL: https://github.com/apache/couchdb/issues/796#issuecomment-370187107 Hi, I have the same issue with 60 dbs, running on a local computer, each db has 4 views, up to 2 millions docs for every DB. I am getting emfile error, and shard db. https://gist.github.com/adyshimony/d5b9fec95a7ec7544095cef398960c5c I set up the config exactly as written here http://docs.couchdb.org/en/stable/maintenance/performance.html#system-resource-limits Checking ulimit -n returns 64000 Also set max_dbs_open is 15k Also added [couchdb] update_lru_on_read = false Any clew can help me out here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #1010: Compaction of a heavily updated view - file descriptor leak (known as ".deleted" issue)
wohali commented on issue #1010: Compaction of a heavily updated view - file descriptor leak (known as ".deleted" issue) URL: https://github.com/apache/couchdb/issues/1010#issuecomment-370185337 I'll leave this open as a 1.x issue in the eventuality someone wants to contribute a patch. Once 3.x releases I will mass-close all 1.x issues. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] davisp opened a new pull request #1201: This fixes couch_bt_engine:fold_local_docs/4
davisp opened a new pull request #1201: This fixes couch_bt_engine:fold_local_docs/4 URL: https://github.com/apache/couchdb/pull/1201 Somehow managed to fix without getting it into the PSE PR that was merged the other day. ## Testing recommendations curl _local_docs endpoint ## Checklist - [x] Code is written and works correctly; - [ ] Changes are covered by tests; - [ ] Documentation reflects the changes; This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ondra-novak commented on issue #1010: Compaction of a heavily updated view - file descriptor leak (known as ".deleted" issue)
ondra-novak commented on issue #1010: Compaction of a heavily updated view - file descriptor leak (known as ".deleted" issue) URL: https://github.com/apache/couchdb/issues/1010#issuecomment-370177272 I tried to reproduce the issue on 2.1.1. I have to run compaction directly on port 5986 - on appropriate shard. The deleted file appears as leaked, however it is cleaned after a short period of time (without reporting anything to the log file). It seems that somebody has already fixed this in the newest version. It probably performs some cleanup periodically (aprox each minute) Before you close the issue please verify my observation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ondra-novak commented on issue #1010: Compaction of a heavily updated view - file descriptor leak (known as ".deleted" issue)
ondra-novak commented on issue #1010: Compaction of a heavily updated view - file descriptor leak (known as ".deleted" issue) URL: https://github.com/apache/couchdb/issues/1010#issuecomment-370177272 I tried to reproduce the issue on 2.1.1. I have to run compaction directly on port 5986 - on appropriate shard. The deleted file appears as leaked, however it is cleaned after a short period of time (without reporting anything to the log file). It seems that somebody have already fixed this in the newest version. It probably performs some cleanup periodically (aprox each minute) Before you close the issue please verify my observation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] janl commented on issue #1200: [DISCUSS] CouchDB Request Size Limits
janl commented on issue #1200: [DISCUSS] CouchDB Request Size Limits URL: https://github.com/apache/couchdb/pull/1200#issuecomment-370172189 > `make check` The code is just meant to demonstrate how this could look. This PR is mainly meant for discussion > .ini-* Sorry for bringing this up, I'll open another issue to discuss, it's orthogonal to this discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Jyothishb4 opened a new issue #70: Cannot start coach DB after changing DB file location and log file location
Jyothishb4 opened a new issue #70: Cannot start coach DB after changing DB file
location and log file location
URL: https://github.com/apache/couchdb-docker/issues/70
Hello Exports,
I am getting below error after restarting coach db service. Before restart I
can changed Datafile location and logfile location. Can you please help me to
identify the issue.
[error] 2018-03-03T17:56:33.279000Z couchdb@localhost <0.276.0>
CRASH REPORT Process (<0.276.0>) with 0 neighbors exited with reason: killed
at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240);
initial_call: {couch_index_updater,init,['Argument__1']}, ancestors:
[<0.267.0>,<0.264.0>], messages: [], links: [], dictionary: [], trap_exit:
true, status: running, heap_size: 376, stack_size: 27, reductions: 134
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wohali commented on issue #1200: [DISCUSS] CouchDB Request Size Limits
wohali commented on issue #1200: [DISCUSS] CouchDB Request Size Limits URL: https://github.com/apache/couchdb/pull/1200#issuecomment-370164647 OK, I've had a read through. Looks good, though I have some comments. In 2.2.0, shouldn't we raise the default `max_http_request_size` or reduce the default `max_document_size` so the default configuration doesn't end up with docs too big to replicate? We should support `infinity` as a valid value for all of these `max_*` settings. We already do support `infinity for `max_attachment_size`. We should enforce `max_attachment_* = 0` on the node-local port. We can either ignore, mirror, or hard-code the other limits on the node-local port. (And we should keep moving towards removing the node-local port entirely for 3.0.) I am +1 on the idea of setting `max_attachments_per_doc = 0` to disable attachments, and documenting this as a feature. We can even include text about it improving CouchDB performance, and point people at recommended external solutions for binary management if desired, especially if this is a pattern we are actively trying to discourage. I am -0 on `local.ini-recommended`, as I feel our defaults should already be the recommendations, for any release. If those settings can't yet be the default (as they would be in 2.2.0 vs. 3.0.0) then we should stick them in `local.ini` and just comment them out. People are already used to un-commenting lines in `local.ini` for things they want, let's continue with that pattern. The only way I'd think `local.ini-*` files would make sense is if we wanted to start shipping different settings for different standard configurations: standalone node, 3-node cluster, db-per-user setups, etc. and that's still a bit of a bandaid. (The Debian packaging attempts to hide this type of configuration from the user; why not remove the smarts from the package and stick them right in the product instead?) As a side-note, I'd like to move towards any (and all?) `default.ini` settings to be commented out, meaning the file is effectively empty for a new install and acts almost as documentation. We see in the field plenty of people who keep their old `default.ini` files around - this is especially a problem with the older `Docker` containers - so it'd be nice to eliminate that problem. I know this is challenging for our handler definitions right now, but at least numerical settings like these `max_*` settings could be resolved. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #1200: [DISCUSS] CouchDB Request Size Limits
wohali commented on issue #1200: [DISCUSS] CouchDB Request Size Limits URL: https://github.com/apache/couchdb/pull/1200#issuecomment-370164647 OK, I've had a read through. Looks good, though I have some comments. In 2.2.0, shouldn't we raise the default `max_http_request_size` or reduce the default `max_document_size` so the default configuration doesn't end up with docs too big to replicate? We should support `infinity` as a valid value for all of these `max_*` settings. We already do support `infinity for `max_attachment_size`. We should enforce `max_attachment_* = 0` on the node-local port. We can either ignore, mirror, or hard-code the other limits on the node-local port. (And we should keep moving towards removing the node-local port entirely for 3.0.) I am +1 on the idea of setting `max_attachments_per_doc = 0` to disable attachments, and documenting this as a feature. We can even include text about it improving CouchDB performance, and point people at recommended external solutions for binary management if desired, especially if this is a pattern we are actively trying to discourage. I am -0 on `local.ini-recommended`, as I feel our defaults should already be the recommendations, for any release. If those settings can't yet be the default (as they would be in 2.2.0 vs. 3.0.0) then we should stick them in `local.ini` and just comment them out. People are already used to un-commenting lines in `local.ini` for things they want, let's continue with that pattern. The only way I'd think `local.ini-*` files would make sense is if we wanted to start shipping different settings for different standard configurations: standalone node, 3-node cluster, db-per-user setups, etc. and that's still a bit of a bandaid. (The Debian packaging attempts to hide this type of configuration from the user; why not remove the smarts from the package and stick them right in the product instead? This is well beyond the scope of this PR.) As a side-note, I'd like to move towards any (and all?) `default.ini` settings to be commented out, meaning the file is effectively empty for a new install and acts almost as documentation. We see in the field plenty of people who keep their old `default.ini` files around - this is especially a problem with the older `Docker` containers - so it'd be nice to eliminate that problem. I know this is challenging for our handler definitions right now, but at least numerical settings like these `max_*` settings could be resolved. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #1200: [DISCUSS] CouchDB Request Size Limits
wohali commented on issue #1200: [DISCUSS] CouchDB Request Size Limits
URL: https://github.com/apache/couchdb/pull/1200#issuecomment-370163471
I haven't read everything yet, but you are failing `make check`:
```
couch_doc_tests: doc_from_multi_part_stream_test...*failed*
in function ets:lookup/2
called as lookup(config,{"couchdb","max_attachments_per_document"})
in call from config_meck_original:get/3 (src/config.erl, line 148)
in call from config:get/3
called as get("couchdb","max_attachments_per_document","infinity")
in call from couch_att:max_attachment_count/0 (src/couch_att.erl, line 723)
in call from couch_att:validate_attachment_count/1 (src/couch_att.erl, line
732)
in call from couch_doc:from_json_obj_validate/2 (src/couch_doc.erl, line 138)
in call from couch_doc:doc_from_multi_part_stream/4 (src/couch_doc.erl, line
466)
in call from couch_doc_tests:doc_from_multi_part_stream_test/0
(test/couch_doc_tests.erl, line 34)
**error:badarg
output:<<"">>
couch_doc_tests: doc_to_multi_part_stream_test...ok
couch_doc_tests: len_doc_to_multi_part_stream_test...ok
undefined
*** context setup failed ***
**in function meck_proc:start/2 (src/meck_proc.erl, line 96)
called as start(config,[passthrough])
in call from couch_doc_tests:mock_config/0 (test/couch_doc_tests.erl, line
138)
in call from couch_doc_tests:'-validate_docid_test_/0-fun-34-'/0
(test/couch_doc_tests.erl, line 80)
**error:{already_started,<0.28375.0>}
[done in 0.156 s]
[done in 0.156 s]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wohali commented on issue #1199: Prevent access to Fauxton on node-local port (5986)
wohali commented on issue #1199: Prevent access to Fauxton on node-local port (5986) URL: https://github.com/apache/couchdb/pull/1199#issuecomment-370163316 These were only testing CSP on the node-local port anyway. I can port them to chttpd, give me a day or two. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rnewson commented on issue #1199: Prevent access to Fauxton on node-local port (5986)
rnewson commented on issue #1199: Prevent access to Fauxton on node-local port (5986) URL: https://github.com/apache/couchdb/pull/1199#issuecomment-370152925 losing the CSP tests seems a bit unfortunate. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] janl opened a new pull request #1200: [DISCUSS] CouchDB Request Size Limits
janl opened a new pull request #1200: [DISCUSS] CouchDB Request Size Limits URL: https://github.com/apache/couchdb/pull/1200 *Note: the text below is written in a style that would allow it to be included in the CouchDB 2.2.0 documentation and/or release notes.* # CouchDB Request Size Limits There are multiple configuration variables for CouchDB that determine request size limits. This document explains the configuration variables, how they work together, and why they exist in the first place ## Why Limit Requests by Size Allowing requests of unlimited size to any network server is a [denial of service vector](wikipedia: denial of service). To allow safe operation of CouchDB, even on a network with hostile third parties, various request size limits exist. ## The Request Size Limits `max_http_request_size`: the maximum number bytes a request to a CouchDB server can have. `max_document_size`: the maximum number of bytes for a JSON document written to CouchDB. `max_attachment_size`: the maximum number of bytes for any one attachment written to CouchDB. ## Background There are three distinct ways of getting data into CouchDB: 1. The standard JSON Document API, which uses plain JSON, if binary data is involved, it has to be encoded as base64. The base64 option only exists for legacy reasons and it is not recommended to be used. 2. The standalone attachment API, which allows transferring of binary attachment data without encoding as base64. 3. The multipart HTTP API: it allows the mix of JSON data and binary attachment data without encoding as base64. The CouchDB replicator uses this. In version 2.1, CouchDB started enforcing a 64MB limit for `max_http_request_size` on all requests, but did not apply this to the standalone attachment API. This had the unfortunate side effect that one could create a doc that is smaller than `max_http_request_size` with an attachment that is bigger than `max_http_request_size`. In addition, one could create a doc with two or more attachments that were each smaller than `max_http_request_size` but together bigger than `max_http_request_size`. The result in this scenario now is that these documents could no longer be replicated to CouchDB nodes with the same default configuration (or even to the same node). Regardless to say, this is a very unfortunate user experience: create a number of documents with attachments, and at some not immediately obvious point, replications start failing. ### Large Documents and Attachments While CouchDB works reasonably well with almost any sort of JSON data sizes and attachment sizes. The development team makes recommendations as to the various limits for ideal and optimal uses. CouchDB users may vary from these recommendations, but will need to be okay with the resulting operational implications, like increased CPU & RAM usage as well as increased latency for many core operations. Before CouchDB 2.1.0 there were no real limits imposed, and before CouchDB 2.2.0 the available limits weren?t applied uniformly, leading to surprising behaviour as for example outlined above. CouchDB 2.2.0 and later aims to have a complete set of limits that avoids any unexpected behaviour, but the limits imposed won?t be set by default in order to preserve backwards compatibility. Starting with CouchDB 3.0.0 the recommended limits will be set by default and users migrating from earlier versions of CouchDB need to adjust them, if their use-case requires it. The CouchDB team might produce a utility script that would allow to determine the required settings from an existing CouchDB installation, if resources can be made available for this. Starting with CouchDB 2.2.0, the CouchDB distribution will come with an additional configuration file local.ini-recommended* with the developer-recommended defaults and explanations for what happens when these defaults are exceeded. *An alternative solution could avoid using the `max_attachments_per_doc` and reject attachment additions based on the existing doc + attachments size plus the new attachment size, but this PR/Discussion suggests that having another config value with sensible defaults here will nudge users into doing the right thing* ## Limits by Version In order to account for all use-cases and the interplay of the different APIs, CouchDB 2.2.0 introduces a new limit `max_attachments_per_document`. This allows the application of a formula to show the interplay of all limits: ``` max_http_request_size = max_document_size + multipart HTTP boundary data + max_attachments_per_doc * (max_attachment_size + multipart HTTP boundary data) ``` Using this formula, any doc update (JSON or attachments) can check whether it would exceed
