mmarinchenko opened a new issue #1209: Erlang native query server consumes too much RAM when filters a document with lots of revisions URL: https://github.com/apache/couchdb/issues/1209 Hi all! We have online service which uses CouchDB heavily. It has many databases with frequently updated documents. So we faced a problem. Documents are processed by native query servers with all of its revisions as a one single entity when they are requested with `style=all_docs` parameter. As a result when document with many, many revisions including deleted ones is requested through native Erlang filter CouchDB eats inadequate amount of memory. The reason is as follows (branch `master` is used here just for simplicity, we use CouchDB 1.6 actually): 1. `couch_changes:filter()` reads all revisions of a document into `Docs` and calls `couch_query_servers:filter_docs()` 2. `couch_query_servers:filter_docs()` calls `couch_native_process:prompt()` through `ddoc_prompt()`/`proc_prompt()` 3. And then `couch_native_process:handle_call()` tries to convert the document with all of its revisions `to_binary` at once! Filter processes each revision one by one, right? Why trying to handle all revisions in memory? And if you can handle all revisions in memory then why not to handle in memory entire database? :) Of course, this problem may be workarounded from the maintenance perspective. Old revisions may be purged from the databases, databases may be compacted and so on. But this is not mean that current behavior is an example of good software design, I guess. ## Expected Behavior Assume that the RAM in our galaxy is not infinite, and the document may have many and many and many revisions. Simply put do not load all revisions at once. After all each revision is a document itself. ## Steps to Reproduce 1. Create database with document which has a lot of revisions (if you have very much amount of RAM you may try to create 10 000 revisions or even greater). 2. Enable Erlang native query server in local.ini and write any simple filter using Erlang (the document from point 1 is not required to pass this filter). 3. Request the _changes feed using created Erlang filter with "style=all_docs" parameter. Just to clarify you may emulate 16 parallel queries as follows: `for i in {1..16}; do curl -u admin:admin "http://localhost:5984/db/_changes?feed=normal&style=all_docs&filter=app%2FverySimpleErlangFilter"; & done` 4. Run htop and look at how CouchDB eats all your memory! ## Environment * Version used: 1.6.0 * Operating System and version: Ubuntu 16.04
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
