[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090410#comment-13090410
]
Filipe Manana commented on COUCHDB-1258:
----------------------------------------
I have a similar case. Database with about 2.6 million deleted documents.
The map function is not emitting any key/value and memory consumption goes to
about 4Gb.
After the indexer finishes, and the system goes idle, memory usage stays at
about 2Gb forever.
Inspecting the erlang processes, a couch_view_group process is using about 2Gb
of memory:
7> process_info(list_to_pid("<0.276.0>")).
[{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.278.0>,<0.132.0>]},
{dictionary,[{'$ancestors',[<0.275.0>]},
{'$initial_call',{couch_view_group,init,1}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.37.0>},
{total_heap_size,323345205},
{heap_size,38263080},
{stack_size,9},
{reductions,1041902893},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,538}]},
{suspending,[]}]
8>
Manually invoking the garbage collector against this process releases memory
and after that the process is using only about 7Kb of memory:
8> erlang:garbage_collect(list_to_pid("<0.276.0>")).
true
9> process_info(list_to_pid("<0.276.0>")).
[{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.278.0>,<0.132.0>]},
{dictionary,[{'$ancestors',[<0.275.0>]},
{'$initial_call',{couch_view_group,init,1}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.37.0>},
{total_heap_size,987},
{heap_size,987},
{stack_size,9},
{reductions,1041902893},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]
10>
I have no idea why the view group is using so much memory.
My best call is that we need to call couch_util:should_flush/1 everytime we
receive a partial_update message in the view group.
This was with OT R14B03.
> eheap_alloc OOM errors when attempting to build selected views
> --------------------------------------------------------------
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
> Reporter: James Cohen
> Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
> "_id": "_design/cleanup",
> "_rev": "5-e004fbab278355e9d08763877e5a8295",
> "views": {
> "byDate": {
> "map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
> }
> }
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
> "_id": "_design/cleanup",
> "_rev": "6-3823be6b72ca2441e235addfece6900c",
> "views": {
> "byDate": {
> "map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
> }
> }
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
