[
https://issues.apache.org/jira/browse/COUCHDB-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899987#comment-13899987
]
Rohit Sharma commented on COUCHDB-2058:
---------------------------------------
Paul,
I have more stats for you, this time with vmstat and free -m. Also, could you
please clarify on couple of the last suggestion you made i.e.
1- disabling compression: we are not doing compression by the process, is it
something in the CouchDB configuration which can be disabled or is happening
without us knowing. We also do not have automatic compression enabled in config.
2- Not sure how to replace ejson with mochijson2. Our process is written in
Node.js and not sure where I need to make this replacement.
If you could direct me, I can try different things to isolate the issue.
So far I have tried running the process on Mac, Ubuntu, Red Hat, Windows -
every where the behaviour is same. I also have tried it with CouchDB 1.3 and
1.5.
Before running the process
vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 4117312 56576 2582912 0 0 5 57 277 112 6 1 92 0
0
free -m
total used free shared buffers cached
Mem: 7872 3851 4020 0 55 2522
-/+ buffers/cache: 1274 6598
Swap: 2015 0 2015
Top
Tasks: 126 total, 1 running, 125 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 1.0%sy, 0.0%ni, 98.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8061508k total, 3944212k used, 4117296k free, 56564k buffers
Swap: 2064376k total, 0k used, 2064376k free, 2582912k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13623 root 20 0 533m 26m 4216 S 0.3 0.3 0:01.00
/usr/local/lib/erlang/erts-5.10.4/bin/beam.smp -Bd -K true -A 4 -- -root
/usr/local/lib/erlang -progname erl -- -home /root -- -nosh
After running the process
vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 2220 1347560 44424 2221356 0 0 4 78 313 318 8 1 91 0
0
free -m
total used free shared buffers cached
Mem: 7872 6558 1313 0 43 2169
-/+ buffers/cache: 4345 3526
Swap: 2015 2 2013
Cpu(s): 1.0%us, 1.2%sy, 0.0%ni, 97.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8061508k total, 6712020k used, 1349488k free, 44416k buffers
Swap: 2064376k total, 2220k used, 2062156k free, 2221356k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13623 root 20 0 3924m 3.0g 4372 S 2.0 39.4 36:03.63
/usr/local/lib/erlang/erts-5.10.4/bin/beam.smp -Bd -K true -A 4 -- -root
/usr/local/lib/erlang -progname erl -- -home /root -- -nosh
> CouchDB Memory Leak - Beam.smp
> ------------------------------
>
> Key: COUCHDB-2058
> URL: https://issues.apache.org/jira/browse/COUCHDB-2058
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Database Core
> Reporter: Rohit Sharma
>
> Hello,
> I am experiencing performance issue with CouchDB.
> Use Case: I am working on a process that retrieves the data from RDBMS and
> process them into JSON document and POST them to the CouchDB.
> I am trying to POST around half a million documents, most of them in batches
> (_bulk_doc) of 10,000 and have tried with batch of 5,000, 15,000, and 20,000.
> Whole process takes around 90-100 minutes.
> During the life of the process, Memory Consumption by CouchDB keeps on
> growing and memory is not released when CouchDB has finished working.
> So if the memory consumption by CouchDB was 60% at the time process finishes,
> memory consumption will remain 60% and not reducing.
> Subsequently, when the process starts running again. memory consumption is
> Maxed out and CouchDB restarts itself. This restart fails the process that I
> am running. Looking at the Syslogs , I see Out Of Memory Error by the CouchDB
> process and killing statement.
> The CouchDb process that has the issue is the "beam.smp" of Erlang.
> At this point, I have tried upgrading the memory of the server to see if this
> resolves the issue, unfortunately, the issue persists. Memory Leak is there
> and Usage keeps on growing until CouchDB restarts/crashed.
> I also have tried running garbage collection from Erlang command
> (erlang:garbage_collect().) line but it didn't do anything.
> At this point, I am out of ideas and not sure what is going on here. Any
> input/suggestion is highly appreciated!
> Env:
> Platform: Linux (Red Hat release 6.4 (Santiago))
> CouchDB: 1.3 and have tried with 1.5 as well
> RAM: Tried with 2G, 4G, and 8G
> CPU: 2 cores
> Process:/usr/lib64/erlang/erts-5.8.5/bin/beam.smp -Bd -K true -A 4 -- -root
> /usr/lib64/erlang
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
