Paul, Thanks for the insight! The "3 copies" explains a lot. I have 8g of RAM and two more or less identical machines, one running a perl script to parse the data that gets up to about 50% of RAM with each parsed file, one running database servers (including couchdb) and accepting data. It makes sense that it would die, as 3 times 50% is more than 100!
James On Fri, May 01, 2009 at 03:59:49PM -0700, Paul Davis wrote: > James, > > This sounds like you're hitting Erlang's out of memory handling. Which > is to shutdown the entire VM. Part of the Erlang way is to fail fast > and allow the code to be restarted. It'd probably be better if we > maybe try and detect that we're running out of memory but that seems > like it could get 'interesting' real quick. > > If you're doing things with attachments, you might check out the > standalone API as that should make more efficient use of memory. If > your JSON is really that big, you'll just have to tune your _bulk_docs > size to fit in RAM. And remember that there can be three copies of > your JSON in ram when you have the JSON string, the Erlang term > representation, and the binary before writing to disk. > > HTH, > Paul Davis > > On Fri, May 1, 2009 at 1:41 PM, James Marca <[email protected]> > wrote: > > Hi All, > > > > I'm getting an out of memory type of crash when uploading lots of > > files using bulk docs, and I'm wondering whether > > this is a known issue or user error. > > > > I set the logging to debug and this is what I see: > > > > > > [Fri, 01 May 2009 20:27:30 GMT] [debug] [<0.108.0>] 'POST' > > /d12_june2007/_bulk_docs {1,1} > > Headers: [{'Connection',"TE, close"}, > > {'Content-Length',"257418477"}, > > {'Host',"127.0.0.1:5984"}, > > {"Te","deflate,gzip;q=0.3"}, > > {'User-Agent',"libwww-perl/5.805"}] > > > > [Fri, 01 May 2009 20:31:01 GMT] [info] [<0.108.0>] 127.0.0.1 - - > > 'POST' /d12_june2007/_bulk_docs 201 > > > > > > In top I can watch the RAM usage go up and down until finally it peaks > > and the server is just gone. > > > > That content length is from 1278 documents. I can easily split that > > pile up into fewer documents (and that is what I am going to do) but > > I thought I'd raise the issue here as this feels like a software bug > > somewhere. > > > > I am running the release 0.9, on Gentoo, using the gentoo ebuild, > > x86_64 using the latest (compiled yesterday) release of erlang on > > gentoo. > > > > I'm more than happy to pull new code from git or svn to test things > > out too. > > > > Regards, > > James > > > > -- > > James E. Marca > > Researcher > > Institute of Transportation Studies > > AIRB Suite 4000 > > University of California > > Irvine, CA 92697-3600 > > [email protected] > > > > -- > > This message has been scanned for viruses and > > dangerous content by MailScanner, and is > > believed to be clean. > > > > -- James E. Marca, PhD Researcher Institute of Transportation Studies AIRB Suite 4000 University of California Irvine, CA 92697-3600 [email protected] (949) 824-6287 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
