On Fri, Oct 23, 2009 at 12:27 PM, Adam Kocoloski <[email protected]> wrote: > On Oct 23, 2009, at 2:11 PM, Norman Barker wrote: > >> On Fri, Oct 23, 2009 at 11:33 AM, Paul Davis >> <[email protected]> wrote: >>> >>> On Fri, Oct 23, 2009 at 1:27 PM, Norman Barker <[email protected]> >>> wrote: >>>> >>>> Hi, >>>> >>>> is there a way (in Erlang) to open a couchdb document and to iterate >>>> over the document body without having to open up all of the document >>>> in memory? >>>> >>>> I would like to use a chunked response to keep the system having a low >>>> memory overhead. >>>> >>>> Not a particular couch question, is there a method in erlang to find >>>> the size (as in number of bytes) of a particular term? >>>> >>>> many thanks, >>>> >>>> Norman >>>> >>> >>> Norman, >>> >>> Well, for document JSON we store Erlang term binaries on disk so >>> there's no real way to stream a doc across the wire from disk without >>> loading the whole thing into RAM. Have you noticed CouchDB having >>> memory issues on read loads? Its generally pretty light on its memory >>> requirements for reads. >>> >>> The only way to get the size of a Term in bytes that I know of is the >>> brute force: size(term_to_binary(Term)) method. >>> >>> Paul Davis >>> >> >> I am sending sizeable JSON documents (a couple of mb), as this scales >> by X concurrent users then the problem grows. I have crashed erlang >> when the process gets up to about a 1gb of memory. (Note, this was on >> windows) The workaround is to increase the memory allocation. >> >> Erlang (and couchdb) is fantastic in that it is so light to run as >> opposed to a J2EE server, streaming documents out would be good >> optimisation. Running a couchdb instance in < 30mb of memory space >> would be my ideal. >> >> If you can point me in the right direction then this is something I >> can contribute back, most of my erlang code so far has been specific >> to my application. >> >> Many thanks, >> >> Norman > > Hi Norman, could your application store some of that data as an attachment > to the document? Attachments can be streamed in both directions. Best, > > Adam > >
Hi Adam, it was seeing that attachments could be streamed that got me thinking about the JSON. Unfortunately I couldn't use attachments since I am storing metadata, all of which has to be view-able. Norman
