On 16/05/2009, at 11:07 PM, Adam Kocoloski wrote:
No, I don't believe so. ibrowse accepts a {stream_to, pid()}
option. It accumulates packets until it reaches a threshold
configurable by {stream_chunk_size, integer()} (default 1MB), then
sends the data to the Pid. I don't think ibrowse is writing to disk
at any point in the process. We do see that when streaming really
large attachments, ibrowse becomes the biggest memory user in the
emulator.
This is what I thought was happening, which means that with small
documents with many attachments (say > 1Mb) you could potentially end
up with masses of open connections representing data promises that are
only forced at checkpoint time, so that's not scalable. I think the
number of open ibrowse connections (which I see doesn't neccessariy
match the number of unforced promises), needs to be an input to the
checkpoint decision.
ibrowse does offer a {save_response_to_file, boolean()|filename()}
option that we could possibly leverage.
That sounds like a better idea.
Are you keeping an eye on the memory usage? I think an out of
memory error can trigger this sudden death in Erlang.
Not a memory failure - it happens at the same place with either 1 or
1.5G. Once again I have a repeatable failure that would normally be
random :/ I'm just wondering how to debug it.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
A priest, a minister and a rabbi walk into a bar. The bartender says
"What is this, a joke?"
