Hi Tamas, in this case I'm talking about letting the user decide when
to switch the socket back into {active,once} mode to receive the next
message. We like this feature because it allows us to incrementally
process the result of a call to _changes without using any memory to
actually store the response (which may be 100s of MBs).
I poked around lhttpc when ETC released it, but you're right that it's
still missing a number of features that we use in the replicator. Off
the top of my head I can name
* chunked downloads
* chunked uploads
* streaming response bodies
Regarding binaries, our solution so far has been to make requests that
we can reasonably expect to have large response bodies (e.g.
attachment downloads) on dedicated connections outside the ibrowse
connection pool and garbage_collect() those processes as necessary.
Memory management is certainly a concern for us, and I'm curious to
see how the new version of ibrowse behaves now that it's using
binaries internally as well.
Thanks for bringing lhttpc to our attention, but at the moment it
seems that only ibrowse offers the features that we would like in our
replicator HTTP client. Cheers,
Adam
On Jul 6, 2009, at 6:00 AM, Tamas Nagy wrote:
Hi Adam,
What kind of control of the socket behaviour? lhttpc might be a good
candidate as well as it is steadily building up its feature set with
things which are necessary for couchdb. (like chunked).
Arguably ibrowse is a much mature client supporting a lot of
different options (and lhttpc might not have all the required
features yet), but with the recent introduction of using binaries
combined with the long lived processes inside ibrowse can result in
nasty memory blowups as binaries are reference counted in the VM
hence the GC might not be able to get rid of the huge binaries fast
enough during data transfer.
Regards,
Tamas
Tamas Nagy
Erlang Training & Consulting
http://www.erlang-consulting.com
On 4 Jul 2009, at 01:02, Adam Kocoloski wrote:
On Jul 3, 2009, at 7:28 PM, Chris Anderson wrote:
Especially if we can get the replicator based on _changes, and
then truly deprecate the update_notification process
Chandru Mullaparthi gave us a nice assist on that front today with
an update to ibrowse that lets us control the socket behavior. As
far as I know ibrowse is the only Erlang HTTP client that does this
correctly. One month will be more than enough time to build a
replicator based on _changes now that this piece of the puzzle is
resolved.
Adam
