Hi Adam,
If you could give me a full set a features needed it would be very
useful. I feel like hacking on lhttpc a bit. The ones you listed
already seem like interesting/useful things to add.
Just a note: chunked downloads have been recently added to lhttpc.
Imho it would be interesting to have two choices and test them against
each other in couchdb.
Regards,
Tamas
Tamas Nagy
Erlang Training & Consulting
http://www.erlang-consulting.com
On 6 Jul 2009, at 19:02, Adam Kocoloski wrote:
Hi Tamas, in this case I'm talking about letting the user decide
when to switch the socket back into {active,once} mode to receive
the next message. We like this feature because it allows us to
incrementally process the result of a call to _changes without using
any memory to actually store the response (which may be 100s of MBs).
I poked around lhttpc when ETC released it, but you're right that
it's still missing a number of features that we use in the
replicator. Off the top of my head I can name
* chunked downloads
* chunked uploads
* streaming response bodies
Regarding binaries, our solution so far has been to make requests
that we can reasonably expect to have large response bodies (e.g.
attachment downloads) on dedicated connections outside the ibrowse
connection pool and garbage_collect() those processes as necessary.
Memory management is certainly a concern for us, and I'm curious to
see how the new version of ibrowse behaves now that it's using
binaries internally as well.
Thanks for bringing lhttpc to our attention, but at the moment it
seems that only ibrowse offers the features that we would like in
our replicator HTTP client. Cheers,
Adam
On Jul 6, 2009, at 6:00 AM, Tamas Nagy wrote:
Hi Adam,
What kind of control of the socket behaviour? lhttpc might be a
good candidate as well as it is steadily building up its feature
set with things which are necessary for couchdb. (like chunked).
Arguably ibrowse is a much mature client supporting a lot of
different options (and lhttpc might not have all the required
features yet), but with the recent introduction of using binaries
combined with the long lived processes inside ibrowse can result in
nasty memory blowups as binaries are reference counted in the VM
hence the GC might not be able to get rid of the huge binaries fast
enough during data transfer.
Regards,
Tamas
Tamas Nagy
Erlang Training & Consulting
http://www.erlang-consulting.com
On 4 Jul 2009, at 01:02, Adam Kocoloski wrote:
On Jul 3, 2009, at 7:28 PM, Chris Anderson wrote:
Especially if we can get the replicator based on _changes, and
then truly deprecate the update_notification process
Chandru Mullaparthi gave us a nice assist on that front today with
an update to ibrowse that lets us control the socket behavior. As
far as I know ibrowse is the only Erlang HTTP client that does
this correctly. One month will be more than enough time to build
a replicator based on _changes now that this piece of the puzzle
is resolved.
Adam
