I tried to get pipelining working with cURL, but no luck. It looks to me that cURL multi API is overcomplicated and not-too-well documented...
So I have taken a look at libwww and it works great! I created (copied a sample) simple program to load given URL many times using only persistent connection and using pipelining. And results are much better than I expected: loading 1000 times http://nnov.ru takes 1:29,49 with persistent connection and only 22,946s with pipelining! I will experiment with replacing current cURL implementation with libwww. I am not familiar with Darcs sources, advices are welcome. After a quick look at External.hs it looks to me like we can provide a function which takes a list of URLs instead of one, and fetches them using pipelining. I think this will require minimum changes to Darcs sources. Am I correct with this? Regards, Dmitry 2007/12/18, Dmitry Kurochkin <[EMAIL PROTECTED]>: > After looking at this a bit more it does not look so good to me. > > I thought that darcs uses a new TCP connection for each patch. But > wireshark shows that it uses persistent connection already. > So there is no TCP handshake overhead. > > Pipelining is sending multiple requests without waiting for response. > This would give some benefit but I am not sure it worth the price of > moving to cURL multi interface. > > I think I will write a script which grabs darcs repo using cURL with > just persistent connection and with pipelining. And depending on > results we can decide what to do. > > Regards, > Dmitry > > > David Roundy wrote: > > Hmmmm. Alas, curl 7.16 is not yet in debian stable (my default for when we > > can require something), but that's still very interesting! Do you by any > > chance have experience coding with libcurl? We've only ever used the "easy" > > interface, and if you'd like to take a shot at updating src/hscurl.c to use > > the "multi" interface, that would be great! > > > > (We'd need to have a configure check so we'd work with older libcurl > > versions, but I could handle that if you'd rather not.) > > > > David > > > > On Tue, Dec 18, 2007 at 01:06:26AM +0300, Dmitry Kurochkin wrote: > > > >> Hi David, > >> > >> According to cURL changelog pipeline support has been added in version > >> 7.16.0: > >> > >> CURLMOPT_PIPELINING added for enabling HTTP pipelined transfers. > >> > >> I did not use it myself but from the docs it looks like this is just > >> what you want. Quote: > >> > >> CURLMOPT_PIPELINING > >> > >> Pass a long set to 1 to enable or 0 to disable. Enabling pipelining on a > >> multi handle will make it attempt to perform HTTP Pipelining as far as > >> possible for transfers using this handle. This means that if you add a > >> second request that can use an already existing connection, the second > >> request will be "piped" on the same connection rather than being > >> executed in parallell. (Added in 7.16.0) > >> > >> Thanks for your work on darcs (and xmonad :))! > >> > >> Regards, > >> Dmitry > >> > > _______________________________________________ > > darcs-devel mailing list > > [email protected] > > http://lists.osuosl.org/mailman/listinfo/darcs-devel > > > > > _______________________________________________ darcs-devel mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-devel
