Doğacan Güney wrote:
That being said, I remember reading that it is better to keep a connection
open (by using HTTP/1.1) than opening a new connection for each request. I
am not sure, but I think it is also easier on the server too....
This is not as simple as it looks. We can reuse a connection if we don't
open too many connections at the same time ... if a fetchlist consists
of 64k unique hosts you will have a hard time opening that many sockets! :)
So, the benefit from using keepalive connections is mitigated by the
fact that we need to limit the number of open sockets on our side. Also,
consecutive requests to the same host may be separated by a long wait
(due to all fetcher threads being busy fetching other stuff) - from the
point of view of resource utilization, is it still better then to keep
the connection open?
I also upgraded httpclient from 3.0.1 (which is in the trunk) to 3.1
(which is the latest stable 3.* release) and things seem to be working just
fine.
It may be a good idea to update all external libraries before we release 1.0
if they are stable.
I agree.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
