I'm seeing some perf issues out of libcurl (7.19.3) over time relating to the hostcache, even if I've got it disabled.
I'm using curl multi with a ton of easy handles (900 which seems to optimize throughput for us) here's some relevant config: curl_easy_setopt(curl[i]->curl, CURLOPT_DNS_CACHE_TIMEOUT, (long)0); curl_easy_setopt(curl[i]->curl, CURLOPT_FRESH_CONNECT, (long)1); curl_easy_setopt(curl[i]->curl, CURLOPT_FORBID_REUSE, (long)1); We've got a caching DNS server running locally, so we don't need any more DNS cache (our perf is much worse with any of the above reversed). and here's some interesting oprofile output toward the end of our app (a web crawler), after we've pulled ~8million pages from more than 50k domains: samples % linenr info image name app name symbol name 1086204 44.1576 hostip.c:0 libcurl.so.4.1.1 libcurl.so.4.1.1 hostcache_timestamp_remove 664021 26.9946 (no location information) libcurl.so.4.1.1 libcurl.so.4.1.1 Curl_hash_clean_with_criterium 207088 8.4188 (no location information) libcurl.so.4.1.1 libcurl.so.4.1.1 Curl_hash_pick 92158 3.7465 charset.h:204 retrieve retrieve CleanNulls(char*, char*, unsigned long, unsigned long, StatisticSet*) For comparison I included the relative time spent in "CleanNulls", our own function does a pass over each response body (via a for loop) rewriting any zeros to the utf-8 sequence for "?". If I'm not mistaken, this is telling me that about 70% of our CPU time is spent in libcurl using the DNS cache, which I tried to disable. * Am I right that hostip.c, that hostcache_timestamp_remove, Curl_hash_clean_with_criterium, Curl_hash_pick are all related to the DNS cache? * Is there some other option to disable the DNS cache more completely? * Is there a compile time option to disable it so we don't even have to dive into the functions at run-time? --Nick
------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
