I had problems with blocking dns requests in my crawler and I switched to c-ares. I have to say the performance gain is impresive.
On the other hand I'm trying to run some experiments using CURLOPT_LOW_SPEED_LIMIT 512 CURLOPT_LOW_SPEED_TIME 180 In combination with CURLOPT_CONNECTTIMEOUT 60 But I'm not observing any timeouts (using curl_multi_info_read). As the handles have been for some minutes stuck with the same url, with no activity checked by tcpdump. My questions are: 1 - Any ideas what to try next? 2 - Is there a way to interrupt a transfer, I don't see any call in the easy interface for this. Since I'm accounting the time and transfer I could manually kill those stuck connections. Thanks. Pedro. PD: debug output from my program showing the problem: handle 0: ROBOTS: 0 h 4 m: 0 k: http://blog.omnigroup.com/robots.txt handle 1: ROBOTS: 0 h 4 m: 0 k: http://zoetrope.stripgenerator.com/robots.txt handle 2: ROBOTS: 0 h 4 m: 0 k: http://zoomit.wordpress.com/robots.txt handle 3: ROBOTS: 0 h 2 m: 0 k: http://zangre.blogspot.com/robots.txt handle 4: ROBOTS: 0 h 4 m: 0 k: http://zozo2k3.blogspot.com/robots.txt handle 5: ROBOTS: 0 h 4 m: 0 k: http://zumodevidrio.blogspot.com/robots.txt handle 6: ROBOTS: 0 h 4 m: 0 k: http://losrojosnousabansombrero.blogspot.com/robots.txt handle 7: ROBOTS: 0 h 4 m: 0 k: http://zhaojunzheblog.blog.sohu.com/robots.txt handle 8: ROBOTS: 0 h 4 m: 0 k: http://mariettamorales.blogspot.com/robots.txt handle 9: ROBOTS: 0 h 4 m: 0 k: http://mediciland.blogspot.com/robots.txt handle 10: ROBOTS: 0 h 4 m: 0 k: http://zoetrope101.spaces.live.com/robots.txt handle 11: ROBOTS: 0 h 4 m: 0 k: http://zettwoch.blogspot.com/robots.txt handle 12: ROBOTS: 0 h 4 m: 0 k: http://zvonnews.sourceforge.net/robots.txt handle 13: ROBOTS: 0 h 4 m: 0 k: http://zoetrope101.blogspot.com/robots.txt handle 14: ROBOTS: 0 h 4 m: 0 k: http://zerozofoke.blogspot.com/robots.txt handle 15: ROBOTS: 0 h 4 m: 0 k: http://zoetrope.tumblr.com/robots.txt handle 16: ROBOTS: 0 h 4 m: 0 k: http://paulauster.blogspot.com/robots.txt handle 17: ROBOTS: 0 h 4 m: 0 k: http://zonda.wordpress.com/robots.txt handle 18: ROBOTS: 0 h 4 m: 0 k: http://zoemini.blogspot.com/robots.txt handle 19: ROBOTS: 0 h 4 m: 0 k: http://romavictrix.blogspot.com/robots.txt down: 6.46 MiB 0.00 KB/s ------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
