You can check the retry middleware: https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/retry.py#L29
It uses the process_exception method to intercept network/timeout issues. Rolando On Thu, Jun 23, 2016 at 10:49 PM, ym zhang <zhangyiming...@gmail.com> wrote: > Did you make it ? I have ran to similiary problem here. I was using scrapy > with unstable proxies too, spider always closed early than expected for > all kinds of network exception. I am curioused about how you managed it > .Can you give me some help here. Thanks:) > > On Thursday, April 4, 2013 at 9:44:22 PM UTC+8, lin di wrote: >> >> sorry to disturb you guys. >> >> finally, i found , process_exception function in download middleware can >> capture it. if it doesn't work, i will try hook signal spider_error. >> >> 在 2013年4月4日星期四UTC+8下午9时40分27秒，lin di写道： >>> >>> >>> i use free proxys at http://www.freeproxylists.net/. when one ip >>> gets error, i need to switch to next low latency ip. >>> >>> there exists several kinds of error like >>> >>> "failed 1 times): An error occurred while connecting: 104: >>> Connection reset by peer", >>> "Connection to the other side was lost in a non-clean >>> fashion.", >>> Connection was refused by other side: 111: Connection refused, >>> User timeout caused connection failure >>> >>> how can i capture them? >>> >>> thanks in advance. >>> >> -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to scrapy-users+unsubscr...@googlegroups.com. > To post to this group, send email to scrapy-users@googlegroups.com. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
