Use proxy, and final scrapyd log stats:
2015-12-04 21:03:46 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 332,
'downloader/exception_type_count/scrapy.core.downloader.handlers.http11.TunnelError':
21,
'downloader/exception_type_count/twisted.internet.error.TimeoutError': 229,
'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived':
82,
...
}
def create_foo_request():
return Request(url=url, callback=callback, headers=headers,
meta=meta_merged,
errback=self._request_errback)
def _request_errback(self, failure):
# Expect to handle TunnelError, TimeoutError, ResponseNeverReceived
But can't catch any error at _request_errback(...)
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
