Hi there, I am working on some monitoring for my python/scrapy deployment using one of the commercial APM tools. I was able to instrument the parsing of the response as well as the pipeline which pushes the items into an ElasticSearch instance. You can see in the attached screenshot how that is visualized in the tool. I would now also like to see the outgoing calls that Scrapy is making through the downloader to actually crawl the http pages (which is obviously happening before parsing and pipelining). But I can't figure out where in the code the actual http call is made so that I could put my monitoring hook around it. Could you guys please point me to the class that is actually doing the http calls ?
Thanks Philipp -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
