I tried to raise an Exception in the crawler as well, that didnt quite stick either but it might just catch scrapy:s own exceptions?. I may have expressed my goal somewhat poorly. What I am primarily interested in is the log entries over a certain level so the users can see if a certain operation caused an error and possibly solve it themselves. The package https://github.com/mdgart/sentrylogs is doing this for nginx by reading the logfiles so I´d guess I could just fork that and squeeze scrapy logs into that.
Thanks for the tip regarding the logstash/kibana anyways, will have a look into that for sure! On Tuesday, 25 March 2014 12:13:55 UTC+1, Jordi Llonch wrote: > > scrapy_sentry is sending errors to scrapy, not all logs. > > Sentry is not the right tool for what you're trying to achieve. For that > purpose I will suggest you logstash with kibana3. > > Cheers, > Jordi > > > 2014-03-25 21:13 GMT+11:00 <[email protected] <javascript:>>: > >> I want to jack the scrapy logs into my Sentry instance as well but cant >> seem to make it report anything to Sentry. >> Created a few errors in the crawler with the msg method that is available >> in scrapy.log.py . >> Is there anything else I can try? >> >> >> On Tuesday, 26 November 2013 05:18:32 UTC+1, Jordi Llonch wrote: >> >>> Try and raise an exception anywhere in your crawler and check if it's >>> logged. >>> >>> Cheers, >>> Jordi >>> >>> >>> 2013/11/26 Jordi Llonch <[email protected]> >>> >>>> Hi D, >>>> >>>> I am the maintainer. Have you got the SENTRY_DSN defined in the >>>> settings.py or as an environment variable? Have you setup the EXTENSIONS >>>> in >>>> settings.py? >>>> >>>> Thanks, >>>> Jordi >>>> >>>> >>>> 2013/11/26 D <[email protected]> >>>> >>>>> I'm trying to get scrapy-sentry to work, and even though the debug >>>>> info shows that the extension has been loaded (which means settings got >>>>> picked up), and no errors seem to pop up in the logs (but plenty of >>>>> scrapy-related information), and the DSN entry is freshly copied from >>>>> sentry api page, nothing shows up in my sentry dashboard... (hitting >>>>> sentry manually through raven seems to work just fine, so connection is >>>>> valid). >>>>> >>>>> is this project still maintained? is there anything else i can try to >>>>> troubleshoot this? >>>>> >>>>> Thanks, >>>>> -A >>>>> >>>>> >>>>> On Monday, February 4, 2013 4:09:58 PM UTC-5, Jordi Llonch wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> There's a new contributed package: scrapy-sentry that glues scrapy >>>>>> and sentry. >>>>>> https://github.com/llonchj/scrapy-sentry<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fllonchj%2Fscrapy-sentry&sa=D&sntz=1&usg=AFQjCNGKTigEmgXjYu9qwnZ4k5aj2FXqdA> >>>>>> >>>>>> Comments are welcome. >>>>>> >>>>>> Jordi >>>>>> >>>>>> >>>>>> El miércoles, 23 de enero de 2013 16:26:19 UTC+11, Pablo Hoffman >>>>>> escribió: >>>>>>> >>>>>>> If you want to catch only spider errors, I would consider writing a >>>>>>> spider middleware and implementing the >>>>>>> process_spider_exception()<http://www.google.com/url?q=http%3A%2F%2Fdoc.scrapy.org%2Fen%2Flatest%2Ftopics%2Fspider-middleware.html%23scrapy.contrib.spidermiddleware.SpiderMiddleware.process_spider_exception&sa=D&sntz=1&usg=AFQjCNEl6BhefxMrrYuC7sBvdwu1vCUFeg>method. >>>>>>> >>>>>>> If you want to catch all kind of errors, I would consider writing a >>>>>>> custom log observer. This is an undocumented feature, but check the >>>>>>> following links for more info about it: >>>>>>> - source code of scrapy.log >>>>>>> module<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fscrapy%2Fscrapy%2Fblob%2Fmaster%2Fscrapy%2Flog.py&sa=D&sntz=1&usg=AFQjCNFTYEGZEQlDMOeieX5Zx94apGwxpQ> >>>>>>> - how twisted logging >>>>>>> works<http://www.google.com/url?q=http%3A%2F%2Ftwistedmatrix.com%2Fdocuments%2F12.2.0%2Fcore%2Fhowto%2Flogging.html&sa=D&sntz=1&usg=AFQjCNHFneesMvaHIngqueJZLSpR8eE1-Q> >>>>>>> (scrapy >>>>>>> uses twisted logging) >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Jan 19, 2013 at 1:05 AM, clj <[email protected]> wrote: >>>>>>> >>>>>>>> i want to make scrapy working with sentry ( >>>>>>>> https://www.getsentry.com/docs/<https://www.google.com/url?q=https%3A%2F%2Fwww.getsentry.com%2Fdocs%2F&sa=D&sntz=1&usg=AFQjCNE6nyqrOJcGu5h2UeJ3mCScF9E2ZA> >>>>>>>> , an event logging system) >>>>>>>> catch all errors happened in scrapy, then send to sentry. >>>>>>>> >>>>>>>> my thought is: >>>>>>>> i can mixin some sentry's error recording code in >>>>>>>> https://github.com/scrapy/scrapy/blob/master/scrapy/core/scraper.py<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fscrapy%2Fscrapy%2Fblob%2Fmaster%2Fscrapy%2Fcore%2Fscraper.py&sa=D&sntz=1&usg=AFQjCNE8_XFUVPZ4gPhHjoSGuXMdBZsj8w>at >>>>>>>> line 192 and 208 >>>>>>>> >>>>>>>> def _log_download_errors(self, spider_failure, download_failure, >>>>>>>> request, spider): >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> """Log and silence errors that come from the engine (typically >>>>>>>> download >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> errors that got propagated thru here) >>>>>>>> """ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> if spider_failure is download_failure: >>>>>>>> errmsg = spider_failure.getErrorMessage() >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> if errmsg: >>>>>>>> 192 *log.msg(format='Error downloading %(request)s: >>>>>>>> %(errmsg)s',* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> * level=log.ERROR, spider=spider, >>>>>>>> request=request, errmsg=errmsg)* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> return >>>>>>>> return spider_failure >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> def _itemproc_finished(self, output, item, response, spider): >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> """ItemProcessor finished for the given ``item`` and returned >>>>>>>> ``output`` >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> """ >>>>>>>> self.slots[spider].itemproc_size -= 1 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> if isinstance(output, Failure): >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ex = output.value >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> if isinstance(ex, DropItem): >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> logkws = self.logformatter.dropped(item, ex, response, >>>>>>>> spider) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> log.msg(spider=spider, **logkws) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> return >>>>>>>> self.signals.send_catch_log_deferred(signal=signals.item_dropped, \ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> item=item, spider=spider, exception=output.value) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> else: >>>>>>>> 208 *log.err(output, 'Error processing %s' % item, >>>>>>>> spider=spider)* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> else: >>>>>>>> logkws = self.logformatter.scraped(output, response, >>>>>>>> spider) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> log.msg(spider=spider, **logkws) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> return >>>>>>>> self.signals.send_catch_log_deferred(signal=signals.item_scraped, \ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> item=output, response=response, spider=spider) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> question: >>>>>>>> 1, is my consider right? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2, a better or more nature way 。(maybe in middleware? but how?) >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "scrapy-users" group. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msg/scrapy-users/-/5SpOkH_OHEoJ. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit this group at http://groups.google.com/ >>>>>>>> group/scrapy-users?hl=en. >>>>>>>> >>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "scrapy-users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/scrapy-users. >>>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
