Given that you are setting up the engine on your own, you are missing the call to crawler.install(): https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L27
Even though the global scrapy.project.crawler is going to be deprecated, the inspect_response function still uses it: https://github.com/scrapy/scrapy/blob/master/scrapy/shell.py#L129 Perhaps inspect_response should require the spider argument in order to access the crawler object. Rolando On Fri, Feb 21, 2014 at 6:35 PM, John <[email protected]> wrote: > So, putting my scraper under the fresh project and executing directly from > scrapy rather than the django manage.py works. > > I realize this is the scrapy user group however would anyone have an idea > as to why inspect_response doesn't work under django management commands? I > have also noticed some problems with getting pdb.set_trace() to work in the > same situation. > > Here is how I'm setting up and calling the scraper from django (maybe > something has changed in 0.22? This worked on an older version): > settings_module = > importlib.import_module('scrapers_2014.scrapers_2014.settings') > > settings = CrawlerSettings(settings_module) > settings.overrides['ITEM_PIPELINES'] = > self.select_pipeline(options) > crawler = Crawler(settings) > crawler.signals.connect(reactor.stop, signal=signals.spider_closed) > crawler.configure() > crawler.crawl(self._spider) > crawler.start() > log.start() > reactor.run() > > > On Saturday, 22 February 2014 08:20:46 UTC+10, John wrote: >> >> Hmm, it does not fail in a fresh project... That is interesting. I am >> calling it from within my parse callback. The only difference I see is that >> I am using 'BaseSpider' as the parent class and not 'Spider'. I've tried >> changing this but that hasn't made a difference. >> >> Scrapy version info: >> Scrapy : 0.22.2 >> lxml : 3.3.1.0 >> libxml2 : 2.7.8 >> Twisted : 13.2.0 >> Python : 2.7.3 (default, Apr 10 2013, 06:20:15) - [GCC 4.6.3] >> Platform: Linux-3.2.0-57-generic-x86_64-with-Ubuntu-12.04-precise >> >> from scrapy.project import crawler is the line causing the import error. >> >> The other major difference between my fresh project and the project I'm >> working on is that my spider is called from inside a django command... I >> think that is an avenue that needs further investigation. Initially I had >> wanted my scraper to dump straight to the django db but now I'm using an >> intermediary JSON dump so that may no longer be necessary... >> >> On Saturday, 22 February 2014 00:24:06 UTC+10, Rolando Espinoza La fuente >> wrote: >>> >>> Does it fails in a fresh project? How/Where are you calling the >>> function? What's the output of "scrapy version -v"? >>> >>> Sometimes the crawler import error is due to a "from scrapy.project >>> import crawler". >>> >>> Alternatively, I like to use >>> >>> from IPython import embed; embed() >>> >>> instead of inspect_response because it gives me access to the current >>> variables. Although inspect_response gives you the handy shell shortcuts >>> like view(response). >>> >>> Rolando >>> >>> >>> On Fri, Feb 21, 2014 at 7:20 AM, John <[email protected]> wrote: >>> >>>> Hi Everyone, >>>> >>>> I'm trying to debug my scraper and have discovered the >>>> inspect_response() function which looks quite useful. >>>> >>>> However when importing it I get the following exception: >>>> exceptions.ImportError: >>>> cannot import name crawler >>>> >>>> I have also attempted using inspect_response from the python shell >>>> and get the same error. >>>> >>>> I'm using scrapy 0.22.2. Has anyone else encountered this error? What >>>> more information can I provide to investigate this? >>>> >>>> Cheers, >>>> John >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "scrapy-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/scrapy-users. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
