Hi Dimitri thanks for your answer I don't think it's a memory problem our server has 14gb of ram and we are very far from using it totally For the pipeline good idea, I will try your suggestion and run it with pipeline disabled
Le mercredi 3 février 2016 01:55:00 UTC+1, Dimitris Kouzis - Loukas a écrit : > > Doesn't look good. > > Try with 20 URLs before 300k. > Try running from an instance with a bit more memory > Do you have any pipelines/code you don't control? Try disabling them to > see what happens. > Make sure the server can access the target website. > > Cheers > Dimitris > > On Monday, February 1, 2016 at 3:17:35 PM UTC, Uncharted wrote: >> >> Hello Everyone >> >> I've got a problem with scrapy freezing sometimes >> when i look with telnet the requests seems to be enqueued but nothing >> happens >> >> do you have suggestions where i can find/debug the blocking point ? >> >> Thanks >> >> >> Example >> >> Execution engine status >> >> time()-engine.start_time : 6524.52261305 >> engine.has_capacity() : False >> len(engine.downloader.active) : 0 >> engine.scraper.is_idle() : False >> engine.spider.name : consumer >> engine.spider_is_idle(engine.spider) : False >> engine.slot.closing : False >> len(engine.slot.inprogress) : 0 >> len(engine.slot.scheduler.dqs or []) : 0 >> len(engine.slot.scheduler.mqs) : 315598 >> len(engine.scraper.slot.queue) : 0 >> len(engine.scraper.slot.active) : 0 >> engine.scraper.slot.active_size : 0 >> engine.scraper.slot.itemproc_size : 0 >> engine.scraper.slot.needs_backout() : False >> >> -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
