Doesn't look good. Try with 20 URLs before 300k. Try running from an instance with a bit more memory Do you have any pipelines/code you don't control? Try disabling them to see what happens. Make sure the server can access the target website.
Cheers Dimitris On Monday, February 1, 2016 at 3:17:35 PM UTC, Uncharted wrote: > > Hello Everyone > > I've got a problem with scrapy freezing sometimes > when i look with telnet the requests seems to be enqueued but nothing > happens > > do you have suggestions where i can find/debug the blocking point ? > > Thanks > > > Example > > Execution engine status > > time()-engine.start_time : 6524.52261305 > engine.has_capacity() : False > len(engine.downloader.active) : 0 > engine.scraper.is_idle() : False > engine.spider.name : consumer > engine.spider_is_idle(engine.spider) : False > engine.slot.closing : False > len(engine.slot.inprogress) : 0 > len(engine.slot.scheduler.dqs or []) : 0 > len(engine.slot.scheduler.mqs) : 315598 > len(engine.scraper.slot.queue) : 0 > len(engine.scraper.slot.active) : 0 > engine.scraper.slot.active_size : 0 > engine.scraper.slot.itemproc_size : 0 > engine.scraper.slot.needs_backout() : False > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
