I have also asked this question in stackoverflow. http://stackoverflow.com/questions/30808294/middleware-to-limit-the-amount-of-requests-downloaded-by-scrapy
On Friday, June 12, 2015 at 6:42:49 PM UTC+2, Leonardo Casanova wrote: > > Hi, > > As the title says my problem is that I need to limit the amount of parse > callbacks being processed concurrently as the are quite memory intensive > (some of them create a selenium instance). > To avoid this I want to prevent scrapy from downloading new requests while > parse callbacks are still being processed. > > I have written this middleware to reschedule requests if they exceed a > certain limit. > However at some point the scraping seems to stop and no new requests are > made. > Can you help me check if I missed something? > > Best Regards > Leo > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
