I'm having issues with my spider getting stuck crawling calendars (endless loop due to inifinite amount of links from dynamically rendered dates) and huge forums where it spends hours getting nowhere. Is there anyway to have scrapy move on to the next start_url after a defined amount of time? Any help appriciated!
-- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
