Many solutions here. Here are just two of many possible approaches: 1. Two phases: a) crawl the site limiting URLs only to that site. After you're done, aggregate all the external links and do another crawling job 2. Hack the priority of the requests as you yield them. http://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request
On Monday, March 7, 2016 at 10:37:55 AM UTC, Mario wrote: > > Hi guys, I wrote simple domain web scraper. Is it possible to first crawl > initial site entirely and then move on to the next domain/domains? > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
