I have made the changes and its indeed working. Thank you very much on your help. Much much appreciated.
On Sunday, March 5, 2017 at 12:44:38 PM UTC+8, Parth Verma wrote: > > One change that I did was : > > RELATED_SELECTOR = '#watch-related a ::attr(href)' > for article in response.css(RELATED_SELECTOR).extract(): > > if article: > yield scrapy.Request( > response.urljoin(article), > callback=self.parse, #dont_filter=True > ) > > This throws the spider into an infinite loop. > > > On Saturday, 4 March 2017 23:12:27 UTC+5:30, JoeJoe wrote: >> >> >> -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
