One change that I did was :
RELATED_SELECTOR = '#watch-related a ::attr(href)'
for article in response.css(RELATED_SELECTOR).extract():
if article:
yield scrapy.Request(
response.urljoin(article),
callback=self.parse, #dont_filter=True
)
This throws the spider into an infinite loop.
On Saturday, 4 March 2017 23:12:27 UTC+5:30, JoeJoe wrote:
>
>
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
