Looks like the spider I provided doesn't apply in this case in regards to the for loop, you can take that out. Here's an updated paste:
http://pastebin.com/KwzXcQh5 I realized in my original paste it didn't have the right visitedURLs code. Within the "productpage" function, after the if statement, thats where it adds the url to the set. On Fri, Oct 9, 2015 at 9:04 AM, house_scraper <rmi...@gmail.com> wrote: > Is visitedURLs some kind of in-built property of a spider? I don't see > any code here that ever adds a visited url to visitedURLs? > > Also, why do we need a for loop "for url in urls:" ? > > I was assuming that for each product, there is only one "more details" > page, once you know that link you can just yield one request to follow it, > rather than a whole loop of requests? > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to scrapy-users+unsubscr...@googlegroups.com. > To post to this group, send email to scrapy-users@googlegroups.com. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
