> Key is not yielding the item on that first pass, but rather the request to jump into the 'more details' function
AHA! I knew there would be a way to do this working with the twisted philosophy rather than against it! Thanks! On Wednesday, 7 October 2015 13:56:26 UTC-7, Jeremy D wrote: > > Hey, > > I posted this in another thread > > http://pastebin.com/H7zLw1FK > > You're right, it will process all listings first and then do the more > details listings. There really isnt any 'games' or matching up, its pretty > seamless to do this. Key is not yielding the item on that first pass, but > rather the request to jump into the 'more details' function. > > Someone may know a way to have scrapy view the 'more details' listing > before moving on to the second listing, but I don't. > > On Wed, Oct 7, 2015 at 4:50 PM, house_scraper <rmi...@gmail.com > <javascript:>> wrote: > >> Hi all, >> >> the page I am scraping has the following structure: >> >> >> *Listing 1* >> >> *Link to more details for listing 1* >> >> *Listing 2* >> >> *Link to more details for listing 2* >> >> *Listing 2* >> >> *Link to more details for listing 2* >> >> >> >> >> >> I want to follow the "*Link to more details for listing 1" * from my >> spider code that processes listing 1, fill a field for item 1, and then >> move onto listing 2, follow the listing 2 link, etc and iterate through all >> the items in order. >> >> However, scrapy's twistedreactor/callback philosophy seems to prevent >> this, and forces me to *first *do all the items without following links, >> and *then *follow all the links using a linkextractor and play games to >> match up each followed link to the listing/item it came from. >> >> Can I just confirm that this is correct? >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to scrapy-users...@googlegroups.com <javascript:>. >> To post to this group, send email to scrapy...@googlegroups.com >> <javascript:>. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
