Hi Jun, This can be a banning strategy from the site. You are crawling it fast and from one single IP, they might be detecting this and what a better way of banning you than returning bad responses? I'm not completely sure about this, but I think it's very likely that this is happening.
2014-11-13 18:32 GMT-02:00 <j...@borderxlab.com>: > (I've posted this on github but because I couldn't attach non-images on > github so I repost here.) > > Hi Scrapy experts, > I am new to Scrapy. But it's really a cool tool. I like it very much! > Recently I came across a problem. I am not sure if this is an existing > issue -- I searched ON Google, github and didn't find anything similar. > Here is my question. I wrote a crawler to crawl products on mango.com. I > set CONCURRENT_REQUESTS = 16 in settings.py. However, I found sometimes the > response passed to my parser function is ANOTHER page. I used > inspect_response(response, self) to stop in my parser function to debug. I > found response.url is correct. But if I type view(response), it shows a > completely different page from response.url. Later if I set > CONCURRENT_REQUESTS = 1 and reran the crawler, the problem seems gone. > > I wrote a very simple crawler (mango_test) and pasted my debug results in > this thread. The first couple of responses are correct. But if you look at > the last one in the results. The response.url is > http://shop.mango.com/US/p0/women/accessories/crisscross-leather-sandals/?id=33080334_02&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906. > However, if I run a command to get product name. It extracts a total > different product "Wedge suede ankle boots", which does not exist on the > web page at all. If I run view(response), it also shows a totally different > page. > > I also attached my crawler (mango_test) in this thread. Hopefully it will > help reproduce the issue. > > Thanks, > Jun > > ======Debug results > > scrapy crawl spider_mt > > 2014-11-13 12:19:11-0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: > mango_test) > > 2014-11-13 12:19:11-0800 [scrapy] INFO: Optional features available: ssl, > http11 > > 2014-11-13 12:19:11-0800 [scrapy] INFO: Overridden settings: > {'NEWSPIDER_MODULE': 'mango_test.spiders', 'SPIDER_MODULES': > ['mango_test.spiders'], 'BOT_NAME': 'mango_test'} > > 2014-11-13 12:19:11-0800 [scrapy] INFO: Enabled extensions: LogStats, > TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState > > 2014-11-13 12:19:12-0800 [scrapy] INFO: Enabled downloader middlewares: > HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, > RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, > HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, > ChunkedTransferMiddleware, DownloaderStats > > 2014-11-13 12:19:12-0800 [scrapy] INFO: Enabled spider middlewares: > HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, > UrlLengthMiddleware, DepthMiddleware > > 2014-11-13 12:19:12-0800 [scrapy] INFO: Enabled item pipelines: > > 2014-11-13 12:19:12-0800 [spider_mt] INFO: Spider opened > > 2014-11-13 12:19:12-0800 [spider_mt] INFO: Crawled 0 pages (at 0 > pages/min), scraped 0 items (at 0 items/min) > > 2014-11-13 12:19:12-0800 [scrapy] DEBUG: Telnet console listening on > 127.0.0.1:6025 > > 2014-11-13 12:19:12-0800 [scrapy] DEBUG: Web service listening on > 127.0.0.1:6082 > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women> (referer: None) > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Filtered offsite request to ' > www.facebook.com': <GET http://www.facebook.com/mango.com> > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Filtered offsite request to ' > www.twitter.com': <GET http://www.twitter.com/mango> > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Filtered offsite request to ' > plus.google.com': <GET https://plus.google.com/117396569580388428996/posts > > > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Filtered offsite request to ' > instagram.com': <GET http://instagram.com/mango> > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Filtered offsite request to ' > pinterest.com': <GET http://pinterest.com/mangofashion/> > > 2014-11-13 12:19:12-0800 [spider_mt] DEBUG: Filtered offsite request to ' > www.youtube.com': <GET http://www.youtube.com/MANGOFASHION> > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/brooklyn-girl> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/men/new> (referer: http://shop.mango.com/US/women > ) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/catalogue/2014-autumn> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/foulards-and-scarves> > (referer: http://shop.mango.com/US/women) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/catalogue> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/lookbooks> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/new> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:13-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/lookbooks/fw-2014-preview> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/bags> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/hats-and-caps> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/leather-goods> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/belts> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/clothing/jumpsuits> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/jewellery> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Redirecting (302) to <GET > http://shop.mango.com/preHome.faces?state=violeta_400_US> from <GET > http://shop.mango.com/countries/women> > > 2014-11-13 12:19:14-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/violeta/accessories/shoes> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/social> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/social/tv> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Filtered duplicate request: > <GET http://shop.mango.com/US/violeta/new> - no more duplicates will be > shown (see DUPEFILTER_DEBUG to show all duplicates) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/catalogue/2014-autumn> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/lookbooks/september> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/lookbooks> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/the-magazine> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/lookbooks/november> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:15-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/lookbooks/mango-premium> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/preHome.faces?state=violeta_400_US> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/catalogue> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/catalogue/touch-2014> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/catalogue/winter-2014> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/other-accessories> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/gloves> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:16-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/sunglasses> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/foulards-and-scarves> > (referer: http://shop.mango.com/US/women) > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/hats-and-caps> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/belts> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Redirecting (302) to <GET > http://shop.mango.com/index.faces?state=she_400_US> from <GET > http://shop.mango.com/entradaPaises.faces?pais=> > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/jewellery> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/leather-goods> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:17-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/bags> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:18-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories/shoes> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:18-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/accessories> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:18-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/chequeRegalo.faces> (referer: > http://shop.mango.com/US/women/social) > > 2014-11-13 12:19:19-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/p0/women/accessories/check-scarf/?id=33035678_92&ident=0_accesorio52%2C452_0_1415909108906&n=1&s=accesorios.fulares&ts=1415909108906> > (referer: http://shop.mango.com/US/women/accessories/gloves) > > 2014-11-13 12:19:19-0800 [spider_mt] DEBUG: Now parsing > http://shop.mango.com/US/p0/women/accessories/check-scarf/?id=33035678_92&ident=0_accesorio52%2C452_0_1415909108906&n=1&s=accesorios.fulares&ts=1415909108906 > > [u'\n\t\t\t\t\t\t\t\tCheck scarf\n\t\t\t\t\t\t\t'] > > > http://shop.mango.com/US/p0/women/accessories/check-scarf/?id=33035678_92&ident=0_accesorio52%2C452_0_1415909108906&n=1&s=accesorios.fulares&ts=1415909108906 > > [s] Available Scrapy objects: > > [s] crawler <scrapy.crawler.Crawler object at 0x10d351950> > > [s] item {} > > [s] request <GET > http://shop.mango.com/US/p0/women/accessories/check-scarf/?id=33035678_92&ident=0_accesorio52%2C452_0_1415909108906&n=1&s=accesorios.fulares&ts=1415909108906 > > > > [s] response <200 > http://shop.mango.com/US/p0/women/accessories/check-scarf/?id=33035678_92&ident=0_accesorio52%2C452_0_1415909108906&n=1&s=accesorios.fulares&ts=1415909108906 > > > > [s] settings <scrapy.settings.Settings object at 0x10c79aa50> > > [s] spider <MangoTestSpider 'spider_mt' at 0x10d360250> > > [s] Useful shortcuts: > > [s] shelp() Shell help (print this help) > > [s] view(response) View response in a browser > > > In [*1*]: > > Do you really want to exit ([y]/n)? y > > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Scraped from <200 > http://shop.mango.com/US/p0/women/accessories/check-scarf/?id=33035678_92&ident=0_accesorio52%2C452_0_1415909108906&n=1&s=accesorios.fulares&ts=1415909108906 > > > > {'name': [u'\n\t\t\t\t\t\t\t\tCheck scarf\n\t\t\t\t\t\t\t']} > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/clothing/premium> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/clothing/jumpsuits> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/clothing/shorts> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/m/women/accessories/black/?m=color&v=9> > (referer: http://shop.mango.com/US/women/accessories/other-accessories) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/p0/women/accessories/knit-wool-blend-scarf/?id=33025533_02&ident=0_accesorio45%2C445_0_1415909108906&n=1&s=accesorios.gorros&ts=1415909108906> > (referer: http://shop.mango.com/US/women/accessories/foulards-and-scarves) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/m/women/accessories/sale/?m=coleccion&v=true> > (referer: http://shop.mango.com/US/women/accessories/sunglasses) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/p0/women/accessories/heel-lace-up-ankle-boots/?id=33065662_02&ident=0_accesorio48%2C448_0_1415909108906&n=1&s=accesorios.bisuteria&ts=1415909108906> > (referer: http://shop.mango.com/US/women/accessories/hats-and-caps) > > 2014-11-13 12:19:28-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/women/clothing/essentials> (referer: > http://shop.mango.com/US/women) > > 2014-11-13 12:19:29-0800 [spider_mt] DEBUG: Now parsing > http://shop.mango.com/US/p0/women/accessories/knit-wool-blend-scarf/?id=33025533_02&ident=0_accesorio45%2C445_0_1415909108906&n=1&s=accesorios.gorros&ts=1415909108906 > > [u'\n\t\t\t\t\t\t\t\tKnit wool-blend scarf\n\t\t\t\t\t\t\t'] > > > http://shop.mango.com/US/p0/women/accessories/knit-wool-blend-scarf/?id=33025533_02&ident=0_accesorio45%2C445_0_1415909108906&n=1&s=accesorios.gorros&ts=1415909108906 > > [s] Available Scrapy objects: > > [s] crawler <scrapy.crawler.Crawler object at 0x10d351950> > > [s] item {} > > [s] request <GET > http://shop.mango.com/US/p0/women/accessories/knit-wool-blend-scarf/?id=33025533_02&ident=0_accesorio45%2C445_0_1415909108906&n=1&s=accesorios.gorros&ts=1415909108906 > > > > [s] response <200 > http://shop.mango.com/US/p0/women/accessories/knit-wool-blend-scarf/?id=33025533_02&ident=0_accesorio45%2C445_0_1415909108906&n=1&s=accesorios.gorros&ts=1415909108906 > > > > [s] settings <scrapy.settings.Settings object at 0x10c79aa50> > > [s] spider <MangoTestSpider 'spider_mt' at 0x10d360250> > > [s] Useful shortcuts: > > [s] shelp() Shell help (print this help) > > [s] view(response) View response in a browser > > > In [*1*]: > > Do you really want to exit ([y]/n)? y > > > 2014-11-13 12:19:39-0800 [spider_mt] DEBUG: Scraped from <200 > http://shop.mango.com/US/p0/women/accessories/knit-wool-blend-scarf/?id=33025533_02&ident=0_accesorio45%2C445_0_1415909108906&n=1&s=accesorios.gorros&ts=1415909108906 > > > > {'name': [u'\n\t\t\t\t\t\t\t\tKnit wool-blend scarf\n\t\t\t\t\t\t\t']} > > 2014-11-13 12:19:39-0800 [spider_mt] DEBUG: Now parsing > http://shop.mango.com/US/p0/women/accessories/heel-lace-up-ankle-boots/?id=33065662_02&ident=0_accesorio48%2C448_0_1415909108906&n=1&s=accesorios.bisuteria&ts=1415909108906 > > [u'\n\t\t\t\t\t\t\t\tHeel lace-up ankle boots\n\t\t\t\t\t\t\t'] > > > http://shop.mango.com/US/p0/women/accessories/heel-lace-up-ankle-boots/?id=33065662_02&ident=0_accesorio48%2C448_0_1415909108906&n=1&s=accesorios.bisuteria&ts=1415909108906 > > [s] Available Scrapy objects: > > [s] crawler <scrapy.crawler.Crawler object at 0x10d351950> > > [s] item {} > > [s] request <GET > http://shop.mango.com/US/p0/women/accessories/heel-lace-up-ankle-boots/?id=33065662_02&ident=0_accesorio48%2C448_0_1415909108906&n=1&s=accesorios.bisuteria&ts=1415909108906 > > > > [s] response <200 > http://shop.mango.com/US/p0/women/accessories/heel-lace-up-ankle-boots/?id=33065662_02&ident=0_accesorio48%2C448_0_1415909108906&n=1&s=accesorios.bisuteria&ts=1415909108906 > > > > [s] settings <scrapy.settings.Settings object at 0x10c79aa50> > > [s] spider <MangoTestSpider 'spider_mt' at 0x10d360250> > > [s] Useful shortcuts: > > [s] shelp() Shell help (print this help) > > [s] view(response) View response in a browser > > > In [*1*]: > > Do you really want to exit ([y]/n)? y > > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Scraped from <200 > http://shop.mango.com/US/p0/women/accessories/heel-lace-up-ankle-boots/?id=33065662_02&ident=0_accesorio48%2C448_0_1415909108906&n=1&s=accesorios.bisuteria&ts=1415909108906 > > > > {'name': [u'\n\t\t\t\t\t\t\t\tHeel lace-up ankle boots\n\t\t\t\t\t\t\t']} > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Redirecting (302) to <GET > http://shop.mango.com/preHome.faces?state=she_400_US> from <GET > http://shop.mango.com/index.faces?state=she_400_US> > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/desc/women/accessories/belts/> (referer: > http://shop.mango.com/US/women/accessories/belts) > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Redirecting (302) to <GET > http://shop.mango.com/index.faces?state=she_400_US> from <GET > http://shop.mango.com/US/p0/women/accessories/wedge-suede-ankle-boots/?id=33083656_74&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906 > > > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/p0/women/accessories/crisscross-leather-sandals/?id=33080334_02&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906> > (referer: http://shop.mango.com/US/women/accessories/jewellery) > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Redirecting (302) to <GET > http://shop.mango.com/index.faces?state=she_400_US> from <GET > http://shop.mango.com/US/m/women/accessories/shoes/curve/?m=zapatos42%2C442%2C342&v=Violeta > > > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/p0/women/clothing/wedge-suede-ankle-boots/?id=33083656_74&ident=0__0_1415909108906&n=1&s=prendas.basicos&ts=1415909108906> > (referer: http://shop.mango.com/US/women/accessories/shoes) > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/desc/women/clothing/essentials/> (referer: > http://shop.mango.com/US/women/accessories/shoes) > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Crawled (200) <GET > http://shop.mango.com/US/asc/women/clothing/essentials/> (referer: > http://shop.mango.com/US/women/accessories/shoes) > > 2014-11-13 12:19:51-0800 [spider_mt] DEBUG: Now parsing > http://shop.mango.com/US/p0/women/accessories/crisscross-leather-sandals/?id=33080334_02&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906 > > [u'\n\t\t\t\t\t\t\t\tWedge suede ankle boots\n\t\t\t\t\t\t\t'] > > > http://shop.mango.com/US/p0/women/accessories/crisscross-leather-sandals/?id=33080334_02&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906 > > [s] Available Scrapy objects: > > [s] crawler <scrapy.crawler.Crawler object at 0x10d351950> > > [s] item {} > > [s] request <GET > http://shop.mango.com/US/p0/women/accessories/crisscross-leather-sandals/?id=33080334_02&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906 > > > > [s] response <200 > http://shop.mango.com/US/p0/women/accessories/crisscross-leather-sandals/?id=33080334_02&ident=0__0_1415909108906&n=1&s=accesorios&ts=1415909108906 > > > > [s] settings <scrapy.settings.Settings object at 0x10c79aa50> > > [s] spider <MangoTestSpider 'spider_mt' at 0x10d360250> > > [s] Useful shortcuts: > > [s] shelp() Shell help (print this help) > > [s] view(response) View response in a browser > > > In [*1*]: view(response) > > Out[*1*]: True > > > In [*2*]: response.xpath('//div[@itemprop="name"]/h1/text()').extract() > > Out[*2*]: [u'\n\t\t\t\t\t\t\t\tWedge suede ankle boots\n\t\t\t\t\t\t\t'] > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to scrapy-users+unsubscr...@googlegroups.com. > To post to this group, send email to scrapy-users@googlegroups.com. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- Rocío Aramberri -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
