Linkedin can be a tough site to scrape, as they generally don't want their data in other people's hands. You will need to use a user-agent switcher (you don't mention what UA you are sending), and most likely require a proxy in addition.
If you are looking to scrape the entirety of linkedin, it's > 30 million profiles. I've found it more economical to purchase a linkedin data dump from scrapinghub.com than to scrape it myself. On Mon, Mar 16, 2015 at 8:05 AM, DataScience <[email protected]> wrote: > Hi Scrapy Guys, > > Scrapy returns me an empty list while using shell to pick a simple "title" > field from this web page: http://goo.gl/dBR8P4 > I've used: > > - > > sel.xpath(‘//div[@id="results-rail"]/ul[@class="jobs"]/li[1]/div[@class="content"]/span/a[@class="title"]/text()’).extract() > - > > sel.xpath('html/body/div[3]/div/div[2]/div[2]/div[1]/ul/li[1]/div/span/a').extract() > - ... > > I verified the issue of the POST with XHR using firebug, and I think there > are no relationships with information generated using js code (what do you > think?). > > Can you please help me to figure out with this problem? > Thank you in Advance. > > Best Regards, > K. > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
