Hi Paul, Thanks a lot, your response was very helpful indeed.
Good day, Arye. On Wednesday, March 12, 2014 1:20:19 AM UTC+2, Paul Tremberth wrote: > > Hi Aryé, > > That's because the website is using Javascript to build most of the page. > The elements you want do not appear in the HTML source of the page. > You can check by disabling Javascript and reloading the page. > > To get the HTML your browser creates with Javascript, you could use a > javascript renderer like splash (https://github.com/scrapinghub/splash) > or analyse network activity in your browser's Developer Tools to > understand where the data in the final page comes from. > > Hope this helps. > /Paul > > On Tuesday, March 11, 2014 11:04:14 PM UTC+1, Aryé H wrote: >> >> >> >> Dear list, >> >> I am new to scrapy and would greatly appreciate if someone can explain >> why the first two selectors in the example below return an empty list ? >> I have to go up 2 levels to get a result returned. >> >> >> >> >> ⟫ scrapy shell http://www.snagfilms.com/films/browse/all/offset/0 >> >> >>> sel.xpath('//*[@id="rightSectionBrowse"]') >> >> >> >> >> [] >> >>> sel.xpath('/html/body/div[2]/div/div[2]/div[2]/div[2]') >> >> >> >> [] >> >>> sel.xpath('/html/body/div[2]/div/div[2]') >> >> >> >> [<Selector xpath='/html/body/div[2]/div/div[2]' data=u'<div >> class="module browse-landing-module'>] >> >>> >> >> >> >> These xpath expressions that fail to return data are the minimal and >> absolute paths to the same HTML element highlighted below in the collapsed >> HTML image as copied from firebug. >> >> >> >> <https://lh6.googleusercontent.com/-qQKPiFYfuhw/Ux-FobDVZnI/AAAAAAAAIlY/3UVOvHVzsrU/s1600/collapsedHTML.png> >> >> >> Regards, >> >> Arye. >> >> -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
