The data you are looking for is generated dynamically and the data located in a javascript variable named window.photos.
Also, check my answer on how to figure out where is the content you are looking for: http://stackoverflow.com/questions/34815372/how-to-know-if-web-content-cannot-be-handled-by-scrapy/34822978#34822978 Best, Rolando On Fri, Aug 12, 2016 at 4:13 AM, 李哲 <imlegend...@gmail.com> wrote: > > <https://lh3.googleusercontent.com/-TicEc1iA578/V6121vT2akI/AAAAAAAAADM/m7DFhayq4Scd7R3_GvCbrntf15g8JMiIQCLcB/s1600/08%257E17_15-11-47.bmp> > > > I want to get a random picture url from 500px <https://500px.com/> everyday > moments ? > > So I run the scrapy shell https://500px.com/ to analysis the page, I > found that I can get the url info in the Chrome by using element inspector. > > But I can't find the related info in the response.body, for example I > search in the source code for the pic url which I get in the element > inspector, but I can't find that. Why this happened and how to fix it ? > > > > <https://lh3.googleusercontent.com/-ZGUIVFzLVeA/V6113hkjn3I/AAAAAAAAADE/eZT-G07rUccfmzNJKgVMTjrx8la6lxQbACLcB/s1600/08%257E17_15-06-22.bmp> > > > > <https://lh3.googleusercontent.com/-TicEc1iA578/V6121vT2akI/AAAAAAAAADM/m7DFhayq4Scd7R3_GvCbrntf15g8JMiIQCLcB/s1600/08%257E17_15-11-47.bmp> > > > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to scrapy-users+unsubscr...@googlegroups.com. > To post to this group, send email to scrapy-users@googlegroups.com. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscr...@googlegroups.com. To post to this group, send email to scrapy-users@googlegroups.com. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
