Hey Mario, Not familiar with Scapy's inner workings. However, quite familiar with scraping in general from static/dynamic sites.
For your target, you seem to indicate that it's definitely a POST from the initial page. So, are you able to craft a "curl/wget" with the required cookies/referal/headers to generate the 2nd page from the 1st" Have you, or are you able to capture the actual traffice from browser/server with something like livehttpheaders (FF) or whatever network traffic capture app you use. If we/you can get this from the cmdline curl/wget, then it should be trivial to implement within scrapy.. Paste the above results/data and I'll help... On Thu, May 19, 2016 at 8:42 AM, Mario <[email protected]> wrote: > Hi guys, I’m scraping this page: https://www.chorus.co.nz/speed-dial > > In the search form if you enter address such as: 15 Tirotai Crescent, > Westmere, Auckland > > You would get suggestion of actual address like this: > > [image: suggestion_image] > > When clicked, new page would get loaded that contains 3 data fields that > I’m after: > > [image: data_fields_image] > > If I go to Network → XHR, 1 POST request will be displayed. If I try to > mimic that request I will not get page that I need. > > I know that I can use something like Selenium but that’s not ideal. > > Any help is appreciated, thanks! > > Mario > > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
