I'm new to this, and tried to find help in other posts. The post from Duy
Nguyen https://groups.google.com/d/msg/scrapy-users/aVlsOz7972I Was pretty
close, but i'm still having issues getting the information I want.
I'm trying to scrape results from award miles for the united website at
http://www.united.com/web/en-US/apps/booking/flight/searchAward.aspx but
the results are being redirected and the following code is not able to get
the correct response.
from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.http import FormRequest, Request
class DmozSpider(Spider):
name = "united"
allowed_domains = ["united.com"]
start_urls = [
"http://www.united.com/";
]
def parse(self, response):
return [FormRequest(url=
"http://www.united.com/web/en-US/apps/booking/flight/searchAward.aspx";,
method='POST',
formdata={
'ctl00$ContentInfo$SearchForm$Airports1$Destination$txtDestination':'New
York/Newark, NJ (EWR - Liberty)',
'ctl00$ContentInfo$SearchForm$Airports1$Origin$txtOrigin':'San
Francisco, CA (SFO)',
'ctl00$ContentInfo$SearchForm$DateTimeCabin1$Depdate$txtDptDate':'8/19/2014'
,
'ctl00$ContentInfo$SearchForm$DateTimeCabin1$Depdate1$txtDptDate':
'8/27/2014',
'ctl00$ContentInfo$SearchForm$searchBy$SearchBy':
'rdosearchby3',
},
#dont_click=True,
callback=self.parse_award)]
def parse_award(self, response):
print "status: %s" % response.status
print "headers: %s" % response.headers
print "redirect_urls: %s" % response.meta
filename = response.url.split("/")[-2]
open('resp.html', 'wb').write(response.body)
The response to the post request says the status is 200 ('OK'), but when
using firebug for the same query, it shows the post status as 302 ('Moved
Temporarily'), and the results being available in
http://www.united.com/web/en-US/apps/booking/flight/searchResultAward1.aspx
Any help pointing in the right direction is appreciated!
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.
