Leon:
On Feb 16, 12:57 pm, Leon <[email protected]> wrote: > Cerebrus' right, and some research on request types would do wonders. > > Some fiddling with the FMCSA website turned in some interesting > findings, tho. > > There's two parameters being passed via POST, pv_apcant_id and > pv_vpath. The HTTP request header follows: > > ---x--- > POSThttp://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail > HTTP/1.0 > > Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, > application/x-shockwave-flash, application/x-ms-application, > application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml > +xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, > application/msword, */* > Referer:http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist?n_do... > Accept-Language: us-en > Content-Type: application/x-www-form-urlencoded > UA-CPU: x86 > Proxy-Connection: Keep-Alive > User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET > CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR > 3.0.4506.2152; .NET CLR 3.5.30729) > Pragma: no-cache > Content-Length: 34 > Host: li-public.fmcsa.dot.gov > > pv_apcant_id=16499&pv_vpath=LIVIEW > ---x--- > > When i tried to apply them directly on the URL, it worked like a > charm. > > http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail?pv_... > > It happens because some request implementations (ASP comes to my mind) > checks both collections, QueryStrings and Forms, in order to produce a > result. Seems this site works like this. > > Hope it helps! > > Cheers, > - Leon > > On Feb 16, 8:31 am, Cerebrus <[email protected]> wrote: > > > > > There are a number of things that you need to understand about HTTP > > and request types (GET/POST), most of which cannot be explained in a > > discussion thread like this... you will need to research them. > > > POST requests do not append all form information in the URL as GET > > requests do. I want to clarify that it does not make a difference if > > the form parameters are hidden fields, since this appears to be the > > main aspect that's bothering you. This form uses a POST request and > > that is the reason why you don't see all the values appended to the > > URL. It's not because of the hidden fields. You should do the > > following: > > > 1. Take a look at this thread which provides a useful discussion on > > the subject of scraping -http://tinyurl.com/crj792 > > 2. Download and install Fiddler v2 and use it to plug into and analyze > > your HTTP request as the browser submits information from the first > > page to the second and so on. You will realize that it provides you > > with a list of parameters that are submitted on the request and that > > is what you need to replicate. > > 3. Find out what the Javascript function "submitReportRequest" does on > > your page. It is possible that it modifies the form values. > > 4. None of the hidden fields appear to transmit or upload a file. The > > value field is transmitted as a string. However, there may be other > > fields that we have not seen here, so investigate this. > > 5. Use POST requests if you don't want the full form information to be > > visible in the URL/address bar. > > > You should follow these steps and get back to us if you need more > > information or assistance. Hope this info is helpful !- Hide quoted text - > > - Show quoted text -
