Cerebrus' right, and some research on request types would do wonders. Some fiddling with the FMCSA website turned in some interesting findings, tho.
There's two parameters being passed via POST, pv_apcant_id and pv_vpath. The HTTP request header follows: ---x--- POST http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail HTTP/1.0 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml +xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* Referer: http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist?n_dotno=204961&s_prefix=MC&n_docketno=&s_legalname=&s_dbaname=&s_state= Accept-Language: us-en Content-Type: application/x-www-form-urlencoded UA-CPU: x86 Proxy-Connection: Keep-Alive User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Pragma: no-cache Content-Length: 34 Host: li-public.fmcsa.dot.gov pv_apcant_id=16499&pv_vpath=LIVIEW ---x--- When i tried to apply them directly on the URL, it worked like a charm. http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail?pv_apcant_id=16499&pv_vpath=LIVIEW It happens because some request implementations (ASP comes to my mind) checks both collections, QueryStrings and Forms, in order to produce a result. Seems this site works like this. Hope it helps! Cheers, - Leon On Feb 16, 8:31 am, Cerebrus <[email protected]> wrote: > There are a number of things that you need to understand about HTTP > and request types (GET/POST), most of which cannot be explained in a > discussion thread like this... you will need to research them. > > POST requests do not append all form information in the URL as GET > requests do. I want to clarify that it does not make a difference if > the form parameters are hidden fields, since this appears to be the > main aspect that's bothering you. This form uses a POST request and > that is the reason why you don't see all the values appended to the > URL. It's not because of the hidden fields. You should do the > following: > > 1. Take a look at this thread which provides a useful discussion on > the subject of scraping -http://tinyurl.com/crj792 > 2. Download and install Fiddler v2 and use it to plug into and analyze > your HTTP request as the browser submits information from the first > page to the second and so on. You will realize that it provides you > with a list of parameters that are submitted on the request and that > is what you need to replicate. > 3. Find out what the Javascript function "submitReportRequest" does on > your page. It is possible that it modifies the form values. > 4. None of the hidden fields appear to transmit or upload a file. The > value field is transmitted as a string. However, there may be other > fields that we have not seen here, so investigate this. > 5. Use POST requests if you don't want the full form information to be > visible in the URL/address bar. > > You should follow these steps and get back to us if you need more > information or assistance. Hope this info is helpful !
