Leon:
Thanks for taking the time to help me with this problem. I
really appreciate your advice.
Regards,
Wayne
On Feb 16, 12:57 pm, Leon <[email protected]> wrote:
> Cerebrus' right, and some research on request types would do wonders.
>
> Some fiddling with the FMCSA website turned in some interesting
> findings, tho.
>
> There's two parameters being passed via POST, pv_apcant_id and
> pv_vpath. The HTTP request header follows:
>
> ---x---
> POSThttp://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail
> HTTP/1.0
>
> Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
> application/x-shockwave-flash, application/x-ms-application,
> application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml
> +xml, application/vnd.ms-excel, application/vnd.ms-powerpoint,
> application/msword, */*
> Referer:http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist?n_do...
> Accept-Language: us-en
> Content-Type: application/x-www-form-urlencoded
> UA-CPU: x86
> Proxy-Connection: Keep-Alive
> User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET
> CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR
> 3.0.4506.2152; .NET CLR 3.5.30729)
> Pragma: no-cache
> Content-Length: 34
> Host: li-public.fmcsa.dot.gov
>
> pv_apcant_id=16499&pv_vpath=LIVIEW
> ---x---
>
> When i tried to apply them directly on the URL, it worked like a
> charm.
>
> http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail?pv_...
>
> It happens because some request implementations (ASP comes to my mind)
> checks both collections, QueryStrings and Forms, in order to produce a
> result. Seems this site works like this.
>
> Hope it helps!
>
> Cheers,
> - Leon
>
> On Feb 16, 8:31 am, Cerebrus <[email protected]> wrote:
>
>
>
> > There are a number of things that you need to understand about HTTP
> > and request types (GET/POST), most of which cannot be explained in a
> > discussion thread like this... you will need to research them.
>
> > POST requests do not append all form information in the URL as GET
> > requests do. I want to clarify that it does not make a difference if
> > the form parameters are hidden fields, since this appears to be the
> > main aspect that's bothering you. This form uses a POST request and
> > that is the reason why you don't see all the values appended to the
> > URL. It's not because of the hidden fields. You should do the
> > following:
>
> > 1. Take a look at this thread which provides a useful discussion on
> > the subject of scraping -http://tinyurl.com/crj792
> > 2. Download and install Fiddler v2 and use it to plug into and analyze
> > your HTTP request as the browser submits information from the first
> > page to the second and so on. You will realize that it provides you
> > with a list of parameters that are submitted on the request and that
> > is what you need to replicate.
> > 3. Find out what the Javascript function "submitReportRequest" does on
> > your page. It is possible that it modifies the form values.
> > 4. None of the hidden fields appear to transmit or upload a file. The
> > value field is transmitted as a string. However, there may be other
> > fields that we have not seen here, so investigate this.
> > 5. Use POST requests if you don't want the full form information to be
> > visible in the URL/address bar.
>
> > You should follow these steps and get back to us if you need more
> > information or assistance. Hope this info is helpful !- Hide quoted text -
>
> - Show quoted text -
