I need help. My users have to type information found on a website into a database. I am trying to automate this process for them. I have been able to “scrape” the information from the first page using C#.net web services and insert it into the MS Access database. The user types the MC Number (Motor Carrier Number) of the trucker and the code goes to the website scrapes the information needed and puts it in the DB. The first page was easy. The URL I used in my code is fully displayed in the web browser. I just used that information and a variable to get the web page from my code.
e.g. “http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist? n_dotno=204961&s_prefix=MC&n_docketno=&s_legalname=&s_dbaname=&s_state=” However, I am stymied on the second page because the URL information appears to be partially hidden. For example the URL that links from the above noted page appears in the web browser as “http://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_getdetail” This does not work when used as is. I opened the web page source and it appears that there is “hidden” code or a file that is transmitted to an Oracle server (see below): ********** Button Code (For HTML Report) ************ <BR> <FORM ACTION="pkg_carrquery.prc_getdetail" METHOD="POST"> <INPUT TYPE="hidden" NAME="pv_apcant_id" VALUE="16499"> <INPUT TYPE="hidden" NAME="pv_vpath" VALUE="LIVIEW"> <input type="submit" value="HTML" onClick=""> </FORM> </font></center></td> <td headers="view_details"><center><font size="-1" face="Arial, Helvetica"> <BR> <FORM ACTION="http://li-public.fmcsa.dot.gov/reports/rwservlet"; METHOD="POST" name="reportForm" onSubmit="submitReportRequest (this.rptSummit,this)"> <INPUT TYPE="hidden" NAME="hidden_run_parameters" VALUE="lirpt"> <INPUT TYPE="hidden" NAME="report" VALUE="/u01/oracle/lirpts/ li_carrier.rdf"> <INPUT TYPE="hidden" NAME="p_apcant" VALUE="16499"> <INPUT TYPE="hidden" NAME="p_user" VALUE="WEBLIVIEW"> <INPUT TYPE="submit" VALUE="Report" name="rptSummit"> </FORM> </td> I checked the Oracle website and various Internet sites and I have tried many different combinations, but I be darned if I can figure it out. I have also tried using various free screen scrapers to try to find the “full” URL information – without any luck. Can anyone help? Note also: This is a public website containing information about truckers licensed to haul goods in the United States.
