Again
not .Net but classic ASP using the ServerXMLHTTP object, I used this code rcedit
to Dan Powederhill - apologies for pasting the whole lot here but the link
to the author's site doens't work
uses
something like this:
<%
set
myPage = new clsScreenScrape_Class
myPage.URL = "">myPage.FormVariables = "startdate=0.0.0&stopdate=0.0.0&IndexBox=XYZZY&Searchtype=1&search=" & server.urlencode(request.querystring("term"))
sSourceHTML = myPage.Source
set myPage = nothing
myPage.URL = "">myPage.FormVariables = "startdate=0.0.0&stopdate=0.0.0&IndexBox=XYZZY&Searchtype=1&search=" & server.urlencode(request.querystring("term"))
sSourceHTML = myPage.Source
set myPage = nothing
response.write(sSourceHTML)
%>
loads
of documentation on the MS site about the object - only problem I had was
working through a firewall proxy - you have to set the proxy settings somewhere
in the registry, can't remember where off the top of my head
<%
'************************************************************************************************************************
'*
'* Class Name : clsScreenScrape_Class
'* Author : Dan Powderhill
'* Email : [EMAIL PROTECTED]
'* Start Date : 31 March 2004
'* Last Update : 31 March 2004
'* URL : http://www.diado.com/classes/screenscrape
'*http://www.diado.com/classes/screenscrape
'************************************************************************************************************************
'*
'* Description : Provides a quick and easy method of screenscraping
'*
'* Usage Instructions : See above URL for full usage instructions
'*
'* Usage Restrictions : Please drop me a line if you intend to use this on any projects, commercial or otherwise, for no
'* other reason than I'd be interested to know where and how it's being used. Please do not sell
'* this code as your own, although you are free to package it with other work which you may be
'* selling. If you want to acknowledge me on the site you use this code on, that'd be nice, but it's
'* not mandatory. See http://www.diado.com/linktous for information on linking to Diado.com
'*
'************************************************************************************************************************
Class clsScreenScrape_Class
Public Version
Public URL
Public FormVariables
Public ResponseCode
Private INTERNAL_Source
Private objXML
Private Sub Class_Initialize()
Version = "0.1"
'Create the XMLHTTP object
set objXML = Server.CreateObject("MSXML2.ServerXMLHTTP")
' objXML.setProxy 0, "www-cache.reith.bbc.co.uk", ""
'************************************************************************************************************************
'*
'* Class Name : clsScreenScrape_Class
'* Author : Dan Powderhill
'* Email : [EMAIL PROTECTED]
'* Start Date : 31 March 2004
'* Last Update : 31 March 2004
'* URL : http://www.diado.com/classes/screenscrape
'*http://www.diado.com/classes/screenscrape
'************************************************************************************************************************
'*
'* Description : Provides a quick and easy method of screenscraping
'*
'* Usage Instructions : See above URL for full usage instructions
'*
'* Usage Restrictions : Please drop me a line if you intend to use this on any projects, commercial or otherwise, for no
'* other reason than I'd be interested to know where and how it's being used. Please do not sell
'* this code as your own, although you are free to package it with other work which you may be
'* selling. If you want to acknowledge me on the site you use this code on, that'd be nice, but it's
'* not mandatory. See http://www.diado.com/linktous for information on linking to Diado.com
'*
'************************************************************************************************************************
Class clsScreenScrape_Class
Public Version
Public URL
Public FormVariables
Public ResponseCode
Private INTERNAL_Source
Private objXML
Private Sub Class_Initialize()
Version = "0.1"
'Create the XMLHTTP object
set objXML = Server.CreateObject("MSXML2.ServerXMLHTTP")
' objXML.setProxy 0, "www-cache.reith.bbc.co.uk", ""
end sub
Public Property Let Source(sSource)
err.raise 9001, "clsScreenScrape", "Source property is read-only"
End Property
Public Property Get Source()
'Check we've been passed a URL - if not, raise and error
if len(URL) > 0 then
'If the FormVariables property has been populated, we need to submit the form data via POST.
if len(trim(FormVariables)) > 0 then
objXML.Open "POST", URL, false
' Indicate that the body of the request contains form data
objXML.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
'If the FormVariables property has not been populated then we need to request the URL using GET
else
objXML.Open "GET", URL, false
end if
'Send the request along with any form variables if they have been populated
objXML.Send(FormVariables)
'Store the response text from the request and return it as the property
INTERNAL_Source = objXML.ResponseText
Source = INTERNAL_Source
'Store the response code in the ResponseCode property
ResponseCode = objXML.Status
else
'No URL - raise error
err.raise 9002, "clsScreenScrape", "URL property has zero length"
end if
End Property
Private Sub Class_Terminate()
'Destroy the XMLHTTP object
set objXML = nothing
End Sub
end Class
%>
Public Property Let Source(sSource)
err.raise 9001, "clsScreenScrape", "Source property is read-only"
End Property
Public Property Get Source()
'Check we've been passed a URL - if not, raise and error
if len(URL) > 0 then
'If the FormVariables property has been populated, we need to submit the form data via POST.
if len(trim(FormVariables)) > 0 then
objXML.Open "POST", URL, false
' Indicate that the body of the request contains form data
objXML.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
'If the FormVariables property has not been populated then we need to request the URL using GET
else
objXML.Open "GET", URL, false
end if
'Send the request along with any form variables if they have been populated
objXML.Send(FormVariables)
'Store the response text from the request and return it as the property
INTERNAL_Source = objXML.ResponseText
Source = INTERNAL_Source
'Store the response code in the ResponseCode property
ResponseCode = objXML.Status
else
'No URL - raise error
err.raise 9002, "clsScreenScrape", "URL property has zero length"
end if
End Property
Private Sub Class_Terminate()
'Destroy the XMLHTTP object
set objXML = nothing
End Sub
end Class
%>
again,
apologies for the size of the posting, not sure what the house rules
are...
m
Matt
Hardiman
Systems Developer,
Radio News Operations, Radio News, BBC
News
( Work: 020 755 72696
2
Fax: 020
783 66348
* Room 237SE, BBC Bush House, London, WC2B 4PH
8
mailto:[EMAIL PROTECTED]
:
http://news.gateway.bbc.co.uk/radionews/operations/ - Intranet
:
http://connect.gateway.bbc.co.uk/Record.asp?IDN=72415 - BBC Connect profile
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mario Menti
Sent: 24 July 2006 17:20
To: [email protected]
Subject: Re: [backstage] Screen Scraping Advice ...
Just in case it helps..
Mario.
On 7/24/06, Murray, Simon
(IED) <[EMAIL PROTECTED]
> wrote:
Hi AllPlease forgive this off-topic post, but I am working a project which requires screen-scraping a variety of data from several third party websites and, after countless hours on google, am looking to be pointed in the right direction ... and I can't think of a more informed group of individuals to ask for assistance (creep, creep).Some time ago I wrote a simple screen scrape script in classic ASP using the Internet Transfer Protocol (InetCtls.Inet) which had it's limitations. I'm interested in using .Net and the HttpWebRequest class, but would welcome any guidance on the subject particularly when accessing data spanning across multiple pages.Thanks in advanceSimon
This is not an offer (or solicitation of an offer) to buy/sell the securities/instruments mentioned or an official confirmation. Morgan Stanley may deal as principal in or own or act as market maker for securities/instruments mentioned or may advise the issuers. This is not research and is not from MS Research but it may refer to a research analyst/research report. Unless indicated, these views are the author's and may differ from those of Morgan Stanley research or others in the Firm. We do not represent this is accurate or complete and we may not update this. Past performance is not indicative of future returns. For additional information, research reports and important disclosures, contact me or see https://secure.ms.com/servlet/cls . You should not use e-mail to request, authorize or effect the purchase or sale of any security or instrument, to send transfer instructions, or to effect any other transactions. We cannot guarantee that any such requests received via e-mail will be processed in a timely manner. This communication is solely for the addressee(s) and may contain confidential information. We do not waive confidentiality by mistransmission. Contact me if you do not wish to receive these communications. In the UK, this communication is directed in the UK to those persons who are market counterparties or intermediate customers (as defined in the UK Financial Services Authority's rules).
