"Murray, Simon \(IED\)" <[EMAIL PROTECTED]> writes:
> Hi All
>
> Please forgive this off-topic post, but I am working a project which
> requires screen-scraping a variety of data from several third party
> websites and, after countless hours on google, am looking to be pointed
> in the right direction ... and I can't think of a more informed group of
> individuals to ask for assistance (creep, creep).
>
> Some time ago I wrote a simple screen scrape script in classic ASP using
> the Internet Transfer Protocol (InetCtls.Inet) which had it's
> limitations. I'm interested in using .Net and the HttpWebRequest class,
> but would welcome any guidance on the subject particularly when
> accessing data spanning across multiple pages.
I've done quite a bit of screen scraping using just XSLT.
My favoured XSLT is libxslt which sits on top of libxml2 (mainly for
linux but you can get it for the other operating systems).
libxslt does massaging of HTML to well formed XML. All XSLT engines
will allow you to directly read HTML over HTTP if you want to and do
so inside the XSLT process, eg: like this:
<xsl:variable name="aunty" select="document('http://www.bbc.co.uk')"/>
One technique I use a lot is to XSLT a file into a CSV format and then
process it further with simple shell tools. This means identifying the
various fields that you want to form a record. Don't worry about
normalization - it doesn't matter for tasks like this.
When I finally have a tool that I like I tend to then parcel it up
replacing the shell bits with a "real" programming language but
keeping the XSLT.
The most frustrating part of any screen scrape with XSLT is actually
finding the xpath of the point in the document where you're gonna
extract from.
This is hard.
It's best when the page provider has tagged the data you want with a
class or something. That actually happens quite a lot now because
people are using them for CSS. The designer's CSS gain can also be the
screen scrapers gain.
Indeed, as soon as things are classes semantically, I would argue that
it's not screen scraping anymore. It's just data processing.
One other thing: I would definitely try and separate the processing
from the presentation. The processing can fail for all sorts of
reasons... you don't want requests for the presentation to be causing
failures all the time... you want to be in control of the scrape.
Of course, that isn't always possible. I wrote a screen scrape login
authenticator for Natwest's online banking app. Clearly one wouldn't
want to cache anything with that /8->
--
Nic Ferrier
http://www.tapsellferrier.co.uk for all your tapsell ferrier needs
-
Sent via the backstage.bbc.co.uk discussion group. To unsubscribe, please
visit http://backstage.bbc.co.uk/archives/2005/01/mailing_list.html.
Unofficial list archive: http://www.mail-archive.com/[email protected]/
