I have used TagSoup to parse the HTML and get the elements of interest. http://ccil.org/~cowan/XML/tagsoup/
On Mon, Jun 28, 2010 at 4:06 PM, Boris Aleksandrovsky <[email protected]> wrote: > I was wondering if any of you know of any open-source solutions for general > issues which arise in web crawling - how do you remove > headers/footers/javascript and generally cleanup html of a web-page before > indexing? We have a first-pass solution implemented using custom code, but > this must be a problem which a lot of people face, so I am asking here. > > Thanks, > Boris > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
