On Wed, 2008-03-19 at 11:07 +0200, Jukka Zitting wrote:
> Hi,
>
> On Tue, Mar 18, 2008 at 10:28 PM, Jukka Zitting <[EMAIL PROTECTED]> wrote:
> > Currently I think only the HTML parser preserves links as <A
> > href="..">...</A> tags, and you can catch them from the SAX events
> > generated by Tika. However, the HTML parser should really be producing
> > <a href="...">...</a> tags using the standard XHTML namespace.
>
> I fixed that with TIKA-128, so you can now use something like this:
>
> final List<String> links = new ArrayList<String>();
> handler = new TeeContentHandler(handler, new DefaultHandler() {
> public void startElement(
> String uri, String local, String name, Attributes attributes)
> {
> String ns = XHTMLContentHandler.XHTML;
> if (ns.equals(uri) && "a".equals(local)) {
> String href = attributes.getValue(ns, "href");
> if (href != null) {
> links.add(href);
> }
> }
> }
> });
> parser.parse(stream, handler, metadata);
>
> Perhaps we should turn that into a Tika utility class, something like
> org.apache.tika.sax.LinkContentHandler.
Sounds good. We should add more elements for the link recognition,
though.
I mean ATM we are looking for <a/> but for a crawler that scraps the
whole page all external resources are links and needs to be saved.
Meaning for xhtml the following elements are important:
- <img src="..."/>
- <link href="..."/>
- <script src="..."/>
In css files there can as well be links to either images or other css
files.
> Is it important just to get a
> list of link URIs, or would the link text also be interesting to a
> typical client?
For a crawler IMO only the link.
Thanks for the update I will have a closer look ASAP.
salu2
>
> BR,
>
> Jukka Zitting
--
Thorsten Scherler thorsten.at.apache.org
Open Source Java consulting, training and solutions
