Hi, On Tue, Mar 18, 2008 at 9:54 PM, Thorsten Scherler <[EMAIL PROTECTED]> wrote: > Essential part of a crawler is to extract links from the document it > parses. How do extracting out-links fit in tika? > > I mean I can use tika for populating the SAX-handler and extract the > links from there, or would it makes sense to do it in the parser?
Good point. So far we've mostly focused on extracting just the text content of the documents with indexing as the main use case in mind. However, the crawler use case is important and Tika should also support extraction of links from documents. Currently I think only the HTML parser preserves links as <A href="..">...</A> tags, and you can catch them from the SAX events generated by Tika. However, the HTML parser should really be producing <a href="...">...</a> tags using the standard XHTML namespace. We should add similar support also for the other parsers that can extract link information from documents. PS. There is also the ParserPostProcessor decorator that uses a regexp to detect links within the document text, and populates the "outlinks" metadata property with the detected links. I think we should replace that functionality with a ContentHandler decorator that does the same thing using <a href="..."/> tags. BR, Jukka Zitting
