Hi, On Wed, Mar 19, 2008 at 11:33 PM, Thorsten Scherler <[EMAIL PROTECTED]> wrote: > Sounds good. We should add more elements for the link recognition, > though. > > I mean ATM we are looking for <a/> but for a crawler that scraps the > whole page all external resources are links and needs to be saved. > > Meaning for xhtml the following elements are important: > - <img src="..."/> > - <link href="..."/> > - <script src="..."/>
Good point! I'm wondering how we should best handle those in Tika, i.e. as <img/> and <script/> tags don't really have much meaning in the scope of text extraction. Perhaps we should map <img src="..." alt="..."/> to <a href="...">...</a> or something like that to keep the client view simple. Not sure what to do with <script/> tags, perhaps those links should go a metadata property? I don't think inline scripts should be part of the extracted text content (but others may disagree), so script links should probably also not be included in the XHTML output. > In css files there can as well be links to either images or other css > files. We don't currently have explicit CSS parser support in Tika, the plain text extractor comes closest. I'll see if we could add something that would allow easy detection of links within CSS files. On a related note, currently there's no base URI support in Tika. We probably should add that, and treat a dc:identifier URI (if available) as the base unless one has been explicitly specified in the document. Also, if a base URI is available, Tika should automatically make all relative URIs absolute to make client life easier. BR, Jukka Zitting
