Hi, On Thu, Mar 20, 2008 at 11:56 PM, Thorsten Scherler <[EMAIL PROTECTED]> wrote: > You said in another mail that the outlinks are stored in a metadata > object. Why not store it in an exclusive object for it. I guess besides > the depth variable (which does not make an awful lot of sense for tika)
I'd prefer to keep the Tika interfaces as general as possible and avoid special cases that only serve a single use case. If there's a reasonable way for a parser to report outgoing links through the existing API, then I think we should use it. We can then of course add a utility class that takes such information and presents it in a more convenient way to a specific use case or client, but the basic mechanism should be as generic as possible. For example, while (AFAIUI) you're only interested in getting the set of outgoing URIs in a document, some other crawler/indexers might also want to know the link text and perhaps even the surrounding words as context information to use when indexing the target document. Instead of extending an explicit Outlink class with such information (it's not clear what all information would be needed), I'd rather use the existing XHTML SAX stream for that. > > I don't think inline scripts should be part of > > the extracted text content (but others may disagree), so script links > > should probably also not be included in the XHTML output. > > I agree. However AJAX is becoming more popular and some page content > only can be reached via scripts. Not sure where this leaves us. We could perhaps add a configuration option to the HTML parser to keep or drop any inline scripts, comments, and/or style sheets. See also the other thread for potential syntax-aware parsing of JavaScript and CSS. We could turn the HTML parser into a composite parser class that calls other language-specific parsers when it encounters inline content in some specific language. > > We don't currently have explicit CSS parser support in Tika, the plain > > text extractor comes closest. I'll see if we could add something that > > would allow easy detection of links within CSS files. > > http://svn.apache.org/repos/asf/forrest/trunk/main/webapp/resources/chaperon/ > In forrest we are using chaperon for this, but not sure whether tika > should parse css files. Is similar to inline scripts, or? Seems useful! Anything that goes beyond text/plain is good... > > On a related note, currently there's no base URI support in Tika. We > > probably should add that, and treat a dc:identifier URI (if available) > > as the base unless one has been explicitly specified in the document. > > Also, if a base URI is available, Tika should automatically make all > > relative URIs absolute to make client life easier. > > IMO that should be the problem of the client because there are situation > where the client may prefer relative links. Good point. Perhaps another place where we should make the parser configurable, as other clients would probably be better served with "cooked" links so they wouldn't need to worry about base URIs etc. BR, Jukka Zitting
