Hi Jukka,
On Sep 27, 2009, at 3:31am, Jukka Zitting wrote:
On Fri, Sep 25, 2009 at 8:21 PM, Ken Krugler
<kkrugler_li...@transpac.com> wrote:
1. The handler's startElement() never gets called with the <base>
tag. I'm
assuming this is because <base> isn't part of the SAFE_ELEMENTS set.
But without the base tag, you can't correctly resolve relative URLs
in
anchor tags.
Seems like <base> should be part of the SAFE_ELEMENTS set.
Instead of passing the <base> element to the client (and thus
requiring it to keep track of it to correctly resolve local links),
I'd rather use it inside Tika to automatically turn local links to
absolute URLs in the parse output. Such a mechanism could also
leverage out-of-band information like a base URL given in the
RESOURCE_NAME_KEY metadata entry.
Makes sense. I'll file an issue and create a patch.
How as this set of tags derived?
The purpose of the SAFE_ELEMENTS set is to only pass out tags that may
be useful in inferring the semantic structure of the incoming HTML
document and to ensure that the output conforms to XHTML 1.0 Strict.
Note that in the future we may even start doing something like limited
JavaScript and CSS processing to "render" the incoming page to better
decide what outgoing XHTML structure best matches the content seen by
someone reading the page on a browser. This will help general purpose
Tika-based web crawlers to avoid SEO tricks like "display: none;" or
JavaScript DOM reorganization. So as a general rule clients should not
assume a one-to-one mapping between the HTML input and XHTML output
tags in Tika.
So recently I had to derive URLs from a web page, where the
information I needed was in the attribute of a <span> element.
It sounds like the direction Tika is going would mean I couldn't use
it for this use case.
I understand why the output might not match the input, which is fine.
But what's the benefit of stripping arbitrary tags?
And given the heavy use of CSS/Ajax, how is it possible to decide what
might or might not be useful for determining "semantic structure"?
E.g. "ignore footers" implies layout, and layout is often done using
CSS.
2. The handler's characters() method gets called with the following
text
Untitled
\n\n
link1
\n
link2
\n\n
\n
\n
The first six calls make sense to me.
The last two calls (with a single \n) happen just before
endElement("body")
is called, and this is unexpected.
From the offset in the buffer, passed to characters(), these are
the return
_after_ the </body> tag. If I put any number of returns in between
the
</body> and </html>, they all get passed to characters() before the
endElement("body") call. This seems like a bug.
Has anybody else noticed this?
No, but you're right in that it seems like a bug.
OK, I'll file a Jira issue.
Thanks,
-- Ken
