W dniu 2010-12-03 01:07, Antoni Mylka pisze:
Hello Aperture
(cc tika-dev, may be interesting for you too)
Brought our TikaMimeTypeIdentifier up to date with the latest Tika
trunk. I increased the number of bytes passed to the identifier to
512KB. It's a lot, but these days CPU is cheap. This large buffer will
allow for most small files to fit in their entirety.
With this, the container aware detection really started to work. OOXML,
ODT and MS-Office-based formats can now be recognized without the file
name as long as file is small. In my use cases, most of the unnamed .doc
files I meet come in email attachments or embedded in other docs. They
tend to be small and will fit in 512KB, so that's where the most bang
for the buck is to be found.
- support for string patterns in UTF-16 documents. E.g. Tika can't
recognize XML, or HTML in a full UTF-16 file
Still doesn't work. Do such documents occur in the wild at all? Anyone
with more experience in CJK? Do CJK websites use UTF-16? I myself
haven't seen one.
- support for allowsWhiteSpace before a pattern, e.g. Tika had
problems recognizing the<html> tag if there is some whitespace in
front of it (now it works around that limitation in a good enough way
though, so it's actually not a problem)
workaround works
- support for multiple parent types.
- quattro pro 6 used a wordperfect magic, while later ones used
office magics,
- older Corel Presentations used wordperfect magic, newer use office,
- works spreadsheets 3.0 used a wordperfect magic, 4.0 used their
own format, 7.0 uses office
The problem with Tika, is that it treats all those cases correctly
when only the name is provided, but when both name and bytes are
provided, the byte-based mime type trumps the name-based mime type,
because name-based is not a specialization of byte-based (because one
type can only have a single parent, so if we say that office is the
parent of works, we won't recognize works 3.0 and 4.0 but only 7.0).
Still doesn't work, will see what I can do about that. It's about
historical formats though, which rarely occur in practice. We don't have
Extractors for them anyway, so it's not much of a real problem.
- getExtensionsFor(String mimeType), useful in many apps, in tika the
the mime knowledge base is hidden in private fields and
package-protected classes
This has been actually implemented last month. I wrote a
getExtensionsFor method in TikaMimeTypeIdentifier.
With this I would like to "officially" deprecate the
MagicMimeTypeIdentifier in favour of the TikaMimeTypeIdentifier. If
nobody objects, I'd add the @deprecated javadoc annotation in near future.
See http://bit.ly/iTqCs0 for details of what we can do now.
Tika, you rule.
Antoni Myłka
[email protected]
