Hi Antoni, Awesome! Thanks for the FYI here...
Cheers, Chris On Jun 10, 2011, at 5:22 PM, Antoni Mylka wrote: > W dniu 2010-12-03 01:07, Antoni Mylka pisze: >> Hello Aperture >> >> (cc tika-dev, may be interesting for you too) > > Brought our TikaMimeTypeIdentifier up to date with the latest Tika > trunk. I increased the number of bytes passed to the identifier to > 512KB. It's a lot, but these days CPU is cheap. This large buffer will > allow for most small files to fit in their entirety. > > With this, the container aware detection really started to work. OOXML, > ODT and MS-Office-based formats can now be recognized without the file > name as long as file is small. In my use cases, most of the unnamed .doc > files I meet come in email attachments or embedded in other docs. They > tend to be small and will fit in 512KB, so that's where the most bang > for the buck is to be found. > >> - support for string patterns in UTF-16 documents. E.g. Tika can't >> recognize XML, or HTML in a full UTF-16 file > > Still doesn't work. Do such documents occur in the wild at all? Anyone > with more experience in CJK? Do CJK websites use UTF-16? I myself > haven't seen one. > >> - support for allowsWhiteSpace before a pattern, e.g. Tika had >> problems recognizing the<html> tag if there is some whitespace in >> front of it (now it works around that limitation in a good enough way >> though, so it's actually not a problem) > > workaround works > >> - support for multiple parent types. >> - quattro pro 6 used a wordperfect magic, while later ones used >> office magics, >> - older Corel Presentations used wordperfect magic, newer use office, >> - works spreadsheets 3.0 used a wordperfect magic, 4.0 used their >> own format, 7.0 uses office >> The problem with Tika, is that it treats all those cases correctly >> when only the name is provided, but when both name and bytes are >> provided, the byte-based mime type trumps the name-based mime type, >> because name-based is not a specialization of byte-based (because one >> type can only have a single parent, so if we say that office is the >> parent of works, we won't recognize works 3.0 and 4.0 but only 7.0). > > Still doesn't work, will see what I can do about that. It's about > historical formats though, which rarely occur in practice. We don't have > Extractors for them anyway, so it's not much of a real problem. > >> - getExtensionsFor(String mimeType), useful in many apps, in tika the >> the mime knowledge base is hidden in private fields and >> package-protected classes > > This has been actually implemented last month. I wrote a > getExtensionsFor method in TikaMimeTypeIdentifier. > > With this I would like to "officially" deprecate the > MagicMimeTypeIdentifier in favour of the TikaMimeTypeIdentifier. If > nobody objects, I'd add the @deprecated javadoc annotation in near future. > > See http://bit.ly/iTqCs0 for details of what we can do now. > > Tika, you rule. > > Antoni Myłka > [email protected] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
