W dniu 2011-06-14 11:50, Arjohn Kampman pisze:
On 11/06/2011 02:22, Antoni Mylka wrote:
Brought our TikaMimeTypeIdentifier up to date with the latest Tika
trunk. I increased the number of bytes passed to the identifier to
512KB. It's a lot, but these days CPU is cheap. This large buffer will
allow for most small files to fit in their entirety.
Hi Antoni,
I don't understand this file size limitation. The way I would imagine
how the extraction of an MS Word document would/could work is something
like this:
- Aperture picks up a file.
- The mime type identifier investigates the file's header, detects an
Ole header and sets the item's mime type to "application/x-oleobject".
- Aperture passes the ole-tagged data to an Ole processor. This
processor detects a Word document, changes the item's mime type to
"application/msword" and forwards the data to a MSWord processor.
I suspect that the current mime type identifier also assumes the task of
interpreting the container format. Is that correct? If so, would it make
sense to modify this to the above behaviour?
This was a private mail, but I'm replying to both mailing lists, why
not. Mime type identification is at the core of what we all do.
You are right. There is still room for improvement. ZipContainerDetector
creates a temp file, which I'd rather avoid and with POI detector, the
entire stream is parsed once in detector, and for the second time in the
extractor/parser, which is bad for performance. The POI detector should
also be allowed to consume the ENTIRE stream, not the first few
kilobytes, as it is now.
Moreover, I have seen cases where the PoiFSContainerDetector didn't know
some ancient Quattro-Pro format, but the MimeTypes could recognize it
correctly on the basis of the name. That's why combining the
magic-based, zip-based and poi-based detection should follow a slightly
different logic.
So as it is now, in Aperture the TikaMimeTypeIdentifier is worse in
terms of performance (cpu (poi), and disk activity (zip)) than
MagicMimeTypeIdentifier. The poi detection is also limited to files
whose size is below a certain arbitrary threshold. Yet the detection
accuracy is still better.
Will see what I can do about the reservations outlined above.
Antoni Myłka
[email protected]
