Hello!
14.06.2011 17:53, Antoni Mylka пишет:
Doesn't the "we'll need to buffer the whole file for zip anyway" boil down to
the question of using the commons-compress ZipFile vs. ZipArchiveInputStream? I
know that in a general case the zip file format isn't well suited for streaming
processing, which makes ZipArchiveInputStream less reliable. The stream can
contain entries which aren't supposed to appear in the zip or multiple entries
with the same name. Yet if I agree to that, I can crawl 50M zips in email
attachments without copying them.
You are already committed to using ZipFile in zip-processing code so using
TikaInputStream.getFile() in ZipContainerDetector is not a problem. We stay with
ZipArchiveInputStream (for the time being) and would therefore be interested in
a stream-based ZipContainerDetector consuming just a few kilobytes, knowing that
in certain cases the accuracy may drop, because the entries in a zip are in
general unordered.
It's a reliability vs. performance tradeoff. Or am I missing something?
Tika detects datatype and extracts text in one pass through supplied input stream.
OOXML parser requires random access to ZIP archive files, so there is only two
alternativies - to buffer data in memory or store it on disk. Overhead appears
only when you need just to detect data type without text extraction.
best wishes, Max
