On Tue, 14 Jun 2011, Antoni Mylka wrote:
We'll need to buffer the whole file for zip either way. The current way
will create a temp file if you start with an input stream (not if you have
a file already), will scan through the file looking for entries that'll
identify the file. The parser needs the whole file, so if we did a
streaming parse of the file for detection we'd need to have buffered so we
can rewind for the parser
Why?
Doesn't the "we'll need to buffer the whole file for zip anyway" boil
down to the question of using the commons-compress ZipFile vs.
ZipArchiveInputStream?
No, it's because we need two different things to process the zip file.
Firstly there's the detector, and only once that has finished can the
parser have a go. Even if we did have streaming parsing, we need to buffer
the whole contents of the zip so we could rewind and let the parser then
look at the same zip. In that situation you probably might as well do the
buffering with the file, and then get the advantages of zip parsing from a
file that it offers.
Pass in a TikaInputStream. That supports attaching the opened (and
processed) container to the stream, so the parser can re-use it.
I know. I was referring to making the Aperture extractors aware of the fact
that they can reuse the NPOIFileSystem, which is something I want to
implement before we fully migrate. For us, the problem is that we give only
the first few KB of a file to the mime type identifier, therefore for larger
files the PoiContainerDetector can NOT build a proper poi filesystem for
extractors to reuse.
Correct. As there's no requirement for the properties table to live at the
front of the file, you can't even do nasty shortcut hacks. If you want to
know what's in the ole2 file, you do have to load the whole thing. At
least NPOIFSFileSystem makes this quicker and easier than it used to be!
Nick
