W dniu 2011-06-14 16:11, Nick Burch pisze:
On Tue, 14 Jun 2011, Antoni Mylka wrote:
We'll need to buffer the whole file for zip either way. The current way
will create a temp file if you start with an input stream (not if you
have
a file already), will scan through the file looking for entries that'll
identify the file. The parser needs the whole file, so if we did a
streaming parse of the file for detection we'd need to have buffered
so we
can rewind for the parser
Why?
Doesn't the "we'll need to buffer the whole file for zip anyway" boil
down to the question of using the commons-compress ZipFile vs.
ZipArchiveInputStream?
No, it's because we need two different things to process the zip file.
Firstly there's the detector, and only once that has finished can the
parser have a go. Even if we did have streaming parsing, we need to
buffer the whole contents of the zip so we could rewind and let the
parser then look at the same zip. In that situation you probably might
as well do the buffering with the file, and then get the advantages of
zip parsing from a file that it offers.
But the ZipContainerDetector works by looking for the existence of
certain entries with known names and in certain cases it reads the
content of those entries (e.g. "[Content_Types].xml"). If those entries
happen to live at the beginning of the file then you can detect the type
by giving the detector a fixed header (of say 512KB). ZipFile seems to
work well enough with truncated zips, so if an interesting entry fits in
the part of the file you have, you're done. That's what I'm doing right
now. With a fixed-size header, I know how much needs to be buffered and
re-wound before extraction is applied. I accept the fact that there
exist zip files which will not be recognized correctly this way.
The only drawback is the necessity to create the temp file from this
small header. If you're already do lots of filesystem activity in your
app (crawling the files and generating an index), especially on a
desktop machine with a single non-SSD disk, then every fs operation you
don't make, allows the fs cache and disk head to do more stuff in other
places. Yet the profits will likely be small, and the cost of rewriting
the ZipContainerDetector seems large.
So in short what I want to say:
1. I want to do detection based on a fixed-size prefix of a zip file
with graceful degradation on weird files, where interesting entries
aren't at the beginning.
2. Tika can do that already (as far as I understand),
3. It will create a temp file, but this is a non-issue at the moment.
Antoni Myłka
[email protected]
