On Tue, 14 Jun 2011, Antoni Mylka wrote:
The way I see it, overhead appears if I pass a normal ZIP file, which I can process in a streaming way (limitations notwithstanding). Then when I pass the file to the container detector, it has to be buffered regardless of whether the buffering is necessary.
The detection requires checking potentially all the entries in the zip file. I don't think there's any way around that - if the container detector is to do any better than "yup, the outer package is a zip file" we have to check either for certain entries (random access) or check each one in turn (streaming). Only when that's finished can the parser have a go, and that'll want to be at the start too. So, buffering is needed always - it's always necessary.
So overhead appears when I detect the data type without text extraction AND when I detect data type on plain, non-ooxml zips which could be streamed.
If you know for certain that a zip file isn't one of the "zip using" formats (such as ooxml, odf, iworks etc), then don't pass it to the container detector!
The container detector is there for the cases when you don't know that for certain, and want to be able to do better than "the first 4 bytes look like a zip file"
Nick
