pjfanning opened a new pull request, #1363: URL: https://github.com/apache/poi/pull/1363
Fixes heap regressions from recent zip-entry changes that showed up in poi-integration. ### Problem changes 1. **#1236** (main cause): with a temp-file threshold set, unknown-size entries (data-descriptor entries, e.g. those POI writes to a stream) were kept on the heap up to the full threshold instead of going straight to a temp file. `ZipInputStreamZipEntrySource` holds every entry until close, so with a 16MB threshold each such part could sit on the heap at up to 16MB. 2. **#1227 → #1235**: known-size entries over 2MB grow from a 2MB buffer through commons-io's chunked buffer (2+4+8+… MB) plus a final `toByteArray()` copy, e.g. a ~24MB peak for a 10MB entry, where it used to be 20MB. 3. **#1277** hid (1) by lowering `TestAllFiles`' threshold from 16MB to 2MB; users who set a threshold still saw it. ### Changes - `ZipArchiveFakeEntry`: unknown-size entries are buffered in memory up to `min(threshold, 2MB)`, then spilled. The spill log moves from WARN to INFO, since spilling is now routine for large parts (it matches the known-size temp-file log). - `IOUtils.toByteArray` (known length): read straight into one array that grows by doubling, capped at the declared length. A complete read ends in an exactly-sized array with no final copy, and the peak stays below 2× the payload. The 2MB cap on the first allocation from an untrusted size is kept. - `TestAllFiles`: restore the 16MB threshold. ### Measurements (trunk vs this branch) | | trunk | this PR | |---|---|---| | open 8 × 10MB unknown-size parts, 16MB threshold: retained heap | 84.8MB | 1.1MB | | same: allocated | 226MB | 29MB | | 3MB known-size read: allocated | 9.0MB | 5.0MB | | 10MB known-size read: peak live | ~24MB | ~18MB | poi-integration (`CI=true`, 16MB threshold) ran locally with no OOM or GCLocker stalls. The one failure was a local `FileNotFoundException (Operation not permitted)` writing an HPSF temp copy of `ForShifting.xls`, which this change doesn't touch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
