mattcasters opened a new issue, #7710: URL: https://github.com/apache/hop/issues/7710
### Feature description Add a dedicated **Local Unzip** (or **Streaming Unzip**) workflow action optimized for large zip archives that contain **hundreds of thousands to millions of small files** (e.g. bulk dumps like [SEC EDGAR submissions.zip](https://www.sec.gov/Archives/edgar/daily-index/bulkdata/submissions.zip)). This would complement the existing **Unzip** action, which correctly uses Apache Commons VFS (`zip:…`) and is the right default for remote/layered VFS sources, wildcards across VFS, etc.—but pays a high fixed cost when the archive is a plain local file with a huge entry count. Related: #2235 (per-entry zip filesystem close made Unzip catastrophically slow; fixed by releasing the zip filesystem once). After that fix, Unzip is usable again (~order of ~1000 files/s in one measurement), but the design still materializes the entire archive as VFS objects before extracting. ### Motivation / problem The current Unzip action roughly does: 1. Open `zip:` + path via Hop VFS 2. `zipFile.findFiles(AllFileSelector)` → **`FileObject[]` of every entry** 3. Loop: extract each entry Opening a VFS zip layer already forces Commons VFS `ZipFileSystem.init()` to: - Parse the zip central directory once (fine) - Build a **full in-memory tree**: one `ZipFileObject` + `FileName` per entry, parent links, `HashMap` cache, root child set Then `findFiles` **walks that tree again** and materializes a second structure (`List` → `FileObject[]`) holding references to **all** entries before the first content byte is written. For ~1M small files that means: | Cost | Effect | |------|--------| | Heap | Hundreds of MB+ of objects only needed one-at-a-time | | GC | Extra pressure during extract | | Latency before first file | Seconds–tens of seconds of pure index/tree work | | Peak memory | Tree cache **and** full `items[]` at the same time | A normal unzip tool only needs the central directory (or a sequential stream) and processes **one entry at a time**. Holding every entry as a VFS `FileObject` is convenience for the virtual-filesystem model, not a requirement of “extract this zip to a folder.” A larger copy buffer would not help much for tiny JSON/text members; this is about **structure and memory**, not per-byte I/O. ### Proposed solution New action (name TBD), e.g. **Local Unzip** / **Streaming Unzip**, that: 1. Targets a **local** (or locally staged) zip file 2. Iterates with `java.util.zip.ZipFile` (or `ZipInputStream` for pure sequential use) 3. For each `ZipEntry`: wildcard match on name → exist policy → create path → stream copy via `HopVfs` (or NIO) for the target 4. Closes the archive **once** at the end Sketch: ```text ZipFile → for each ZipEntry → match wildcards on entry name → resolve target path / if-file-exists policy → copy stream → close once ``` Still parse the central directory **once** (normal for `ZipFile`). Do **not** build a million VFS nodes or a million-long `FileObject[]`. Keep existing **Unzip** for full VFS/`zip:` semantics (remote, nested layers, etc.). ### Suggested feature set (v1) Align enough with current Unzip to be useful as a drop-in for the “huge local zip” case: - Source zip path (variables) - Target directory + create folder - Entry include/exclude wildcards (base name and/or path) - If file exists: skip / overwrite / fail (size-based policies optional later) - Optional “create root folder from zip name” - Optional set original modification time from entry - After unzip: do nothing / delete zip / move zip - Success conditions (no errors / error limit / min files) - **Do not** add all extracted paths to result filenames by default (1M result entries is its own problem); optional and documented as expensive Document clearly: prefer this action for **local** archives with very large entry counts; use classic Unzip when you need general VFS layering. ### Alternatives considered 1. **Only optimize existing Unzip** (replace `findFiles` with `ZipFile` iteration when the source is local) - Pros: one action - Cons: dual code paths, harder testing, risk of subtle VFS vs local differences; a separate action makes the trade-off explicit 2. **Leave as-is and document “use external unzip for huge archives”** - Acceptable workaround, but Hop should handle common bulk-data zips well after #2235 3. **Parallel extract** - Possible later; orthogonal to streaming/materialization. v1 should stay single-threaded and correct ### Acceptance criteria (draft) - [ ] New workflow action with GUI, docs, samples, i18n stubs - [ ] Extracts a local zip with tens/hundreds of thousands of small files without allocating an entry `FileObject[]` / full VFS zip tree for all members - [ ] Peak heap stays roughly proportional to concurrent work (one/few entries), not O(entry count) VFS objects - [ ] Wildcards + basic if-file-exists + target folder behavior covered by unit tests - [ ] Docs cross-link Unzip vs Local/Streaming Unzip and reference #2235 ### Issue Priority Priority: 3 ### Issue Component Component: Actions / VFS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
