sramazzina commented on issue #2059: URL: https://github.com/apache/hop/issues/2059#issuecomment-5455472485
Hi @usbrandon, thanks for the patience on this one — it's been open for a while. Good news: this already works today, thanks to Apache VFS. Hop registers layered file system providers for `gz`, `zip`, `tar`, `tgz` and `bz2`, so File Metadata can read a compressed CSV without any change to the transform. You just point it at the file with a layered URI instead of a plain path. For gzip — which is probably what your Python script produces: ``` gz:file:///path/to/report.csv.gz!/report.csv ``` For a CSV inside a zip archive: ``` zip:file:///path/to/report.zip!/report.csv ``` The part after `!/` is the name of the file *inside* the archive. For zip it has to match exactly — it's case sensitive, and if the archive was created with a folder structure you need the full internal path (`!/data/report.csv`). `unzip -l yourfile.zip` shows you the exact name to use. Gzip is more forgiving, since a gzip stream only ever contains one file. We tested this against the transform's actual read path (charset detection and delimiter detection) and the results on a gzipped and a zipped CSV are identical to the ones on the plain file — same charset, same delimiter, same field types. It also works when the file name comes from an incoming field, so it fits an injection setup like the one you described: File Metadata reads the compressed file, and the Text File Input downstream handles the decompression itself with its own Compression setting. Two rough edges worth knowing about: - If you forget the `!/name` part and pass just the archive, you get `... !/ because it is not a file`, which isn't very self-explanatory — it means "tell me which file inside the archive you want". - If the internal name doesn't match, the transform doesn't fail: it emits an empty row and moves on, so it looks like nothing happened. Since compression itself is already covered, we'll use this issue to track a small fix limited to those two cases: a clearer message when an archive is passed without naming a file inside it, and a warning in the log instead of a silently empty row when the file can't be found. Nothing else changes in the transform. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
