akshatshenoi-db opened a new pull request, #57615: URL: https://github.com/apache/spark/pull/57615
### What changes were proposed in this pull request? This extends the archive-reader feature to the ORC data source, so a `.tar`/`.tar.gz`/`.tgz`/`.zip`/`.7z` archive of ORC files is read as if it were a directory of the entries it contains, gated by `spark.sql.files.archive.reader.enabled` (default false). It follows the series that added archive support to CSV (SPARK-57135, SPARK-57321), JSON (SPARK-57419), text (SPARK-57478), XML (SPARK-57479), Avro (SPARK-57481), the zip container (SPARK-57705), the `SupportsArchiveFormat` trait extraction (SPARK-58110), and Parquet (SPARK-57590). ORC is a random-access format like Parquet: a footer read needs the whole file on disk, so it cannot be streamed entry-by-entry. It consumes the random-access surface already on the `SupportsArchiveFormat` trait (added by the Parquet port): - `OrcFileFormat` mixes in `SupportsArchiveFormat`, returns `archiveEntryFilter = true` (ORC part-files are often extensionless, so every entry is read), reports `isSplitable = false` for an archive path, and in `buildReaderWithPartitionValues` dispatches an archive input to `readLocalizedEntries`, which unpacks each entry to a temp file and runs the normal single-file reader over it. - `OrcUtils` routes schema inference through the archive: the non-merge path samples the first entry (`readArchiveSchemas(stopAtFirst = true)`, matching ORC's existing sample-one behavior, SPARK-11500), and the `mergeSchema` parallel path folds every entry (whole-archive-atomic on a corrupt entry). A corrupt archive surfaces the same `cannotReadFooterForFileError` a corrupt loose file does. The shared test base is refactored so the localize-path tests (vectorized-reader read parity, abandoned-read/corrupt-archive temp-dir cleanup, extensionless entries, `mergeSchema` field union) live in `ArchiveReadSuiteBase` behind `localizesEntries`, keyed on new `archiveTempDirPrefix` / `vectorizedReaderConfKey` hooks, instead of being duplicated per random-access format. `ParquetArchiveReadBase` is slimmed to set those hooks; `OrcArchiveReadBase` reuses them. ### Why are the changes needed? ORC was the remaining columnar format without archive-read support; this brings it to parity with Parquet and the streaming formats, so users can read archived ORC datasets without unpacking them first. Folding the shared random-access tests into the base avoids duplicating them for every columnar format. ### Does this PR introduce _any_ user-facing change? Yes. When `spark.sql.files.archive.reader.enabled` is true (default false), the ORC data source reads supported archives as directories of their entries, for both scan and schema inference. With the flag off (the default) there is no behavior change. ### How was this patch tested? New `OrcTarArchiveReadSuite`, `OrcZipArchiveReadSuite`, and `OrcSevenZArchiveReadSuite` run the shared `ArchiveReadSuiteBase` matrix over ORC (read parity with a directory, schema inference, vectorized on/off, corrupt/missing handling, `mergeSchema` union, temp-dir cleanup). The refactored localize-path tests continue to run for Parquet via `ParquetArchiveReadBase`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
