akshatshenoi-db opened a new pull request, #57808: URL: https://github.com/apache/spark/pull/57808
### What changes were proposed in this pull request? This PR adds a new file-source option `archivePathFilter`: a glob that selects which **inner entries** of an archive are ingested, matched against each entry's full path within the archive, so patterns like `subdir/*` and `*/*.csv` work. It continues the archive-read series (SPARK-57135 / SPARK-57321 CSV, SPARK-57419 JSON, SPARK-57478 text, SPARK-57479 XML, SPARK-57481 Avro, SPARK-57590 Parquet, SPARK-57591 ORC, SPARK-58382 binaryFile, SPARK-58110 trait extraction). Details: - `FileSourceOptions` gains the `archivePathFilter` option (the glob string) plus `archivePathFilterPattern`, a `@transient lazy val` holding the compiled Hadoop `GlobPattern`. Keeping the compiled matcher on the options object means the glob is compiled once per executor JVM rather than once per archive; it is `transient` because `GlobPattern` is not serializable, so the string travels and executors recompile on first use. The glob string is validated on the driver, so an invalid glob raises a clear `IllegalArgumentException` naming the option. - `SupportsArchiveFormat.readArchiveEntries` and the random-access `readLocalizedEntries` / `localizeEntries` take an optional filter (default `None`, so existing callers are unaffected), applied in `shouldSkipEntry`. - The option is threaded from the read paths of CSV, JSON, text, XML, Avro, binaryFile, Parquet and ORC. The filter is applied **in addition to** `ignoredPathSegmentRegex` (both must pass), so hidden entries stay hidden even when they match the glob. ### Why are the changes needed? `pathGlobFilter` cannot express this. It is applied during file listing, so it must already match the archive file itself for that archive to be read at all; it cannot simultaneously select a subset of the entries *inside* the archive. Without `archivePathFilter` there is no way to read only part of an archive, so a user wanting a few entries has to read and discard the rest. ### Does this PR introduce _any_ user-facing change? Yes. A new read option `archivePathFilter` (a glob string, no default) is available to the file sources that support archive reads. When set, only archive entries whose full inner path matches the glob are ingested. Reads that do not set it are unaffected, and the whole archive-read feature remains gated by `spark.sql.files.archive.reader.enabled` (default `false`). ### How was this patch tested? New shared cases in `ArchiveReadSuiteBase`, which run for every (format, container) pair -- CSV, JSON, XML, text, Avro, Parquet, ORC and binaryFile across tar, zip and 7z: - selection by inner path (`sub/*`), - an extension glob matching across subdirectories, including an entry that does not match the glob, - a glob matching no entry yields zero rows, - composition with `ignoredPathSegmentRegex` (a hidden entry matching the glob is still skipped). New unit cases in `SupportsArchiveFormatSuite` cover the entry-skip logic directly (glob matching the full path, `*` crossing `/`, no-match, interaction with hidden-entry filtering) and invalid-glob rejection. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
