akshatshenoi-db opened a new pull request, #57805: URL: https://github.com/apache/spark/pull/57805
### What changes were proposed in this pull request? Adds tests asserting that reading an archive exposes the **parent archive file's** values in the `_metadata` column (`file_path`, `file_name`, `file_size`, `file_modification_time`, `file_block_start`, `file_block_length`). An archive is read as a single non-splittable `PartitionedFile`, so the generic `FileFormat` metadata extractors already produce the parent archive's values (`file_block_start` = 0, `file_block_length` = `file_size` = the archive's size on disk). This PR is therefore **test-only**: no production change was needed. The tests pin the behavior so it cannot silently regress. Inner-file `_metadata` is explicitly a **non-goal**: because the archive is a single split, the `_metadata` mechanism operates at `PartitionedFile` granularity and cannot attribute a row to its originating inner entry. This follows the archive-reader series: SPARK-57135 / SPARK-57321 (CSV), SPARK-57419 (JSON), SPARK-57478 (text), SPARK-57479 (XML), SPARK-57481 (Avro), SPARK-58382 (binaryFile), SPARK-57705 (zip), SPARK-58246 (7z). ### Why are the changes needed? The parent-archive `_metadata` contract was entirely untested. A future change to file splitting or to the metadata extractors could regress it unnoticed, and users reading archives rely on `_metadata` to identify the source archive file. ### Does this PR introduce any user-facing change? No. Test-only; archive reading remains gated by `spark.sql.files.archive.reader.enabled` (default false). ### How was this patch tested? A new shared test in `ArchiveReadSuiteBase`, which runs for every format x container suite (csv/json/xml/avro x tar/zip/7z), asserting every row of a multi-entry archive carries the same parent-archive values, with `file_block_start = 0` and `file_block_length = file_size = ` the archive's size. Plus the parallel test in `TextArchiveReadBase` and `BinaryFileArchiveReadBase`, which do not extend `ArchiveReadSuiteBase` (their row shapes differ). For binaryFile with `wholeFile=false` the test additionally pins that `_metadata` stays parent-only even though the `path`/`length` **data** columns are sourced per entry. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
