LuciferYang opened a new pull request, #58340: URL: https://github.com/apache/spark/pull/58340
### What changes were proposed in this pull request? `FileTable` declares `TableCapability.SCAN_MERGING`, which lets the built-in file formats (Parquet, ORC, CSV, JSON, text, Avro) take part in the DSv2 scan merging added by #57360 (SPARK-40259). That PR put the whole mechanism on Spark's side: `PlanMerger` drives `V2ScanRelationPushDown.rebuildScan` to rebuild the merged scan, and a source supplies no merge logic of its own. It only declares that what its scans read is determined by the filters pushed and the columns pruned. The description of #57360 noted that no built-in source declared the capability yet; this is that follow-up. The production change is one line plus a comment recording why the contract holds for a file table: `fileIndex` is a table-level lazy val, so every scan built from the table lists the same files, and `newScanBuilder` returns a fresh builder over `mergedOptions(options)`. The rest is a new test suite and two documentation updates. ### Why are the changes needed? Two scans of the same file table that differ only in their projected columns cannot be reused today. A file source folds its data filters into the `FileScan` object, where they are used to list files and prune row groups, and `FileScan.equals` compares them. Two subquery scans over the same path are therefore not canonically identical, and `PlanMerger`'s identical-plan fast path does not fire. On the V1 path those filters sit in a `Filter` above an identical `LogicalRelation`, so the fast path does fire. The V2 file source path merges strictly less than V1 does, and declaring the capability closes most of that gap. ### Does this PR introduce _any_ user-facing change? Yes, on the V2 file source read path, which a format reaches only when it is removed from `spark.sql.sources.useV1SourceList`. Plan shape: subqueries over the same file table that differ only in their projected columns now collapse into a single scan reading the union of those columns. This needs no new configuration and is on by default. Results: for CSV and JSON the merged scan parses the union of both scans' columns, so with `mode` set to `DROPMALFORMED` a record malformed only in the columns the other scan reads is now dropped for both. V1 already read the union after merging, so this makes the two paths agree. The migration guide records it. `docs/sql-performance-tuning.md`: the `dsv2SymmetricFilterPropagation` entry ended with "no built-in source does", which this PR falsifies, so that sentence is rewritten. Three gaps are left in place: 1. Two scans with *different* partition filters do not merge, while V1 merges them. A partition filter is fully enforced by the V2 scan and reported as strict, so widening it to `OR` would leave the merged scan returning rows nothing above it filters out. V1 keeps the filter in a `Filter` node until physical planning, so symmetric propagation can widen it there. This is the differing-strict-filter follow-up #57360 lists. `FileSourceV2PlanMergingSuite` pins the current numbers so the gap stays visible. 2. Parquet and ORC nested columns do not merge while nested schema pruning is on. Each side narrows the struct to the field it reads, so the read column is no longer a same-type subset of the relation's column, and the field ordinals in the extractors above the scan would be resolved against the wrong type. #57360 lists widening to the union of nested fields as a follow-up. 3. Differing data filters need `spark.sql.optimizer.mergeSubplans.filterPropagation.dsv2SymmetricFilterPropagation.enabled`, which defaults to false. On the V2 path that config alone is enough, where V1 needs the broader `symmetricFilterPropagation`, because the strict filters are equal and widening the best-effort ones cannot change the row set the scan is required to return. I have not measured TPC-DS yet, so this PR makes no performance claim. ### How was this patch tested? New suite `FileSourceV2PlanMergingSuite`, 12 tests. Every built-in file table declares the capability; scans differing only in projected columns merge for parquet, orc, json and csv; scans over the same partition filter merge with that filter re-pushed strict onto the rebuilt scan; three scans merge into one; differing data filters merge only with the dsv2 config on. Declines are covered too: differing partition filters, nested-pruned columns, a pushed aggregate, and two different tables. Then V1/V2 parity on three shapes, the partition-filter gap pinned as the one shape where they disagree, and the CSV and JSON `DROPMALFORMED` behavior. Every test asserts the plan is on the V2 path before asserting anything about merging. SQL-on-file and catalog tables resolve to the V1 `FileFormat` regardless of `spark.sql.sources.useV1SourceList`, so a suite driven by that config alone would quietly run V1 on both sides. Removing `SCAN_MERGING` from `FileTable.CAPABILITIES` fails 7 of the 12 tests. Of the 5 that still pass, 4 assert a decline and one merges through the identical-plan fast path, which needs no capability. Regression: the `planmerging` suites, `ExplainSuite` and `ExplainSuiteAE`, `FileBasedDataSourceSuite`, `FileTableSuite`, `OrcV2SchemaPruningSuite`, `ParquetV2SchemaPruningSuite`, `ParquetV2FilterSuite`, `SubquerySuite`, `SameResultSuite`, `ParquetV2AggregatePushDownSuite`, `OrcV2AggregatePushDownSuite`, `DataSourceV2Suite` and `AvroV2Suite`. No golden file or `PlanStabilitySuite` plan needed regenerating. Those build their tables with `CREATE TABLE ... USING <format>`, which resolves to the V1 `FileFormat`, so none of them reaches a V2 file scan. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
