qlong opened a new pull request, #56556: URL: https://github.com/apache/spark/pull/56556
### What changes were proposed in this pull request? Two fixes in `pushVariantExtractions` (called by`V2ScaqqnRelationPushDown.pushDownVariants`): 1. **Guard against double-visit**: Add a `pushedVariants.isEmpty` sentinel check so the inner `ScanBuilderHolder` leaf visit (caused by `transformDown` recursing into the child after returning the plan unchanged) returns immediately. This ensures `builder.pushVariantExtractions` is called exactly once per holder. 2. **Eager column pruning**: While `projectList` and `filters` are in scope, call `builder.pruneColumns(requiredSchema)` for builders implementing `SupportsPushDownRequiredColumns` and trim `sHolder.output` to the required columns. By the time `buildScanWithPushedVariants` calls `build()`, the builder already has the correct pruned schema. This is similiar to how buildScanWithPushedAggregate works. Jira: https://issues.apache.org/jira/browse/SPARK-57499 ### Why are the changes needed? Two bugs on the accepted variant pushdown path: **Issue 1 — column pruning is skipped.** `buildScanWithPushedVariants` calls `builder.build()` and replaces the `ScanBuilderHolder` with a `DataSourceV2ScanRelation`. The subsequent `pruneColumns` rule matches only `ScanBuilderHolder` nodes, so it is a no-op and `builder.pruneColumns()` is never called. The scan reads the full table schema including unreferenced columns. For unreferenced `VARIANT` columns this is especially costly — each is fully reconstructed from its shredded Parquet tree on every row. **Issue 2 — invalid plan on native Parquet V2.** `pushDownVariants` uses `transformDown`, which recurses into the child `ScanBuilderHolder` after returning the plan unchanged. The bare `ScanBuilderHolder` matches `PhysicalOperation` a second time, collecting unreferenced sibling `VARIANT` columns as full-variant requests and pushing them to the builder again. `ParquetScanBuilder` overwrites its state on every call, so the second push clobbers the correct extraction from the first, producing a dangling `ExprId` in the projection: ``` !Project [variant_get(v1#57, $.x) ...] -- stale ExprId, marked invalid +- BatchScan parquet [a#66, v1#67, v2#68] PushedVariantExtractions: [v2:"$"] -- sibling variant pushed, not v1:$.x ``` This causes a runtime failure: ``` [INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find v1#57 in [a#72,v1#73,v2#74] ``` Both issues affect DSv2 sources implementing `SupportsPushDownVariantExtractions` and are gated on accepted variant pushdown. When pushdown is declined or disabled the `ScanBuilderHolder` survives and `pruneColumns` runs normally. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Added new unit tests - manual testing with spark-sql ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude code (Sonnet 4.6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
