rdtr opened a new pull request, #13588:
URL: https://github.com/apache/gluten/pull/13588

   Depends on #13520. Only the last commit belongs to this PR: 
https://github.com/rdtr/incubator-gluten/commit/d7ea1639ff49631de4a43747c3c783bcc5c0db95
   
   ## What changes were proposed in this pull request?
   
   Spark 4.3 (SPARK-58265) can derive DPP partition values from the full rows 
of an existing broadcast, through `ProjectedBroadcastValueSubqueryExec`. It is 
off by default, behind the internal conf 
`spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.enabled`. 
#13520 excludes its 24 tests (4 test names over the V1 and V2 suites) with a 
generic comment.
   
   Projection pruning isn't supported with columnar broadcasts, and this PR 
doesn't add it. With the conf on and AQE on, the broadcast is a columnar 
`BuildSideRelation`. Spark's projection reads a `HashedRelation`, hits an error 
and marks the domain unavailable, so every partition is scanned and answers 
stay correct. With AQE off, `RewriteSubqueryBroadcast` doesn't touch the new 
node. Its exchange stays a row `BroadcastExchangeExec`, so pruning works but 
the build side is broadcast a second time instead of reusing the join's 
columnar broadcast.
   
   The 24 tests stay excluded, now with the actual reason. 18 are on AQE-on 
suites and depend on pruning values that Gluten doesn't produce. 6 are the 
composite-key test on AQE-off suites, which stops at the 
`lastAttemptValueForDataset` assert. I didn't trace which Gluten node makes 
that return None, and the asserts after it didn't run.
   
   This adds two Gluten tests to the V1 suites, one per behavior, with the conf 
on and ANSI off so the plan stays native:
   
   - `GlutenDynamicPartitionPruningV1SuiteAEOn`: "SPARK-58265: projected 
broadcast values fail open on a columnar broadcast" checks the answer, a native 
scan, an unavailable domain, `projectionDisabled` = 1 and `projectionErrors` = 
1.
   - `GlutenDynamicPartitionPruningV1SuiteAEOff`: "SPARK-58265: projected 
broadcast values come from a separate row broadcast" checks the answer, the 
projected values, a row exchange under the subquery and no `ReusedExchangeExec`.
   
   The first test is excluded on `AEOnDisableScan`, since it asserts the native 
scan.
   
   Supporting this would take a Spark 4.3 shim, because the node only exists in 
4.3, and it has to keep Spark's node, because `InSubqueryExec` only recognizes 
the unavailable result on that type. The child would expose the columnar 
broadcast as a lazily built `HashedRelation` and implement 
`BroadcastExchangeLike` for Spark's size check, and the exchange would need 
rewriting to columnar for reuse without AQE. That's roughly 100+ lines for an 
off-by-default optimization, so I left it out. Let me know if you'd want it.
   
   ## How was this patch tested?
   
   Spark 4.3.0, all 12 `GlutenDynamicPartitionPruning*` suites with 
`SPARK_ANSI_SQL_MODE=false`: 462 passed, 0 failed, 67 ignored. Before this 
change, with the 24 excludes removed, the same suites had 24 failures. Only 
`gluten-ut/spark43` test files change.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to