cloud-fan commented on code in PR #57956:
URL: https://github.com/apache/spark/pull/57956#discussion_r3796860083
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PullOutVariantExtractions.scala:
##########
@@ -177,11 +177,20 @@ object PullOutVariantExtractions extends
Rule[LogicalPlan] {
case _ => false
}
+ private def isJoinHoistable(e: Expression): Boolean = {
Review Comment:
Yes, `PUSH_VARIANT_INTO_SCAN_DEFER_CAST_ERROR` is intentional here. Unlike
`CombineFilters` and `PushPredicateThroughJoin`, scan pushdown has a per-row
cast-error companion column that preserves the original evaluation point. A
throwable extraction can therefore safely cross a join when deferral is
enabled; without deferral, the rule stops hoisting it across the join.
I documented this distinction beside `isJoinHoistable` and expanded the
config description in a12124e39a5. I also added the suggested test pinning
hoistable `VariantGet` and `Cast` as non-throwable. The focused test passed in
all four V1/V2 reader suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
