qlong commented on code in PR #57956:
URL: https://github.com/apache/spark/pull/57956#discussion_r3768326619
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PullOutVariantExtractions.scala:
##########
@@ -177,11 +177,20 @@ object PullOutVariantExtractions extends
Rule[LogicalPlan] {
case _ => false
}
+ private def isJoinHoistable(e: Expression): Boolean = {
Review Comment:
Is there a reason to add this new gate? isJoinHoistable gate will not hoist
a strict variant_get across a Join unless
PUSH_VARIANT_INTO_SCAN_DEFER_CAST_ERROR is set. The ungated behavior is
documented between line 108-122, and is more consistent with existing
bahaviors. It seems this gate singles out variant for cross a join while
allowing the other two cases to raise:
1. variant_get below a Filter → raises
2. cast(v as int) across a Join → raises (test @844, pre-existing base-PR
behavior)
3. variant_get across a Join → suppressed by the gate (test @881, new here)
I think it good that l three have consistent behavior. We also have existing
switches:
- pushVariantIntoScan.pullOutExtractions = false to turn off the pull-out
rule completely
- pushVariantIntoScan.deferCastError = true to defer the error past
eliminated rows uniformly for every strict extraction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
