rich7420 commented on code in PR #5972:
URL: https://github.com/apache/datafusion-comet/pull/5972#discussion_r4053570308
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -814,6 +825,249 @@ case class CometExecRule(session: SparkSession)
}
}
+ /** Keep ANSI next_day in Spark's row pipeline where consumers can skip or
catch its errors. */
+ private def preserveNextDayEvaluationMasks(plan: SparkPlan): SparkPlan = {
+ def nextDayName(expr: Expression): Option[String] = expr.collectFirst {
+ // Both the native kernel and dispatcher can throw on rows that Spark
skips.
+ case nextDay: NextDay if nextDay.failOnError => "next_day"
Review Comment:
Please exclude provably safe instances from this guard. On Spark 4.1 with
ANSI enabled, `SELECT next_day(d, 'Monday') FROM t LIMIT 1` changes from
`CometProject` and `CometCollectLimit` with the base rule to Spark `Project`
and `CollectLimit` on this head. Here `d` is a date column and `Monday` is a
valid literal, so the invalid-weekday error cannot occur. Please keep this case
native and add an admission control alongside the skipped-invalid-weekday
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
