andygrove commented on code in PR #4365:
URL: https://github.com/apache/datafusion-comet/pull/4365#discussion_r3274765279
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -249,6 +249,9 @@ object QueryPlanSerde extends Logging with CometExprShim
with CometTypeShim {
private val conversionExpressions: Map[Class[_ <: Expression],
CometExpressionSerde[_]] = Map(
classOf[Cast] -> CometCast)
+ private val jsonExpressions: Map[Class[_ <: Expression],
CometExpressionSerde[_]] = Map(
+ classOf[LengthOfJsonArray] -> CometScalarFunction("json_array_length"))
Review Comment:
Spark's `JsonExpressionUtils.lengthOfJsonArray` uses a Jackson parser
configured with
`ALLOW_SINGLE_QUOTES` and `ALLOW_UNESCAPED_CONTROL_CHARS`, and uses
streaming parsing that ignores trailing content after the array's closing
bracket. `serde_json::from_str::<Value>` is much stricter. Could you mark
`getSupportLevel` as `Incompatible(Some("..."))` and add
`getIncompatibleReasons()` so that the auto-generated compat doc surfaces the
limitation to users, and the function is gated behind
`spark.comet.expression.LengthOfJsonArray.allowIncompatible=true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
