peterxcli opened a new issue, #5428: URL: https://github.com/apache/datafusion-comet/issues/5428
### What is the problem the feature request solves? Spark 4.x provides `parse_json` and `try_parse_json` to construct `VariantType` values from strings, but Comet has no serializer or native evaluator for either function. A query such as the following cannot remain native: ```sql SELECT parse_json(json_text) FROM t; SELECT try_parse_json(json_text) FROM t; ``` Spark implements both forms with one [`ParseJson` expression](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L48-L78): `parse_json` throws for malformed input while `try_parse_json` returns SQL NULL. The runtime also distinguishes malformed input and the Variant size limit and honors `spark.sql.variant.allowDuplicateKeys` in [`VariantExpressionEvalUtils.parseJson`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L34-L55). ### Describe the potential solution Add Spark-4-specific serialization and a native parser for the two expressions: - parse each non-null input string into canonical Variant `value` and `metadata` bytes; - emit the marked Variant output Field and Spark-compatible `[value, metadata]` children defined by #5425; - preserve JSON null as Variant null and SQL NULL as a null parent row; - honor Spark's duplicate-key setting, Unicode/object-key behavior, numeric representation, and Variant size limit; and - match strict/try malformed-input behavior and Spark-compatible errors. Add focused Spark SQL parity and plan tests for objects, arrays, scalars, JSON null, SQL NULL, Unicode keys, duplicate keys with both configuration modes, malformed JSON, oversize input, and columns before/after the result. ### Additional context Spark registers the exact SQL names `parse_json` and `try_parse_json` in its [4.1.3 function registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L888-L896). Related work: #4295, #5407, and #5425. Non-goals: `to_variant_object`, `CAST(... AS VARIANT)`, Parquet writing, nested Variant columns, shuffle/spill, C2R, Python transport, and Iceberg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
