uros-b commented on code in PR #56864:
URL: https://github.com/apache/spark/pull/56864#discussion_r3735145895
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -1490,6 +1490,60 @@ object VariantArrayAppendExpressionBuilder extends
VariantArrayAppendExpressionB
// scalastyle:on line.size.limit
object TryVariantArrayAppendExpressionBuilder
extends VariantArrayAppendExpressionBuilderBase(false)
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(v[, includeArrays]) - Recursively removes null fields from
variant objects " +
+ "and null elements from arrays, unless `includeArrays` is false, in which
case null " +
+ "elements in arrays are kept. Returns NULL if any argument is NULL.",
+ arguments = """
+ Arguments:
+ * v - A variant value to mutate.
+ * includeArrays - An optional boolean (default true).
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(parse_json('{"a": 1, "b": null, "c": 3}'));
+ {"a":1,"c":3}
+ > SELECT _FUNC_(parse_json('[1, null, 3]'));
+ [1,3]
+ > SELECT _FUNC_(parse_json('{"a": {"b": null, "c": [1, null]}}'));
+ {"a":{"c":[1]}}
+ > SELECT _FUNC_(parse_json('{"a": [1, null], "b": null}'), false);
+ {"a":[1,null]}
+ > SELECT _FUNC_(NULL);
+ NULL
+ """,
Review Comment:
Two behaviors are specified in the VariantBuilder.stripNulls comment but
appear nowhere in the user-facing docs; not in examples, not in the Scala
scaladoc (functions.scala:14558-14566), not in the Python docstring
(builtin.py:23193-23225):
A container emptied by stripping stays {} / [] and the parent field is
retained, rather than collapsing to NULL. {"a": null} yields {}, not NULL.
A top-level variant null (parse_json('null')) is returned unchanged.
Both are covered by tests, so they're intentional, they just need to be
visible. An example like > SELECT _FUNC_(parse_json('{"a": null}')); → {} would
carry most of the weight.
Separately, in the usage string at lines 1495-1497, "removes null fields ...
Returns NULL if any argument is NULL" uses "null" in two different senses one
sentence apart. Saying variant null (the JSON null literal) in the first
sentence would disambiguate, a distinction that matters more in variant code
than anywhere else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
