HyukjinKwon commented on code in PR #99:
URL: https://github.com/apache/spark-connect-rust/pull/99#discussion_r3974693484
##########
python/pyspark/sql/functions.py:
##########
@@ -347,6 +348,28 @@ def sha2(col, numBits):
"""
return _pyfunc_sha2(_to_col(col), numBits)
+
+def variant_strip_nulls(v, include_arrays=True):
+ """Recursively removes variant-null object fields and array elements.
+
+ Args:
+ v: a variant column
+ include_arrays: also strip nulls inside arrays (default True)
+ """
+ return _pyfunc_variant_strip_nulls(_to_col(v), include_arrays)
+
+
+def variant_delete(v, *paths):
Review Comment:
`variant_delete` isn't registered in `_FN_COL_ARGS`, so `_dispatch` runs
every path arg through `_to_col` — a bare-string path like the `"$.b"` in the
new test becomes a column reference `col("$.b")` instead of a string literal.
`variant_get`/`try_variant_get` avoid this with `((0,), None)` so only the
value arg is a column and the path stays a literal (the reference client lits
string paths explicitly). Since variant paths are foldable string literals,
`variant_delete(v, "$.b")` would reach the server as an unresolved attribute
and fail analysis — and the offline test only asserts a `Column` is built, so
it won't catch it.
Recommend adding `'variant_delete': ((0,), None)` to `_FN_COL_ARGS` to match
`variant_get`. (The sibling variant-mutation wrappers
`variant_insert`/`variant_set`/`variant_array_append` and their `try_` forms
look to have the same gap, though they predate this PR.)
I verified the dispatch treatment and `variant_get`'s handling against the
tree but couldn't run it against a live 4.3.0 server, so flagging as a question
— the behavioral parity gate against the RC is the real confirmation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
