holdenk commented on code in PR #59081:
URL: https://github.com/apache/spark/pull/59081#discussion_r4118417509
##########
python/pyspark/sql/transpile.py:
##########
@@ -802,6 +814,17 @@ def _convert_chunk(self, params: List[str], body: ast.AST
| None) -> Column:
f"name {name!r} is not in the UDF's parameter list "
"and free variables / closures are not supported"
)
+ case ast.Call(func=ast.Name(id="len"), args=[arg], keywords=[]):
+ # SPARK-55214: Python ``len`` on a str is the number of
Unicode
+ # points; Spark ``length`` on a string column is character
length
+ # they match for well-formed UTF-8. NULL stays NULL, while
+ # Python's ``len(None)`` raises.
Review Comment:
We probably should address this, we can do this with a conditional similar
to how we handle NaNs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
