Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18906#discussion_r162857718
--- Diff: python/pyspark/sql/functions.py ---
@@ -2231,6 +2239,16 @@ def pandas_udf(f=None, returnType=None,
functionType=None):
... return pd.Series(np.random.randn(len(v))
>>> random = random.asNondeterministic() # doctest: +SKIP
+ .. note:: The user-defined functions are considered to be able to
return null values by default.
+ If your function is not deterministic, call `asNonNullable` on the
user defined function.
--- End diff --
ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
