Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19787#discussion_r151928352
--- Diff: python/pyspark/sql/functions.py ---
@@ -2205,6 +2205,10 @@ def udf(f=None, returnType=StringType()):
rows that do not satisfy the conditions, the suggested workaround
is to incorporate the
condition logic into the functions.
+ .. note:: Users can't rely on short-curcuit evaluation of boolean
expressions to execute
+ conditionally user-defined functions too. For example, the two
functions in an expression
+ like udf1(x) && udf2(y) will be both executed on all rows.
--- End diff --
I think pandas_udf doesn't use in boolean expressions as it returns
pandas.Series.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
