Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20237#discussion_r161373555
--- Diff: python/pyspark/sql/functions.py ---
@@ -2184,6 +2184,11 @@ def pandas_udf(f=None, returnType=None,
functionType=None):
| 8| JOHN DOE| 22|
+----------+--------------+------------+
+ .. note:: The length of `pandas.Series` within a scalar UDF is not
that of the whole input
+ column, but is the length of an internal batch used for each
call to the function.
+ Therefore, this can be used, for example, to ensure the length
of each returned
--- End diff --
I meant to ensure the length of the batch because we declare "The length of
the returned `pandas.Series` must be of the same as the input `pandas.Series`."
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
