Donggu Kang created SPARK-42235:
-----------------------------------
Summary: Missing typing for pandas_udf
Key: SPARK-42235
URL: https://issues.apache.org/jira/browse/SPARK-42235
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.3.0
Reporter: Donggu Kang
The typing stub `site-packages/pyspark/sql/pandas/functions.pyi` has a list of
possible signatures of `pandas_udf`. It is missing a case
```
import pyspark.sql.functions as F
# PySpark3's typing stub error
@F.pandas_udf(
returnType=LongType()
)
def my_udf(dummy_col: pd.Series) -> pd.Series:
...
```
The stub defined `pandas_udf(f, returnType, functionType)` but not
`pandas_udf(f, returnType)`. The official documentation recommends using a
return type hint instead of `returnType`.
```py
# defined
@overload def pandas_udf( f: PandasScalarToScalarFunction, returnType:
Union[AtomicDataTypeOrString, ArrayType], functionType: PandasScalarUDFType, )
-> UserDefinedFunctionLike: ...
# not defined
@overload def pandas_udf( f: PandasScalarToScalarFunction, returnType:
Union[AtomicDataTypeOrString, ArrayType]) -> UserDefinedFunctionLike: ...
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
