Takuya Ueshin created SPARK-37885:
-------------------------------------
Summary: Allow pandas_udf to take type annotations with future
annotations enabled
Key: SPARK-37885
URL: https://issues.apache.org/jira/browse/SPARK-37885
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.3.0
Reporter: Takuya Ueshin
When using {{{}from __future__ import annotations{}}}, the type hints will be
all strings, then pandas UDF type inference won't work as follows:
{code:python}
>>> from __future__ import annotations
>>> from typing import Union
>>> import pandas as pd
>>> from pyspark.sql.functions import pandas_udf
>>> @pandas_udf("long")
... def plus_one(v: Union[pd.Series, pd.DataFrame]) -> pd.Series:
... return v + 1
Traceback (most recent call last):
...
NotImplementedError: Unsupported signature: (v: 'Union[pd.Series,
pd.DataFrame]') -> 'pd.Series'.
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
