[
https://issues.apache.org/jira/browse/SPARK-43189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127959#comment-18127959
]
Shuheng Ma commented on SPARK-43189:
------------------------------------
The PR for SPARK-39142 also fixes this:
https://github.com/apache/spark/pull/59558. The docs example now type-checks,
and the {{type: ignore}} comments are removed from the docs examples. Thanks
for the report!
> No overload variant of "pandas_udf" matches argument type "str"
> ---------------------------------------------------------------
>
> Key: SPARK-43189
> URL: https://issues.apache.org/jira/browse/SPARK-43189
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.2.4, 3.3.2, 3.4.0
> Reporter: Andrew Grigorev
> Priority: Major
>
> h2. Issue
> Users who have mypy enabled in their IDE or CI environment face very verbose
> error messages when using the {{pandas_udf}} function in PySpark. The current
> typing of the {{pandas_udf}} function seems to be causing these issues. As a
> workaround, the official documentation provides examples that use {{{}# type:
> ignore[call-overload]{}}}, but this is not an ideal solution.
> h2. Example
> Here's a code snippet taken from
> [docs|https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#pandas-udfs-a-k-a-vectorized-udfs]
> that triggers the error when mypy is enabled:
> {code:python}
> from pyspark.sql.functions import pandas_udf
> import pandas as pd
> @pandas_udf("col1 string, col2 long")
> def func(s1: pd.Series, s2: pd.Series, s3: pd.DataFrame) -> pd.DataFrame:
> s3['col2'] = s1 + s2.str.len()
> return s3 {code}
> Running mypy on this code results in a long and verbose error message, which
> makes it difficult for users to understand the actual issue and how to
> resolve it.
> h2. Proposed Solution
> We kindly request the PySpark development team to review and improve the
> typing for the {{pandas_udf}} function to prevent these verbose error
> messages from appearing. This improvement will help users who have mypy
> enabled in their development environments to have a better experience when
> using PySpark.
> Furthermore, we suggest updating the official documentation to provide better
> examples that do not rely on {{# type: ignore[call-overload]}} to suppress
> these errors.
> h2. Impact
> By addressing this issue, users of PySpark with mypy enabled in their
> development environment will be able to write and verify their code more
> efficiently, without being overwhelmed by verbose error messages. This will
> lead to a more enjoyable and productive experience when working with PySpark
> and pandas UDFs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
