hzmsh commented on PR #59558: URL: https://github.com/apache/spark/pull/59558#issuecomment-6510212670
Thanks @gaogaotiantian! Both come from the legacy `PandasUDFType` overloads (from the original pyspark-stubs port, SPARK-32714): - **Why so many overloads:** the legacy ones check the function and return type separately for each `functionType`: SCALAR returning a Series, SCALAR returning a DataFrame, SCALAR_ITER, GROUPED_MAP and GROUPED_AGG. Each of those 5 needs 4 overloads for the call forms `pandas_udf(f, rt, ft)`, `@pandas_udf(rt, ft)`, `@pandas_udf(rt, functionType=ft)` and `@pandas_udf(returnType=rt, functionType=ft)`, so 20 in total, about 90 lines, all already on master. This PR adds 8 lines: 3 overloads for the type-hint style without `functionType`, like `udf()`. It also adds `MapType` to some of the existing ones. - **The `overload-overlap` ignores** (added in SPARK-47055): both SCALAR sets accept `str` for `returnType`, since a DDL string can be `"double"` or `"a int, b string"`. So mypy sees their decorator forms as overlapping with different `Callable` return types. One way to remove the ignores is to merge the two SCALAR sets into one that accepts either function shape and either return type. I tried it locally: it drops 4 overloads and all 3 ignores, and mypy and `test_udf.yml` still pass. The cost is that mypy no longer catches a SCALAR UDF with a `StructType` return type and a Series-returning function. Would you like that as a follow-up JIRA, or should I include it in this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
