viirya commented on a change in pull request #27165: [SPARK-28264][PYTHON][SQL]
Support type hints in pandas UDF and rename/move inconsistent pandas UDF types
URL: https://github.com/apache/spark/pull/27165#discussion_r365478625
##########
File path: python/pyspark/sql/pandas/functions.py
##########
@@ -490,31 +405,78 @@ def pandas_udf(f=None, returnType=None,
functionType=None):
eval_type = returnType
else:
# @pandas_udf(dataType) or @pandas_udf(returnType=dataType)
- eval_type = PythonEvalType.SQL_SCALAR_PANDAS_UDF
+ eval_type = None
else:
return_type = returnType
if functionType is not None:
eval_type = functionType
else:
- eval_type = PythonEvalType.SQL_SCALAR_PANDAS_UDF
+ eval_type = None
if return_type is None:
- raise ValueError("Invalid returnType: returnType can not be None")
+ raise ValueError("Invalid return type: returnType can not be None")
if eval_type not in [PythonEvalType.SQL_SCALAR_PANDAS_UDF,
PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF,
PythonEvalType.SQL_GROUPED_AGG_PANDAS_UDF,
PythonEvalType.SQL_MAP_PANDAS_ITER_UDF,
- PythonEvalType.SQL_COGROUPED_MAP_PANDAS_UDF]:
+ PythonEvalType.SQL_COGROUPED_MAP_PANDAS_UDF,
+ None]: # None means it should infer the type from
type hints.
+
raise ValueError("Invalid functionType: "
Review comment:
nit: Invalid function type? As like above `Invalid return type`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
