Hyukjin Kwon created SPARK-60995:
------------------------------------
Summary: Check the arguments of when, bucket, pandas_udf,
arrow_udf and arrow_udtf of the Python client as PySpark does
Key: SPARK-60995
URL: https://issues.apache.org/jira/browse/SPARK-60995
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, these functions check their arguments, before any request,
and the Rust-backed Python client does not:
* {{when(condition, value)}} and {{Column.when}} raise {{PySparkTypeError}}
{{NOT_EXPECTED_TYPE}} for a {{condition}} that is not a {{Column}}. The client
takes a string as a column name, so {{F.when("a", 1)}} is {{CASE WHEN a THEN 1
END}}.
* {{bucket(numBuckets, col)}} and {{partitioning.bucket}} raise
{{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is neither an int nor a
{{Column}}. The client takes a string as a column name. {{bucket}} also warns
that it is deprecated in favor of {{partitioning.bucket}}, as {{years}},
{{months}}, {{days}} and {{hours}} do.
* {{pandas_udf}} and {{arrow_udf}} raise {{CANNOT_BE_NONE}} without a return
type, and {{INVALID_PANDAS_UDF_TYPE}} for an invalid function type. The client
returns a decorator, as {{F.pandas_udf()}} does.
* {{arrow_udtf}} raises {{INVALID_UDTF_HANDLER_TYPE}} for a handler that is not
a class, as {{arrow_udtf(3)}}. The client returns a
{{UserDefinedTableFunction}}.
Check the arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
