[
https://issues.apache.org/jira/browse/SPARK-60995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60995:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Check the arguments of when, bucket, pandas_udf, arrow_udf and arrow_udtf of
> the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60995
> URL: https://issues.apache.org/jira/browse/SPARK-60995
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, these functions check their arguments, before any
> request, and the Rust-backed Python client does not:
> * {{when(condition, value)}} and {{Column.when}} raise {{PySparkTypeError}}
> {{NOT_EXPECTED_TYPE}} for a {{condition}} that is not a {{Column}}. The
> client takes a string as a column name, so {{F.when("a", 1)}} is {{CASE WHEN
> a THEN 1 END}}.
> * {{bucket(numBuckets, col)}} and {{partitioning.bucket}} raise
> {{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is neither an int nor a
> {{Column}}. The client takes a string as a column name. {{bucket}} also warns
> that it is deprecated in favor of {{partitioning.bucket}}, as {{years}},
> {{months}}, {{days}} and {{hours}} do.
> * {{pandas_udf}} and {{arrow_udf}} raise {{CANNOT_BE_NONE}} without a return
> type, and {{INVALID_PANDAS_UDF_TYPE}} for an invalid function type. The
> client returns a decorator, as {{F.pandas_udf()}} does.
> * {{arrow_udtf}} raises {{INVALID_UDTF_HANDLER_TYPE}} for a handler that is
> not a class, as {{arrow_udtf(3)}}. The client returns a
> {{UserDefinedTableFunction}}.
> Check the arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
