[
https://issues.apache.org/jira/browse/SPARK-61289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Meehan updated SPARK-61289:
-------------------------------
Fix Version/s: connect-rust-4.3.0
> Raise NOT_EXPECTED_TYPE for an argument of a higher-order function that is
> not a Column or a column name, in the Python client as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61289
> URL: https://issues.apache.org/jira/browse/SPARK-61289
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark's higher-order functions take each column argument as a Column or a
> column name, and raise NOT_EXPECTED_TYPE for anything else. In the Python
> client of spark-connect-rust, they send a Python list or another value on as
> a literal.
> {code:python}
> from pyspark.sql import SparkSession, functions as F
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> spark.range(1).select(F.transform([1, 2, 3], lambda x: x *
> 10).alias("t")).collect()
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: PySparkTypeError: \[NOT_EXPECTED_TYPE\] Argument
> `col` should be Column or str, got list.
> * spark-connect-rust master 8768cc4: \[Row(t=\[10, 20, 30\])\]. F.exists(\[1,
> 2\], lambda x: x > 1) likewise returns \[Row(e=True)\].
> So code written against the Python client can fail on PySpark. From source:
> _invoke_higher_order_function (python/pyspark/sql/functions.py:551) converts
> its columns with the _to_col at line 58, which returns anything but a str
> unchanged; the _to_col of python/pyspark/sql/functions_generated.py (line 27)
> raises, as PySpark's does. aggregate, array_sort, exists, filter, forall,
> map_filter, map_zip_with, reduce, transform, transform_keys, transform_values
> and zip_with all go through it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
