Tim Meehan created SPARK-61289:
----------------------------------
Summary: Raise NOT_EXPECTED_TYPE for an argument of a higher-order
function that is not a Column or a column name, in the Python client as PySpark
does
Key: SPARK-61289
URL: https://issues.apache.org/jira/browse/SPARK-61289
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
PySpark's higher-order functions take each column argument as a Column or a
column name, and raise NOT_EXPECTED_TYPE for anything else. In the Python
client of spark-connect-rust, they send a Python list or another value on as a
literal.
{code:python}
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.range(1).select(F.transform([1, 2, 3], lambda x: x *
10).alias("t")).collect()
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: PySparkTypeError: \[NOT_EXPECTED_TYPE\] Argument `col`
should be Column or str, got list.
* spark-connect-rust master 8768cc4: \[Row(t=\[10, 20, 30\])\]. F.exists(\[1,
2\], lambda x: x > 1) likewise returns \[Row(e=True)\].
So code written against the Python client can fail on PySpark. From source:
_invoke_higher_order_function (python/pyspark/sql/functions.py:551) converts
its columns with the _to_col at line 58, which returns anything but a str
unchanged; the _to_col of python/pyspark/sql/functions_generated.py (line 27)
raises, as PySpark's does. aggregate, array_sort, exists, filter, forall,
map_filter, map_zip_with, reduce, transform, transform_keys, transform_values
and zip_with all go through it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
