Hyukjin Kwon created SPARK-60948:
------------------------------------
Summary: Render Column reprs in Spark Connect Rust as PySpark
Connect does
Key: SPARK-60948
URL: https://issues.apache.org/jira/browse/SPARK-60948
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect renders a Column as {{Column<'...'>}} of its expression, with
an operator in the form of a function. The Rust-backed Python client renders
operators infix, so these PySpark 4.3.0 reprs differ:
* {{F.col("id") + 1}} is {{Column<'+(id, 1)'>}}; the client gives {{Column<'(id
+ 1)'>}}. The same holds for the other arithmetic and comparison operators, for
{{&}} and {{|}}, which are {{and(...)}} and {{or(...)}}, and for {{between}},
which is {{and(>=(a, 1), <=(a, 2))}}, also within {{CASE WHEN}}.
* {{-F.col("a")}} is {{Column<'negative(a)'>}}, and {{~F.col("a")}} is
{{Column<'not(a)'>}}; the client gives {{(- a)}} and {{(NOT a)}}.
* {{F.col("a").cast("int")}} is {{CAST(a AS INT)}}; the client gives {{CAST(a
AS int)}}.
* {{F.col("*")}} is {{unresolvedstar()}}; the client gives {{*}}.
* {{F.col("a")[0]}} is {{a['0']}}; the client gives {{a[0]}}.
The repr also appears in error messages, such as the one that
{{SparkSession.sql}} raises for a keyword argument that is not a plain column
reference. Render Column reprs as PySpark Connect does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
