Hyukjin Kwon created SPARK-60953:
------------------------------------
Summary: Render regexp_replace and instr columns in Spark Connect
Rust as PySpark does
Key: SPARK-60953
URL: https://issues.apache.org/jira/browse/SPARK-60953
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{repr(F.regexp_replace("s", "a", "b"))}} is
{{Column<'regexp_replace(s, a, b)'>}}, and {{repr(F.instr("s", "b",
occurrence=2))}} is {{Column<'instr(s, b, 1, 2)'>}}. The Rust-backed Python
client renders these columns as the Rust debug form of the expression, such as
{{Column<'CallFunction(CallFunctionWrapper { function_name: "regexp_replace",
arguments: [ColumnReference(...), ...] })'>}}, which also appears in error
messages and plans. Render them as other functions are.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
