[
https://issues.apache.org/jira/browse/SPARK-60953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60953:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Render regexp_replace and instr columns in Spark Connect Rust as PySpark does
> -----------------------------------------------------------------------------
>
> Key: SPARK-60953
> URL: https://issues.apache.org/jira/browse/SPARK-60953
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{repr(F.regexp_replace("s", "a", "b"))}} is
> {{Column<'regexp_replace(s, a, b)'>}}, and {{repr(F.instr("s", "b",
> occurrence=2))}} is {{Column<'instr(s, b, 1, 2)'>}}. The Rust-backed Python
> client renders these columns as the Rust debug form of the expression, such
> as {{Column<'CallFunction(CallFunctionWrapper { function_name:
> "regexp_replace", arguments: [ColumnReference(...), ...] })'>}}, which also
> appears in error messages and plans. Render them as other functions are.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
