Hyukjin Kwon created SPARK-61045:
------------------------------------
Summary: Take the arguments of colRegex, metadataColumn, where and
transpose in the Python client as PySpark does
Key: SPARK-61045
URL: https://issues.apache.org/jira/browse/SPARK-61045
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{df.colRegex(colName)}} and {{df.metadataColumn(colName)}}
raise {{NOT_EXPECTED_TYPE}} of {{colName}} for one that is not a str,
{{df.where(condition)}} raises {{NOT_EXPECTED_TYPE}} of {{condition}} for one
that is neither a Column nor a str, and {{df.transpose(indexColumn)}} takes a
Column or a column name, else {{NOT_EXPECTED_TYPE}} of {{col}}.
The Rust-backed Python client raises a {{TypeError}} for these, and for a
column name of {{transpose}}, as {{df.transpose("id")}}. Take them as PySpark
does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
