[
https://issues.apache.org/jira/browse/SPARK-61011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61011.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 216
[https://github.com/apache/spark-connect-rust/pull/216]
> Refer to the columns of a DataFrame by its plan ID in the Python client as
> PySpark does
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-61011
> URL: https://issues.apache.org/jira/browse/SPARK-61011
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, the columns of a DataFrame refer to it by the plan ID of
> its plan: {{df["name"]}}, {{df.name}} and {{df._col(name)}} are the
> {{ColumnReference}} of the name and the plan ID, {{df["*"]}} the
> {{UnresolvedStar}} of the plan ID, {{df.metadataColumn(name)}} the metadata
> {{ColumnReference}} of it, and {{df.colRegex(regex)}} the Column of the
> {{UnresolvedRegex}} of it. The server resolves them against that plan, so
> that {{df1.join(df2, df1["id"] == df2["id"]).select(df1["id"], df2["w"])}}
> selects the {{id}} of {{df1}}, and a self-join of a DataFrame and one derived
> from it raises {{AMBIGUOUS_COLUMN_REFERENCE}}.
> The Rust-backed Python client refers to them by name only, so the join above
> raises {{AMBIGUOUS_REFERENCE}}, as does {{df1.join(df2, df1.id == df2.id)}};
> and {{colRegex}} returns a DataFrame rather than a Column. Refer to the
> columns of a DataFrame as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
