[
https://issues.apache.org/jira/browse/SPARK-61046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61046.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 245
[https://github.com/apache/spark-connect-rust/pull/245]
> Pivot only a groupBy, and raise the errors of the arguments of pivot, in the
> Python client as PySpark does
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61046
> URL: https://issues.apache.org/jira/browse/SPARK-61046
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{GroupedData.pivot}} pivots only a {{groupBy}}: the
> pivot of a {{rollup}}, a {{cube}} or {{groupingSets}} raises
> {{UNSUPPORTED_OPERATION}} of "PIVOT after ROLLUP", "PIVOT after CUBE" or
> "PIVOT after GROUPING_SETS", and a pivot of a pivot raises it of "Repeated
> PIVOT operation". It then raises {{NOT_EXPECTED_TYPE}} of {{pivot_col}} for a
> pivot column that is not a str, of {{values}} for values that are not a list,
> and of {{value}} for a value that is not a bool, a float, an int or a str.
> The Rust-backed Python client pivots any grouping as a {{groupBy}}:
> {{df.rollup("k").pivot("s").sum("v")}} returns the rows of
> {{df.groupBy("k").pivot("s").sum("v")}}, without the rows of the rollup, and
> a second pivot replaces the first. It raises {{TypeError}}s for the
> arguments, and takes a tuple of values. Raise these as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
