Hyukjin Kwon created SPARK-61046:
------------------------------------
Summary: Pivot only a groupBy, and raise the errors of the
arguments of pivot, in the Python client as PySpark does
Key: SPARK-61046
URL: https://issues.apache.org/jira/browse/SPARK-61046
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{GroupedData.pivot}} pivots only a {{groupBy}}: the pivot
of a {{rollup}}, a {{cube}} or {{groupingSets}} raises
{{UNSUPPORTED_OPERATION}} of "PIVOT after ROLLUP", "PIVOT after CUBE" or "PIVOT
after GROUPING_SETS", and a pivot of a pivot raises it of "Repeated PIVOT
operation". It then raises {{NOT_EXPECTED_TYPE}} of {{pivot_col}} for a pivot
column that is not a str, of {{values}} for values that are not a list, and of
{{value}} for a value that is not a bool, a float, an int or a str.
The Rust-backed Python client pivots any grouping as a {{groupBy}}:
{{df.rollup("k").pivot("s").sum("v")}} returns the rows of
{{df.groupBy("k").pivot("s").sum("v")}}, without the rows of the rollup, and a
second pivot replaces the first. It raises {{TypeError}}s for the arguments,
and takes a tuple of values. Raise these as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
