Yaoxuan Wu created SPARK-60072:
----------------------------------
Summary: DataFrame pivot rejects count('*') with
INVALID_USAGE_OF_STAR_OR_REGEX
Key: SPARK-60072
URL: https://issues.apache.org/jira/browse/SPARK-60072
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 4.0.0
Reporter: Yaoxuan Wu
Since Spark 4.0, a DataFrame pivot whose aggregate is {{count("*")}} fails
analysis. The same query works in 3.5.
{code:java}
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
from pyspark.sql import functions as F
df = spark.createDataFrame([(1, 'a'), (2, None)], "x int, s string")
df.groupBy('s').pivot('x', [1, 2]).agg(F.count('*')).show() {code}
Expected (3.5.8): {{[Row(s=None, 1=None, 2=1), Row(s='a', 1=1, 2=None)]}}
Actual (4.0.0, 4.1.1, 4.2.0):
{code:java}
AnalysisException: [INVALID_USAGE_OF_STAR_OR_REGEX] Invalid usage of '*' in
Pivot. SQLSTATE: 42000;
'Pivot List(s#1), x#0: int, [1, 2], ['count(*)]
+- LogicalRDD [x#0, s#1], false {code}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
