Muhammad Ali Gulzar created SPARK-60750:
-------------------------------------------
Summary: Dataset.cube() and rollup() no longer treat integer
literals as GROUP BY ordinals, changing results since 4.1.0
Key: SPARK-60750
URL: https://issues.apache.org/jira/browse/SPARK-60750
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.4, 4.2.1
Reporter: Muhammad Ali Gulzar
{code:python}
from pyspark.sql import functions as F
df = spark.range(5).toDF("id").withColumn("g", F.col("id") % 2)
df.cube(F.col("g"), F.lit(1)).count().collect()
# expected (4.0.5, and 4.2.1 through Spark Connect): 7 rows, lit(1) is ordinal
1, i.e. cube(g, g)
# [(0, 1, 3), (0, 1, 3), (0, 1, 3), (1, 1, 2), (1, 1, 2), (1, 1, 2), (None,
1, 5)]
# actual (4.2.1, Spark Classic): 6 rows, lit(1) is a constant
# [(0, 1, 3), (0, None, 3), (1, 1, 2), (1, None, 2), (None, 1, 5), (None,
None, 5)]{code}
The issue is in Spark Classic. Spark Connect returns the expected result. The
same with {{{}rollup{}}}, which returns 5 rows either way, but different rows.
With a literal that is not a valid position, 4.0.5 and Spark Connect fail with
GROUP_BY_POS_OUT_OF_RANGE or GROUP_BY_POS_AGGREGATE, while Spark Classic in
4.2.1 accepts it. For example, {{df.cube(F.lit(-28)).count()}} returns
{{{}[(-28, 5), (None, 5)]{}}}. {{groupBy}} is not affected, and all versions
agree when {{{}spark.sql.groupByOrdinal=false{}}}.
Cause: SPARK-51820 ([https://github.com/apache/spark/pull/50606]) moved the
conversion of integer literals to UnresolvedOrdinal out of the analyzer and
into the API layer. In Spark Classic, only {{Dataset.groupBy}} does the
conversion (Dataset.scala, {{{}groupingExpressionsWithOrdinals{}}}), while
{{{}rollup{}}}, {{cube,}} and {{groupingSets}} pass {{cols.map(_.expr)}}
unchanged. SparkConnectPlanner converts the grouping expressions for every
group type. Applying the same conversion in {{{}Dataset.rollup{}}}, {{cube}}
and {{groupingSets}} would fix it.
Reproduces on 4.1.4, 4.2.0, 4.2.1 and master (5.0.0-SNAPSHOT, 2026-10-29).
Spark 3.5.9 and 4.0.5 read the literal as an ordinal, as Spark Connect does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
