Ted Chester Jenks created SPARK-47287:
-----------------------------------------
Summary: Aggregate in not causes
Key: SPARK-47287
URL: https://issues.apache.org/jira/browse/SPARK-47287
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.1
Reporter: Ted Chester Jenks
The below snippet is confirmed working with Spark 3.2.1 and broken Spark 3.4.1.
i believe this is a bug.
{code:java}
Dataset<Row> ds = dummyDataset
.withColumn("flag",
functions.not(functions.coalesce(functions.col("bool1"),
functions.lit(false)).equalTo(true)))
.groupBy("code")
.agg(functions.max(functions.col("flag")).alias("flag"));
ds.show(); {code}
It fails with:
{code:java}
Caused by: java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:208)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpression$7(V2ExpressionBuilder.scala:185)
at scala.Option.map(Option.scala:230)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:184)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33)
at
org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateAggregateFunc(V2ExpressionBuilder.scala:293)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33)
at
org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803)
at
org.apache.spark.sql.execution.datasources.DataSourceStrategy$.translate$1(DataSourceStrategy.scala:700)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
