This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 884e0a675a chore: bump spark-4.2 profile to the released 4.2.0 (#4960)
884e0a675a is described below
commit 884e0a675ac8f3bed38f140485b686b1f5f18c71
Author: Andy Grove <[email protected]>
AuthorDate: Fri Jul 17 13:24:32 2026 -0600
chore: bump spark-4.2 profile to the released 4.2.0 (#4960)
* chore: bump spark-4.2 profile to the released 4.2.0
Spark 4.2.0 is now in Maven Central, so the spark-4.2 profile no longer
needs to target a preview. Point spark.version at 4.2.0 and make the
main sources compile against it.
- Drop the WIP compile-only comment on the spark-4.2 profile.
- Spark 4.2.0 bundles its own copy of
org.apache.datasketches.memory.internal.ResourceImpl inside
spark-catalyst, colliding with the transitive datasketches-memory jar
and failing the BanDuplicateClasses enforcer rule; ignore that class.
- Spark 4.2 removes GeographyVal / GeometryVal and replaces
getGeography / getGeometry on SpecializedGetters with a single
getBinaryView returning BinaryView. CometInternalRowShim can no longer
be shared from the spark-4.1+ source root, so split it into per-version
spark-4.1 and spark-4.2 copies.
- Correct the stale Jetty comment that referenced preview4.
This does not wire up the Spark SQL tests for 4.2; it only moves the
profile onto the released version and keeps it compiling.
* chore: regenerate Spark 4.2 plan-stability golden files
Regenerated with dev/regenerate-golden-files.sh --spark-version 4.2
against the released 4.2.0. The q2, q5, q54 (v1_4) and q5a (v2_7)
goldens are pruned because their plans now match the shared fallback on
4.2.0, and q77a (v2_7) is added to record a degraded plan.
* fix: resolve Spark 4.2 test failures from the version bump
Two Spark 4.2.0 changes surfaced by moving the profile off preview4:
Iceberg scans job: Spark 4.2.0 turned `connector.catalog.View` from an
interface into a class. No Iceberg spark-runtime is published for 4.2, so
the build reuses the 4.0 runtime, whose `SparkView implements View` now
throws IncompatibleClassChangeError at class-load and aborts the Iceberg
suites. Report Iceberg as unavailable on Spark 4.2 in the shared probes so
the suites skip. Guard the fuzz suite's beforeAll with an early return
(cancelling from beforeAll aborts the suite) and add the missing
`assume(icebergAvailable)` to one native-scan test.
Expressions job: Spark 4.2 normalizes NaN / -0.0 for array_distinct and
the array set operations by wrapping their inputs as
`KnownFloatingPointNormalized(ArrayTransform(arr, x ->
NormalizeNaNAndZero(x)))`.
The serde only handled a scalar `NormalizeNaNAndZero` child and fell back.
Since `KnownFloatingPointNormalized` is a runtime no-op tag, serialize any
other child directly and let its serde (the ArrayTransform codegen
dispatcher) carry the normalization, keeping these operations native.
---
pom.xml | 14 +-
spark/pom.xml | 4 +-
.../org/apache/comet/codegen/CometArrayData.scala | 4 +-
.../apache/comet/serde/contraintExpressions.scala | 43 ++-
.../apache/comet/shims/CometInternalRowShim.scala | 5 +-
.../apache/comet/shims/CometInternalRowShim.scala | 4 +-
.../apache/comet/shims/CometInternalRowShim.scala | 0
.../apache/comet/shims/CometInternalRowShim.scala | 15 +-
.../approved-plans-v1_4-spark4_2/q2/extended.txt | 60 ----
.../approved-plans-v1_4-spark4_2/q5/extended.txt | 107 ------
.../approved-plans-v1_4-spark4_2/q54/extended.txt | 113 -------
.../approved-plans-v2_7-spark4_2/q5a/extended.txt | 320 ------------------
.../approved-plans-v2_7-spark4_2/q77a/extended.txt | 362 +++++++++++++++++++++
.../org/apache/comet/CometFuzzIcebergBase.scala | 25 +-
.../org/apache/comet/CometIcebergNativeSuite.scala | 1 +
.../org/apache/comet/CometIcebergTestBase.scala | 18 +-
.../scala/org/apache/comet/CometS3TestBase.scala | 17 +-
17 files changed, 459 insertions(+), 653 deletions(-)
diff --git a/pom.xml b/pom.xml
index 6739c01f29..318e3ca51e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -732,13 +732,12 @@ under the License.
</profile>
<profile>
- <!-- WIP: Spark 4.2 preview support. Targets the 4.2.0-preview4 release;
tests are not
- wired up yet, this profile only aims to compile main sources. -->
+ <!-- Experimental Spark 4.2 support, targeting the 4.2.0 release. -->
<id>spark-4.2</id>
<properties>
<scala.version>2.13.18</scala.version>
<scala.binary.version>2.13</scala.binary.version>
- <spark.version>4.2.0-preview4</spark.version>
+ <spark.version>4.2.0</spark.version>
<spark.version.short>4.2</spark.version.short>
<parquet.version>1.17.0</parquet.version>
<semanticdb.version>4.13.6</semanticdb.version>
@@ -1252,6 +1251,15 @@ under the License.
<ignoreClass>com.google.thirdparty.publicsuffix.PublicSuffixType</ignoreClass>
</ignoreClasses>
</dependency>
+ <dependency>
+ <groupId>org.apache.datasketches</groupId>
+ <artifactId>datasketches-memory</artifactId>
+ <ignoreClasses>
+ <!-- Spark 4.2.0 bundles its own copy of this class in
spark-catalyst,
+ which collides with the transitive
datasketches-memory jar. -->
+
<ignoreClass>org.apache.datasketches.memory.internal.ResourceImpl</ignoreClass>
+ </ignoreClasses>
+ </dependency>
</dependencies>
<findAllDuplicates>true</findAllDuplicates>
<ignoreWhenIdentical>true</ignoreWhenIdentical>
diff --git a/spark/pom.xml b/spark/pom.xml
index 26ca9f6dca..cc3f278ee3 100644
--- a/spark/pom.xml
+++ b/spark/pom.xml
@@ -315,7 +315,9 @@ under the License.
<version>1.10.0</version>
<scope>test</scope>
</dependency>
- <!-- Jetty 11.x for Spark 4.2 (jakarta.servlet); matches Spark
4.2.0-preview4's jetty.version -->
+ <!-- Jetty 11.x for Spark 4.2 (jakarta.servlet). Spark 4.2.0 itself
ships Jetty 12, but
+ the Iceberg REST catalog test helper needs jetty-servlet, which
Jetty 12 replaced
+ with the jetty-ee10-servlet artifact, so the test classpath stays
on 11.x. -->
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
diff --git a/spark/src/main/scala/org/apache/comet/codegen/CometArrayData.scala
b/spark/src/main/scala/org/apache/comet/codegen/CometArrayData.scala
index 308d1e9d96..53d52ff2e0 100644
--- a/spark/src/main/scala/org/apache/comet/codegen/CometArrayData.scala
+++ b/spark/src/main/scala/org/apache/comet/codegen/CometArrayData.scala
@@ -35,8 +35,8 @@ import org.apache.comet.shims.CometInternalRowShim
*
* `ArrayData` and `InternalRow` are sibling abstract classes, so a base aimed
at one cannot serve
* the other. The shared `get(ordinal, dataType)` dispatch lives in
- * [[CometSpecializedGettersDispatch]]. Mixes in [[CometInternalRowShim]] so
Spark 4.x's
- * `getVariant` / `getGeography` / `getGeometry` get throwing defaults.
+ * [[CometSpecializedGettersDispatch]]. Mixes in [[CometInternalRowShim]] so
the getters Spark 4.x
+ * adds to `SpecializedGetters` get throwing defaults.
*/
abstract class CometArrayData extends ArrayData with CometInternalRowShim {
diff --git
a/spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala
b/spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala
index 6eeba5e615..9b30357f9f 100644
--- a/spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala
@@ -27,9 +27,6 @@ import
org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, optExprWithFa
object CometKnownFloatingPointNormalized
extends CometExpressionSerde[KnownFloatingPointNormalized] {
- override def getUnsupportedReasons(): Seq[String] = Seq(
- "Only supports `NormalizeNaNAndZero` child expressions")
-
override def getSupportLevel(expr: KnownFloatingPointNormalized):
SupportLevel = {
expr.child match {
case normalized: NormalizeNaNAndZero =>
@@ -40,9 +37,10 @@ object CometKnownFloatingPointNormalized
Unsupported(Some(s"Unsupported datatype ${wrapped.dataType}"))
}
case _ =>
- Unsupported(
- Some(
- "KnownFloatingPointNormalized only supports NormalizeNaNAndZero
child expressions"))
+ // Nested normalization (array / struct / map).
KnownFloatingPointNormalized is a runtime
+ // no-op tag, so defer to the child's serde: convert serializes the
child directly and
+ // falls back gracefully if it is unsupported.
+ Compatible()
}
}
@@ -51,18 +49,31 @@ object CometKnownFloatingPointNormalized
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
- val wrapped = expr.child.asInstanceOf[NormalizeNaNAndZero].child
+ expr.child match {
+ case normalize: NormalizeNaNAndZero =>
+ // Scalar float/double normalization: unwrap and emit the native
NormalizeNaNAndZero node.
+ // getSupportLevel has already verified the datatype is serializable.
+ val wrapped = normalize.child
+ val dataType = serializeDataType(wrapped.dataType).get
+ val ex = exprToProtoInternal(wrapped, inputs, binding)
+ val optExpr = ex.map { child =>
+ val builder = ExprOuterClass.NormalizeNaNAndZero
+ .newBuilder()
+ .setChild(child)
+ .setDatatype(dataType)
+
ExprOuterClass.Expr.newBuilder().setNormalizeNanAndZero(builder).build()
+ }
+ optExprWithFallbackReason(optExpr, expr, wrapped)
- val dataType = serializeDataType(wrapped.dataType).get
- val ex = exprToProtoInternal(wrapped, inputs, binding)
- val optExpr = ex.map { child =>
- val builder = ExprOuterClass.NormalizeNaNAndZero
- .newBuilder()
- .setChild(child)
- .setDatatype(dataType)
- ExprOuterClass.Expr.newBuilder().setNormalizeNanAndZero(builder).build()
+ case child =>
+ // Nested normalization (array / struct / map). Spark 4.2 normalizes
the inputs to
+ // array_distinct and the array set operations by wrapping them as,
e.g.,
+ // `KnownFloatingPointNormalized(ArrayTransform(arr, x ->
NormalizeNaNAndZero(x)))`.
+ // `KnownFloatingPointNormalized` is a runtime no-op tag, so serialize
the child directly
+ // and let its serde (e.g. the ArrayTransform codegen dispatcher)
carry the normalization.
+ val optExpr = exprToProtoInternal(child, inputs, binding)
+ optExprWithFallbackReason(optExpr, expr, child)
}
- optExprWithFallbackReason(optExpr, expr, wrapped)
}
}
diff --git
a/spark/src/main/spark-3.x/org/apache/comet/shims/CometInternalRowShim.scala
b/spark/src/main/spark-3.x/org/apache/comet/shims/CometInternalRowShim.scala
index 18b3a4e6b3..ee85cdb8e9 100644
--- a/spark/src/main/spark-3.x/org/apache/comet/shims/CometInternalRowShim.scala
+++ b/spark/src/main/spark-3.x/org/apache/comet/shims/CometInternalRowShim.scala
@@ -21,7 +21,8 @@ package org.apache.comet.shims
/**
* Per-profile shim mixed into `CometInternalRow` and `CometArrayData`. Spark
4.x adds abstract
- * `SpecializedGetters` getters (`getVariant` in 4.0, `getGeography` and
`getGeometry` in 4.1)
- * that subclasses must implement; Spark 3.x has none, so this trait is empty.
+ * `SpecializedGetters` getters (`getVariant` in 4.0, `getGeography` and
`getGeometry` in 4.1,
+ * `getBinaryView` in 4.2) that subclasses must implement; Spark 3.x has none,
so this trait is
+ * empty.
*/
trait CometInternalRowShim
diff --git
a/spark/src/main/spark-4.0/org/apache/comet/shims/CometInternalRowShim.scala
b/spark/src/main/spark-4.0/org/apache/comet/shims/CometInternalRowShim.scala
index b855fe3a91..253a7f8f3e 100644
--- a/spark/src/main/spark-4.0/org/apache/comet/shims/CometInternalRowShim.scala
+++ b/spark/src/main/spark-4.0/org/apache/comet/shims/CometInternalRowShim.scala
@@ -24,8 +24,8 @@ import org.apache.spark.unsafe.types.VariantVal
/**
* Throwing defaults for Spark 4.0 `SpecializedGetters` additions:
`getVariant`. Mixed into
* `CometInternalRow` and `CometArrayData` so the codegen kernel's subclasses
satisfy the
- * abstract-method check at class-load time. 4.1 also adds `getGeography` /
`getGeometry` (see the
- * spark-4.1 shim).
+ * abstract-method check at class-load time. 4.1 also adds `getGeography` /
`getGeometry`, and 4.2
+ * replaces those with `getBinaryView` (see the spark-4.1 and spark-4.2 shims).
*/
trait CometInternalRowShim {
def getVariant(ordinal: Int): VariantVal =
diff --git
a/spark/src/main/spark-4.1+/org/apache/comet/shims/CometInternalRowShim.scala
b/spark/src/main/spark-4.1/org/apache/comet/shims/CometInternalRowShim.scala
similarity index 100%
copy from
spark/src/main/spark-4.1+/org/apache/comet/shims/CometInternalRowShim.scala
copy to
spark/src/main/spark-4.1/org/apache/comet/shims/CometInternalRowShim.scala
diff --git
a/spark/src/main/spark-4.1+/org/apache/comet/shims/CometInternalRowShim.scala
b/spark/src/main/spark-4.2/org/apache/comet/shims/CometInternalRowShim.scala
similarity index 71%
rename from
spark/src/main/spark-4.1+/org/apache/comet/shims/CometInternalRowShim.scala
rename to
spark/src/main/spark-4.2/org/apache/comet/shims/CometInternalRowShim.scala
index ce4cb7c06f..36ee7f4c79 100644
---
a/spark/src/main/spark-4.1+/org/apache/comet/shims/CometInternalRowShim.scala
+++ b/spark/src/main/spark-4.2/org/apache/comet/shims/CometInternalRowShim.scala
@@ -19,22 +19,19 @@
package org.apache.comet.shims
-import org.apache.spark.unsafe.types.{GeographyVal, GeometryVal, VariantVal}
+import org.apache.spark.unsafe.types.{BinaryView, VariantVal}
/**
- * Throwing defaults for Spark 4.x `SpecializedGetters` additions:
`getVariant` (4.0),
- * `getGeography` and `getGeometry` (4.1). Mixed into `CometInternalRow` and
`CometArrayData`.
+ * Throwing defaults for Spark 4.x `SpecializedGetters` additions:
`getVariant` (4.0) and
+ * `getBinaryView` (4.2). Spark 4.2 dropped 4.1's `getGeography` /
`getGeometry` in favour of the
+ * single `getBinaryView` accessor. Mixed into `CometInternalRow` and
`CometArrayData`.
*/
trait CometInternalRowShim {
def getVariant(ordinal: Int): VariantVal =
throw new UnsupportedOperationException(
s"${getClass.getSimpleName}: getVariant not supported")
- def getGeography(ordinal: Int): GeographyVal =
+ def getBinaryView(ordinal: Int): BinaryView =
throw new UnsupportedOperationException(
- s"${getClass.getSimpleName}: getGeography not supported")
-
- def getGeometry(ordinal: Int): GeometryVal =
- throw new UnsupportedOperationException(
- s"${getClass.getSimpleName}: getGeometry not supported")
+ s"${getClass.getSimpleName}: getBinaryView not supported")
}
diff --git
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q2/extended.txt
b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q2/extended.txt
deleted file mode 100644
index 60c768385c..0000000000
---
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q2/extended.txt
+++ /dev/null
@@ -1,60 +0,0 @@
-CometNativeColumnarToRow
-+- CometSort
- +- CometExchange
- +- CometProject
- +- CometBroadcastHashJoin
- :- CometProject
- : +- CometBroadcastHashJoin
- : :- CometHashAggregate
- : : +- CometExchange
- : : +- CometHashAggregate
- : : +- CometUnion
- : : :- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometNativeScan parquet
spark_catalog.default.web_sales
- : : : +- CometBroadcastExchange
- : : : +- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometNativeScan parquet
spark_catalog.default.catalog_sales
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.date_dim
- +- CometBroadcastExchange
- +- CometProject
- +- CometBroadcastHashJoin
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometUnion
- : :- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometNativeScan parquet
spark_catalog.default.web_sales
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometProject
- : : +- CometNativeScan parquet
spark_catalog.default.catalog_sales
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.date_dim
- +- CometBroadcastExchange
- +- CometProject
- +- CometFilter
- +- CometNativeScan parquet
spark_catalog.default.date_dim
-
-Comet accelerated 57 out of 57 eligible operators (100%). Final plan contains
1 transitions between Spark and Comet.
\ No newline at end of file
diff --git
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q5/extended.txt
b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q5/extended.txt
deleted file mode 100644
index 16de14eac2..0000000000
---
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q5/extended.txt
+++ /dev/null
@@ -1,107 +0,0 @@
-CometNativeColumnarToRow
-+- CometTakeOrderedAndProject
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometExpand
- +- CometUnion
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometUnion
- : : :- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan parquet
spark_catalog.default.store_sales
- : : : : +- CometSubqueryBroadcast
- : : : : +-
CometBroadcastExchange
- : : : : +- CometProject
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : : +- CometBroadcastExchange
- : : : +- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.store_returns
- : : : +- ReusedSubquery
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.store
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometUnion
- : : :- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan parquet
spark_catalog.default.catalog_sales
- : : : : +- ReusedSubquery
- : : : +- CometBroadcastExchange
- : : : +- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.catalog_returns
- : : : +- ReusedSubquery
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.catalog_page
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometProject
- +- CometBroadcastHashJoin
- :- CometUnion
- : :- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.web_sales
- : : : +- ReusedSubquery
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.date_dim
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometBroadcastExchange
- : : : +- CometNativeScan parquet
spark_catalog.default.web_returns
- : : : +- ReusedSubquery
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan
parquet spark_catalog.default.web_sales
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.date_dim
- +- CometBroadcastExchange
- +- CometProject
- +- CometFilter
- +- CometNativeScan parquet
spark_catalog.default.web_site
-
-Comet accelerated 98 out of 104 eligible operators (94%). Final plan contains
1 transitions between Spark and Comet.
\ No newline at end of file
diff --git
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q54/extended.txt
b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q54/extended.txt
deleted file mode 100644
index 3cabbbdf79..0000000000
---
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q54/extended.txt
+++ /dev/null
@@ -1,113 +0,0 @@
-CometNativeColumnarToRow
-+- CometTakeOrderedAndProject
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometProject
- +- CometBroadcastHashJoin
- :- CometProject
- : +- CometBroadcastHashJoin
- : :- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometHashAggregate
- : : : : +- CometExchange
- : : : : +- CometHashAggregate
- : : : : +- CometProject
- : : : : +-
CometBroadcastHashJoin
- : : : : :- CometProject
- : : : : : +-
CometBroadcastHashJoin
- : : : : : :- CometUnion
- : : : : : : :-
CometProject
- : : : : : : : +-
CometBroadcastHashJoin
- : : : : : : : :-
CometProject
- : : : : : : : :
+- CometFilter
- : : : : : : : :
+- CometNativeScan parquet spark_catalog.default.catalog_sales
- : : : : : : : :
+- CometSubqueryBroadcast
- : : : : : : : :
+- CometBroadcastExchange
- : : : : : : : :
+- CometProject
- : : : : : : : :
+- CometFilter
- : : : : : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
- : : : : : : : +-
CometBroadcastExchange
- : : : : : : :
+- CometProject
- : : : : : : :
+- CometFilter
- : : : : : : :
+- CometNativeScan parquet spark_catalog.default.item
- : : : : : : +-
CometProject
- : : : : : : +-
CometBroadcastHashJoin
- : : : : : : :-
CometProject
- : : : : : : :
+- CometFilter
- : : : : : : :
+- CometNativeScan parquet spark_catalog.default.web_sales
- : : : : : : :
+- ReusedSubquery
- : : : : : : +-
CometBroadcastExchange
- : : : : : :
+- CometProject
- : : : : : :
+- CometFilter
- : : : : : :
+- CometNativeScan parquet spark_catalog.default.item
- : : : : : +-
CometBroadcastExchange
- : : : : : +-
CometProject
- : : : : : +-
CometFilter
- : : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : : : +-
CometBroadcastExchange
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.customer
- : : : +- CometBroadcastExchange
- : : : +- CometFilter
- : : : +- CometNativeScan parquet
spark_catalog.default.store_sales
- : : : +-
CometSubqueryBroadcast
- : : : +-
CometBroadcastExchange
- : : : +- CometProject
- : : : +-
CometFilter
- : : : : :-
ReusedSubquery
- : : : : +-
ReusedSubquery
- : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : : :-
Subquery
- : : : :
+- CometNativeColumnarToRow
- : : : :
+- CometHashAggregate
- : : : :
+- CometExchange
- : : : :
+- CometHashAggregate
- : : : :
+- CometProject
- : : : :
+- CometFilter
- : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
- : : : +-
Subquery
- : : :
+- CometNativeColumnarToRow
- : : :
+- CometHashAggregate
- : : :
+- CometExchange
- : : :
+- CometHashAggregate
- : : :
+- CometProject
- : : :
+- CometFilter
- : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.customer_address
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.store
- +- CometBroadcastExchange
- +- CometProject
- +- CometFilter
- : :- ReusedSubquery
- : +- ReusedSubquery
- +- CometNativeScan parquet
spark_catalog.default.date_dim
- :- Subquery
- : +- CometNativeColumnarToRow
- : +- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometFilter
- : +-
CometNativeScan parquet spark_catalog.default.date_dim
- +- Subquery
- +- CometNativeColumnarToRow
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometProject
- +- CometFilter
- +-
CometNativeScan parquet spark_catalog.default.date_dim
-
-Comet accelerated 95 out of 106 eligible operators (89%). Final plan contains
5 transitions between Spark and Comet.
\ No newline at end of file
diff --git
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q5a/extended.txt
b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q5a/extended.txt
deleted file mode 100644
index 5af7a1556c..0000000000
---
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q5a/extended.txt
+++ /dev/null
@@ -1,320 +0,0 @@
-CometNativeColumnarToRow
-+- CometTakeOrderedAndProject
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometUnion
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometUnion
- : :- CometHashAggregate
- : : +- CometExchange
- : : +- CometHashAggregate
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometUnion
- : : : :- CometProject
- : : : : +- CometBroadcastHashJoin
- : : : : :- CometProject
- : : : : : +- CometFilter
- : : : : : +- CometNativeScan
parquet spark_catalog.default.store_sales
- : : : : : +-
CometSubqueryBroadcast
- : : : : : +-
CometBroadcastExchange
- : : : : : +-
CometProject
- : : : : : +-
CometFilter
- : : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : : : +- CometBroadcastExchange
- : : : : +- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan
parquet spark_catalog.default.date_dim
- : : : +- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan
parquet spark_catalog.default.store_returns
- : : : : +-
ReusedSubquery
- : : : +- CometBroadcastExchange
- : : : +- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan
parquet spark_catalog.default.date_dim
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.store
- : :- CometHashAggregate
- : : +- CometExchange
- : : +- CometHashAggregate
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometUnion
- : : : :- CometProject
- : : : : +- CometBroadcastHashJoin
- : : : : :- CometProject
- : : : : : +- CometFilter
- : : : : : +- CometNativeScan
parquet spark_catalog.default.catalog_sales
- : : : : : +-
ReusedSubquery
- : : : : +- CometBroadcastExchange
- : : : : +- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan
parquet spark_catalog.default.date_dim
- : : : +- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan
parquet spark_catalog.default.catalog_returns
- : : : : +-
ReusedSubquery
- : : : +- CometBroadcastExchange
- : : : +- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan
parquet spark_catalog.default.date_dim
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan parquet
spark_catalog.default.catalog_page
- : +- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometUnion
- : : :- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +- CometNativeScan
parquet spark_catalog.default.web_sales
- : : : : +-
ReusedSubquery
- : : : +- CometBroadcastExchange
- : : : +- CometProject
- : : : +- CometFilter
- : : : +- CometNativeScan
parquet spark_catalog.default.date_dim
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometBroadcastHashJoin
- : : : :-
CometBroadcastExchange
- : : : : +- CometNativeScan
parquet spark_catalog.default.web_returns
- : : : : +-
ReusedSubquery
- : : : +- CometProject
- : : : +- CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.web_sales
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan
parquet spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan parquet
spark_catalog.default.web_site
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometUnion
- : :- CometHashAggregate
- : : +- CometExchange
- : : +- CometHashAggregate
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometUnion
- : : : :- CometProject
- : : : : +-
CometBroadcastHashJoin
- : : : : :- CometProject
- : : : : : +- CometFilter
- : : : : : +-
CometNativeScan parquet spark_catalog.default.store_sales
- : : : : : +-
CometSubqueryBroadcast
- : : : : : +-
CometBroadcastExchange
- : : : : :
+- CometProject
- : : : : :
+- CometFilter
- : : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
- : : : : +-
CometBroadcastExchange
- : : : : +- CometProject
- : : : : +-
CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : : +- CometProject
- : : : +-
CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.store_returns
- : : : : +-
ReusedSubquery
- : : : +-
CometBroadcastExchange
- : : : +- CometProject
- : : : +-
CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan
parquet spark_catalog.default.store
- : :- CometHashAggregate
- : : +- CometExchange
- : : +- CometHashAggregate
- : : +- CometProject
- : : +- CometBroadcastHashJoin
- : : :- CometUnion
- : : : :- CometProject
- : : : : +-
CometBroadcastHashJoin
- : : : : :- CometProject
- : : : : : +- CometFilter
- : : : : : +-
CometNativeScan parquet spark_catalog.default.catalog_sales
- : : : : : +-
ReusedSubquery
- : : : : +-
CometBroadcastExchange
- : : : : +- CometProject
- : : : : +-
CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : : +- CometProject
- : : : +-
CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.catalog_returns
- : : : : +-
ReusedSubquery
- : : : +-
CometBroadcastExchange
- : : : +- CometProject
- : : : +-
CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : +- CometBroadcastExchange
- : : +- CometProject
- : : +- CometFilter
- : : +- CometNativeScan
parquet spark_catalog.default.catalog_page
- : +- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometUnion
- : : :- CometProject
- : : : +-
CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.web_sales
- : : : : +-
ReusedSubquery
- : : : +-
CometBroadcastExchange
- : : : +- CometProject
- : : : +-
CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : +- CometProject
- : : +-
CometBroadcastHashJoin
- : : :- CometProject
- : : : +-
CometBroadcastHashJoin
- : : : :-
CometBroadcastExchange
- : : : : +-
CometNativeScan parquet spark_catalog.default.web_returns
- : : : : +-
ReusedSubquery
- : : : +-
CometProject
- : : : +-
CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.web_sales
- : : +-
CometBroadcastExchange
- : : +- CometProject
- : : +-
CometFilter
- : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan
parquet spark_catalog.default.web_site
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometUnion
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometUnion
- : : :- CometProject
- : : : +-
CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.store_sales
- : : : : +-
CometSubqueryBroadcast
- : : : : +-
CometBroadcastExchange
- : : : :
+- CometProject
- : : : :
+- CometFilter
- : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
- : : : +-
CometBroadcastExchange
- : : : +- CometProject
- : : : +-
CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : +- CometProject
- : : +-
CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.store_returns
- : : : +-
ReusedSubquery
- : : +-
CometBroadcastExchange
- : : +- CometProject
- : : +-
CometFilter
- : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan
parquet spark_catalog.default.store
- :- CometHashAggregate
- : +- CometExchange
- : +- CometHashAggregate
- : +- CometProject
- : +- CometBroadcastHashJoin
- : :- CometUnion
- : : :- CometProject
- : : : +-
CometBroadcastHashJoin
- : : : :- CometProject
- : : : : +- CometFilter
- : : : : +-
CometNativeScan parquet spark_catalog.default.catalog_sales
- : : : : +-
ReusedSubquery
- : : : +-
CometBroadcastExchange
- : : : +- CometProject
- : : : +-
CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : : +- CometProject
- : : +-
CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.catalog_returns
- : : : +-
ReusedSubquery
- : : +-
CometBroadcastExchange
- : : +- CometProject
- : : +-
CometFilter
- : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : +- CometBroadcastExchange
- : +- CometProject
- : +- CometFilter
- : +- CometNativeScan
parquet spark_catalog.default.catalog_page
- +- CometHashAggregate
- +- CometExchange
- +- CometHashAggregate
- +- CometProject
- +- CometBroadcastHashJoin
- :- CometUnion
- : :- CometProject
- : : +-
CometBroadcastHashJoin
- : : :- CometProject
- : : : +- CometFilter
- : : : +-
CometNativeScan parquet spark_catalog.default.web_sales
- : : : +-
ReusedSubquery
- : : +-
CometBroadcastExchange
- : : +- CometProject
- : : +-
CometFilter
- : : +-
CometNativeScan parquet spark_catalog.default.date_dim
- : +- CometProject
- : +-
CometBroadcastHashJoin
- : :- CometProject
- : : +-
CometBroadcastHashJoin
- : : :-
CometBroadcastExchange
- : : : +-
CometNativeScan parquet spark_catalog.default.web_returns
- : : : +-
ReusedSubquery
- : : +-
CometProject
- : : +-
CometFilter
- : : +-
CometNativeScan parquet spark_catalog.default.web_sales
- : +-
CometBroadcastExchange
- : +- CometProject
- : +-
CometFilter
- : +-
CometNativeScan parquet spark_catalog.default.date_dim
- +- CometBroadcastExchange
- +- CometProject
- +- CometFilter
- +- CometNativeScan
parquet spark_catalog.default.web_site
-
-Comet accelerated 299 out of 317 eligible operators (94%). Final plan contains
1 transitions between Spark and Comet.
\ No newline at end of file
diff --git
a/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q77a/extended.txt
b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q77a/extended.txt
new file mode 100644
index 0000000000..cf441b7592
--- /dev/null
+++
b/spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q77a/extended.txt
@@ -0,0 +1,362 @@
+CometNativeColumnarToRow
++- CometTakeOrderedAndProject
+ +- CometHashAggregate
+ +- CometColumnarExchange
+ +- HashAggregate
+ +- Union
+ :- HashAggregate [COMET: Comet aggregate that merges
intermediate buffers requires a Comet child aggregate when the intermediate
buffer formats are incompatible with Spark. Incompatible aggregate function(s):
sum]
+ : +- Exchange
+ : +- HashAggregate
+ : +- Union
+ : :- CometNativeColumnarToRow
+ : : +- CometProject
+ : : +- CometBroadcastHashJoin
+ : : :- CometHashAggregate
+ : : : +- CometExchange
+ : : : +- CometHashAggregate
+ : : : +- CometProject
+ : : : +- CometBroadcastHashJoin
+ : : : :- CometProject
+ : : : : +- CometBroadcastHashJoin
+ : : : : :- CometFilter
+ : : : : : +- CometNativeScan
parquet spark_catalog.default.store_sales
+ : : : : : +-
CometSubqueryBroadcast
+ : : : : : +-
CometBroadcastExchange
+ : : : : : +-
CometProject
+ : : : : : +-
CometFilter
+ : : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : : : : +-
CometBroadcastExchange
+ : : : : +- CometProject
+ : : : : +- CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : : +- CometBroadcastExchange
+ : : : +- CometFilter
+ : : : +- CometNativeScan
parquet spark_catalog.default.store
+ : : +- CometBroadcastExchange
+ : : +- CometHashAggregate
+ : : +- CometExchange
+ : : +- CometHashAggregate
+ : : +- CometProject
+ : : +- CometBroadcastHashJoin
+ : : :- CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :- CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.store_returns
+ : : : : +-
ReusedSubquery
+ : : : +-
CometBroadcastExchange
+ : : : +- CometProject
+ : : : +-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : +- CometBroadcastExchange
+ : : +- CometFilter
+ : : +- CometNativeScan
parquet spark_catalog.default.store
+ : :- Project
+ : : +- BroadcastNestedLoopJoin
+ : : :- BroadcastExchange
+ : : : +- CometNativeColumnarToRow
+ : : : +- CometHashAggregate
+ : : : +- CometExchange
+ : : : +- CometHashAggregate
+ : : : +- CometProject
+ : : : +- CometBroadcastHashJoin
+ : : : :- CometNativeScan
parquet spark_catalog.default.catalog_sales
+ : : : : +- ReusedSubquery
+ : : : +- CometBroadcastExchange
+ : : : +- CometProject
+ : : : +- CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : +- Project
+ : : : :- Subquery
+ : : : : +- CometNativeColumnarToRow
+ : : : : +- CometProject
+ : : : : +- CometHashAggregate
+ : : : : +- CometExchange
+ : : : : +- CometHashAggregate
+ : : : : +- CometProject
+ : : : : +-
CometBroadcastHashJoin
+ : : : : :-
CometNativeScan parquet spark_catalog.default.catalog_returns
+ : : : : : +-
ReusedSubquery
+ : : : : +-
CometBroadcastExchange
+ : : : : +-
CometProject
+ : : : : +-
CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : : +- ReusedSubquery
+ : : +- Scan OneRowRelation [COMET: Scan
OneRowRelation is not supported]
+ : +- CometNativeColumnarToRow
+ : +- CometProject
+ : +- CometBroadcastHashJoin
+ : :- CometHashAggregate
+ : : +- CometExchange
+ : : +- CometHashAggregate
+ : : +- CometProject
+ : : +- CometBroadcastHashJoin
+ : : :- CometProject
+ : : : +- CometBroadcastHashJoin
+ : : : :- CometFilter
+ : : : : +- CometNativeScan
parquet spark_catalog.default.web_sales
+ : : : : +-
ReusedSubquery
+ : : : +-
CometBroadcastExchange
+ : : : +- CometProject
+ : : : +- CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : +- CometBroadcastExchange
+ : : +- CometFilter
+ : : +- CometNativeScan
parquet spark_catalog.default.web_page
+ : +- CometBroadcastExchange
+ : +- CometHashAggregate
+ : +- CometExchange
+ : +- CometHashAggregate
+ : +- CometProject
+ : +- CometBroadcastHashJoin
+ : :- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :- CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.web_returns
+ : : : +-
ReusedSubquery
+ : : +-
CometBroadcastExchange
+ : : +- CometProject
+ : : +-
CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : +- CometBroadcastExchange
+ : +- CometFilter
+ : +- CometNativeScan
parquet spark_catalog.default.web_page
+ :- HashAggregate [COMET: Comet aggregate that merges
intermediate buffers requires a Comet child aggregate when the intermediate
buffer formats are incompatible with Spark. Incompatible aggregate function(s):
sum]
+ : +- Exchange
+ : +- HashAggregate
+ : +- HashAggregate [COMET: Comet aggregate that merges
intermediate buffers requires a Comet child aggregate when the intermediate
buffer formats are incompatible with Spark. Incompatible aggregate function(s):
sum]
+ : +- Exchange
+ : +- HashAggregate
+ : +- Union
+ : :- CometNativeColumnarToRow
+ : : +- CometProject
+ : : +- CometBroadcastHashJoin
+ : : :- CometHashAggregate
+ : : : +- CometExchange
+ : : : +- CometHashAggregate
+ : : : +- CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :- CometProject
+ : : : : +-
CometBroadcastHashJoin
+ : : : : :-
CometFilter
+ : : : : : +-
CometNativeScan parquet spark_catalog.default.store_sales
+ : : : : : +-
CometSubqueryBroadcast
+ : : : : :
+- CometBroadcastExchange
+ : : : : :
+- CometProject
+ : : : : :
+- CometFilter
+ : : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : : : : +-
CometBroadcastExchange
+ : : : : +-
CometProject
+ : : : : +-
CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : : +-
CometBroadcastExchange
+ : : : +- CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.store
+ : : +- CometBroadcastExchange
+ : : +- CometHashAggregate
+ : : +- CometExchange
+ : : +- CometHashAggregate
+ : : +- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :- CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :-
CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.store_returns
+ : : : :
+- ReusedSubquery
+ : : : +-
CometBroadcastExchange
+ : : : +-
CometProject
+ : : : +-
CometFilter
+ : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : : +-
CometBroadcastExchange
+ : : +-
CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.store
+ : :- Project
+ : : +- BroadcastNestedLoopJoin
+ : : :- BroadcastExchange
+ : : : +- CometNativeColumnarToRow
+ : : : +- CometHashAggregate
+ : : : +- CometExchange
+ : : : +- CometHashAggregate
+ : : : +- CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :-
CometNativeScan parquet spark_catalog.default.catalog_sales
+ : : : : +-
ReusedSubquery
+ : : : +-
CometBroadcastExchange
+ : : : +-
CometProject
+ : : : +-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : +- Project
+ : : : :- Subquery
+ : : : : +- CometNativeColumnarToRow
+ : : : : +- CometProject
+ : : : : +- CometHashAggregate
+ : : : : +- CometExchange
+ : : : : +-
CometHashAggregate
+ : : : : +-
CometProject
+ : : : : +-
CometBroadcastHashJoin
+ : : : : :-
CometNativeScan parquet spark_catalog.default.catalog_returns
+ : : : : :
+- ReusedSubquery
+ : : : : +-
CometBroadcastExchange
+ : : : : +-
CometProject
+ : : : :
+- CometFilter
+ : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : : : +- ReusedSubquery
+ : : +- Scan OneRowRelation [COMET:
Scan OneRowRelation is not supported]
+ : +- CometNativeColumnarToRow
+ : +- CometProject
+ : +- CometBroadcastHashJoin
+ : :- CometHashAggregate
+ : : +- CometExchange
+ : : +- CometHashAggregate
+ : : +- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :- CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :-
CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.web_sales
+ : : : : +-
ReusedSubquery
+ : : : +-
CometBroadcastExchange
+ : : : +-
CometProject
+ : : : +-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : +-
CometBroadcastExchange
+ : : +- CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.web_page
+ : +- CometBroadcastExchange
+ : +- CometHashAggregate
+ : +- CometExchange
+ : +- CometHashAggregate
+ : +- CometProject
+ : +-
CometBroadcastHashJoin
+ : :- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.web_returns
+ : : :
+- ReusedSubquery
+ : : +-
CometBroadcastExchange
+ : : +-
CometProject
+ : : +-
CometFilter
+ : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : +-
CometBroadcastExchange
+ : +-
CometFilter
+ : +-
CometNativeScan parquet spark_catalog.default.web_page
+ +- HashAggregate [COMET: Comet aggregate that merges
intermediate buffers requires a Comet child aggregate when the intermediate
buffer formats are incompatible with Spark. Incompatible aggregate function(s):
sum]
+ +- Exchange
+ +- HashAggregate
+ +- HashAggregate [COMET: Comet aggregate that merges
intermediate buffers requires a Comet child aggregate when the intermediate
buffer formats are incompatible with Spark. Incompatible aggregate function(s):
sum]
+ +- Exchange
+ +- HashAggregate
+ +- Union
+ :- CometNativeColumnarToRow
+ : +- CometProject
+ : +- CometBroadcastHashJoin
+ : :- CometHashAggregate
+ : : +- CometExchange
+ : : +- CometHashAggregate
+ : : +- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :- CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :-
CometFilter
+ : : : : +-
CometNativeScan parquet spark_catalog.default.store_sales
+ : : : : +-
CometSubqueryBroadcast
+ : : : :
+- CometBroadcastExchange
+ : : : :
+- CometProject
+ : : : :
+- CometFilter
+ : : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : : : +-
CometBroadcastExchange
+ : : : +-
CometProject
+ : : : +-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : : +-
CometBroadcastExchange
+ : : +- CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.store
+ : +- CometBroadcastExchange
+ : +- CometHashAggregate
+ : +- CometExchange
+ : +- CometHashAggregate
+ : +- CometProject
+ : +-
CometBroadcastHashJoin
+ : :- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.store_returns
+ : : :
+- ReusedSubquery
+ : : +-
CometBroadcastExchange
+ : : +-
CometProject
+ : : +-
CometFilter
+ : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : +-
CometBroadcastExchange
+ : +-
CometFilter
+ : +-
CometNativeScan parquet spark_catalog.default.store
+ :- Project
+ : +- BroadcastNestedLoopJoin
+ : :- BroadcastExchange
+ : : +- CometNativeColumnarToRow
+ : : +- CometHashAggregate
+ : : +- CometExchange
+ : : +- CometHashAggregate
+ : : +- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :-
CometNativeScan parquet spark_catalog.default.catalog_sales
+ : : : +-
ReusedSubquery
+ : : +-
CometBroadcastExchange
+ : : +-
CometProject
+ : : +-
CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : +- Project
+ : : :- Subquery
+ : : : +- CometNativeColumnarToRow
+ : : : +- CometProject
+ : : : +- CometHashAggregate
+ : : : +- CometExchange
+ : : : +-
CometHashAggregate
+ : : : +-
CometProject
+ : : : +-
CometBroadcastHashJoin
+ : : : :-
CometNativeScan parquet spark_catalog.default.catalog_returns
+ : : : :
+- ReusedSubquery
+ : : : +-
CometBroadcastExchange
+ : : : +-
CometProject
+ : : :
+- CometFilter
+ : : :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ : : +- ReusedSubquery
+ : +- Scan OneRowRelation [COMET:
Scan OneRowRelation is not supported]
+ +- CometNativeColumnarToRow
+ +- CometProject
+ +- CometBroadcastHashJoin
+ :- CometHashAggregate
+ : +- CometExchange
+ : +- CometHashAggregate
+ : +- CometProject
+ : +-
CometBroadcastHashJoin
+ : :- CometProject
+ : : +-
CometBroadcastHashJoin
+ : : :-
CometFilter
+ : : : +-
CometNativeScan parquet spark_catalog.default.web_sales
+ : : : +-
ReusedSubquery
+ : : +-
CometBroadcastExchange
+ : : +-
CometProject
+ : : +-
CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.date_dim
+ : +-
CometBroadcastExchange
+ : +- CometFilter
+ : +-
CometNativeScan parquet spark_catalog.default.web_page
+ +- CometBroadcastExchange
+ +- CometHashAggregate
+ +- CometExchange
+ +- CometHashAggregate
+ +- CometProject
+ +-
CometBroadcastHashJoin
+ :- CometProject
+ : +-
CometBroadcastHashJoin
+ : :-
CometFilter
+ : : +-
CometNativeScan parquet spark_catalog.default.web_returns
+ : :
+- ReusedSubquery
+ : +-
CometBroadcastExchange
+ : +-
CometProject
+ : +-
CometFilter
+ :
+- CometNativeScan parquet spark_catalog.default.date_dim
+ +-
CometBroadcastExchange
+ +-
CometFilter
+ +-
CometNativeScan parquet spark_catalog.default.web_page
+
+Comet accelerated 288 out of 347 eligible operators (82%). Final plan contains
13 transitions between Spark and Comet.
\ No newline at end of file
diff --git a/spark/src/test/scala/org/apache/comet/CometFuzzIcebergBase.scala
b/spark/src/test/scala/org/apache/comet/CometFuzzIcebergBase.scala
index 92f36f534b..fbace69140 100644
--- a/spark/src/test/scala/org/apache/comet/CometFuzzIcebergBase.scala
+++ b/spark/src/test/scala/org/apache/comet/CometFuzzIcebergBase.scala
@@ -35,6 +35,7 @@ import
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.DecimalType
+import org.apache.comet.CometSparkSessionExtensions.isSpark42Plus
import org.apache.comet.testing.{DataGenOptions, FuzzDataGenerator,
SchemaGenOptions}
class CometFuzzIcebergBase extends CometTestBase with AdaptiveSparkPlanHelper {
@@ -42,13 +43,18 @@ class CometFuzzIcebergBase extends CometTestBase with
AdaptiveSparkPlanHelper {
var warehouseDir: File = null
val icebergTableName: String = "hadoop_catalog.db.fuzz_test"
- // Skip these tests if Iceberg is not available in classpath
+ // Skip these tests if Iceberg is not available in classpath. Spark 4.2 has
no published Iceberg
+ // spark-runtime yet; the build reuses the 4.0 runtime, whose `SparkView` is
binary-incompatible
+ // with Spark 4.2's `connector.catalog.View` (now a class, not an
interface), so treat Iceberg as
+ // unavailable on 4.2.
private def icebergAvailable: Boolean = {
- try {
- Class.forName("org.apache.iceberg.catalog.Catalog")
- true
- } catch {
- case _: ClassNotFoundException => false
+ !isSpark42Plus && {
+ try {
+ Class.forName("org.apache.iceberg.catalog.Catalog")
+ true
+ } catch {
+ case _: ClassNotFoundException => false
+ }
}
}
@@ -88,7 +94,11 @@ class CometFuzzIcebergBase extends CometTestBase with
AdaptiveSparkPlanHelper {
override def beforeAll(): Unit = {
super.beforeAll()
- assume(icebergAvailable, "Iceberg not available in classpath")
+ // Skip Iceberg setup when Iceberg is unavailable (e.g. on Spark 4.2,
which has no compatible
+ // runtime). `assume` cannot be used here: cancelling from beforeAll
aborts the whole suite.
+ // Each test carries its own `assume(icebergAvailable)` (see the `test`
override) so the tests
+ // report as cancelled instead.
+ if (!icebergAvailable) return
warehouseDir = Files.createTempDirectory("comet-iceberg-fuzz-test").toFile
val random = new Random(42)
withSQLConf(
@@ -146,6 +156,7 @@ class CometFuzzIcebergBase extends CometTestBase with
AdaptiveSparkPlanHelper {
override protected def test(testName: String, testTags: Tag*)(testFun: =>
Any)(implicit
pos: Position): Unit = {
super.test(testName, testTags: _*) {
+ assume(icebergAvailable, "Iceberg not available in classpath")
withSQLConf(
"spark.sql.catalog.hadoop_catalog" ->
"org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.hadoop_catalog.type" -> "hadoop",
diff --git
a/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala
b/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala
index 118d2e6143..7b9bd6b3d6 100644
--- a/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala
@@ -4311,6 +4311,7 @@ class CometIcebergNativeSuite
}
test("CometScanRule should report unsupported metadata columns") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
withTempIcebergDir { warehouseDir =>
withSQLConf(
"spark.sql.catalog.test_cat" ->
"org.apache.iceberg.spark.SparkCatalog",
diff --git a/spark/src/test/scala/org/apache/comet/CometIcebergTestBase.scala
b/spark/src/test/scala/org/apache/comet/CometIcebergTestBase.scala
index be7da6cbe8..33de4f3705 100644
--- a/spark/src/test/scala/org/apache/comet/CometIcebergTestBase.scala
+++ b/spark/src/test/scala/org/apache/comet/CometIcebergTestBase.scala
@@ -22,6 +22,7 @@ package org.apache.comet
import java.io.File
import java.nio.file.Files
+import org.apache.comet.CometSparkSessionExtensions.isSpark42Plus
import org.apache.comet.iceberg.IcebergReflection
/**
@@ -29,13 +30,18 @@ import org.apache.comet.iceberg.IcebergReflection
*/
trait CometIcebergTestBase {
+ // No Iceberg spark-runtime is published for Spark 4.2 yet, so the build
reuses the 4.0 runtime.
+ // That jar is binary-incompatible with Spark 4.2, whose
`connector.catalog.View` is a class
+ // rather than an interface, so loading `SparkView` throws
IncompatibleClassChangeError. Report
+ // Iceberg as unavailable on 4.2 so these suites skip until a compatible
runtime exists.
protected def icebergAvailable: Boolean =
- try {
- IcebergReflection.loadClass("org.apache.iceberg.catalog.Catalog")
- true
- } catch {
- case _: ClassNotFoundException => false
- }
+ !isSpark42Plus &&
+ (try {
+ IcebergReflection.loadClass("org.apache.iceberg.catalog.Catalog")
+ true
+ } catch {
+ case _: ClassNotFoundException => false
+ })
/**
* Whether the Iceberg library on the classpath is at least the given
(major, minor) version.
diff --git a/spark/src/test/scala/org/apache/comet/CometS3TestBase.scala
b/spark/src/test/scala/org/apache/comet/CometS3TestBase.scala
index 467a360483..f8cc9f4e42 100644
--- a/spark/src/test/scala/org/apache/comet/CometS3TestBase.scala
+++ b/spark/src/test/scala/org/apache/comet/CometS3TestBase.scala
@@ -29,6 +29,8 @@ import org.testcontainers.utility.DockerImageName
import org.apache.spark.SparkConf
import org.apache.spark.sql.CometTestBase
+import org.apache.comet.CometSparkSessionExtensions.isSpark42Plus
+
import software.amazon.awssdk.auth.credentials.{AwsBasicCredentials,
StaticCredentialsProvider}
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.{CreateBucketRequest,
HeadBucketRequest}
@@ -66,12 +68,17 @@ trait CometS3TestBase extends CometTestBase {
conf.set("spark.hadoop.fs.s3a.path.style.access", "true")
}
+ // Spark 4.2 has no published Iceberg spark-runtime yet; the build reuses
the 4.0 runtime, whose
+ // `SparkView` is binary-incompatible with Spark 4.2's
`connector.catalog.View` (now a class, not
+ // an interface), so treat Iceberg as unavailable on 4.2.
protected def icebergAvailable: Boolean = {
- try {
- Class.forName("org.apache.iceberg.catalog.Catalog")
- true
- } catch {
- case _: ClassNotFoundException => false
+ !isSpark42Plus && {
+ try {
+ Class.forName("org.apache.iceberg.catalog.Catalog")
+ true
+ } catch {
+ case _: ClassNotFoundException => false
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
