andygrove opened a new pull request, #6666: URL: https://github.com/apache/datafusion-comet/pull/6666
## Which issue does this PR close? No issue. Related to #4949: this removes one of the two things that keep TPC-DS q77a off Comet on Spark 4.2. The other is #5834, which #5889 fixes. ## Rationale for this change Spark plans a query without a `FROM` clause over a `OneRowRelation`: an `RDDScanExec` named `OneRowRelation` before Spark 4.1, and a `OneRowRelationExec` from 4.1. Comet only runs it through the Spark-to-Arrow conversion, which was off by default, so the leaf and every operator above it that needs a Comet child stayed in Spark. That matters more on Spark 4.2, which puts a `OneRowRelation` inside larger plans. In q77a, a single-row aggregate becomes a struct-typed scalar subquery that a projection over a `OneRowRelation` reads, in each `UNION` branch. The row has no columns, so converting it costs almost nothing and has no type-compatibility risk. #6602 gave it a config of its own, so it can be on by default without the range, cache and RDD conversions. Comet's own suites already run with it on, because `CometTestBase` turns it on. When the operator above can't run in Comet, nothing is lost: `EliminateRedundantTransitions` turns `ColumnarToRow(CometSparkToColumnar(leaf))` back into the bare leaf, so no transition is added. ## What changes are included in this PR? - `spark.comet.convert.oneRowRelation.enabled` now defaults to `true`, and its doc says why. - A `CometExecRuleSuite` test checks that, with every `spark.comet.convert` config at its default, only the `OneRowRelation` query converts. - The Spark 4.2 q77a golden: each projection over a `OneRowRelation` now records the remaining blocker, `Unsupported data type: StructType(returns, profit_loss)` (#5834), in place of `Scan OneRowRelation is not supported`. The summary line is unchanged at 69 of 89 operators and 5 transitions, so q77a stays where it is until #5889 lands. - Docs: `datasources.md`, the migration guide's 1.2.0 paragraph on the deprecated settings, and the Spark 4.2 known limitation, which now names #5834. ## How are these changes tested? - The new `CometExecRuleSuite` test. It passes on Spark 3.5 and 4.1, and fails when the default is put back to `false`. - Spark 4.1: `CometExecRuleSuite` and both TPC-DS plan-stability suites, 224 tests passed with no golden changes. - Spark 3.5, the `RDDScanExec` path: the same suites, 223 passed and 1 canceled (a Spark 4.0+ test). - Spark 4.2: regenerating both TPC-DS plan-stability suites changed only q77a. Not run locally: the 3.4 and 4.0 profiles, and the Spark SQL and Iceberg suites. Those suites run on Comet's defaults, so every query without a `FROM` clause in them now goes through Comet, and they need the `run-*` labels for a verdict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
