LuciferYang opened a new pull request, #12840: URL: https://github.com/apache/gluten/pull/12840
## What changes are proposed in this pull request? Fixes #12837, the first checkbox of #12807 (`Migrate important tests to Spark-34+`). Three groups of test coverage exist only in `gluten-ut/spark33` today, so removing that module would delete them outright. All three are added to every surviving version module (3.4, 3.5, 4.0, 4.1). No existing case is removed, so this can land independently of the decision to drop 3.3. 1. `GlutenImplicitsTest`, six `fallbackSummary` cases, which had no equivalent on 3.4+. There is one copy per version module rather than a single copy in the version-agnostic `gluten-ut/test`, for two reasons. The assertions pin exact `numGlutenNodes` counts, which depend on the plan shape and may differ between 3.4 and 4.1. And `gluten-ut/test` cannot host a `GlutenSQLTestsBaseTrait` suite at all: that trait routes every case through `BackendTestSettings.instance`, which reflectively loads `VeloxTestSettings` / `ClickHouseTestSettings` from the per-version modules that `gluten-ut/test` does not depend on. Registered on both backends; the ClickHouse side carries the same three `excludeGlutenTest` entries the spark33 settings had. 2. The four DPP `DisableProject` suites. They landed in spark33 in `caf959e46` (2023) and the 34/35/40/41 modules created later never picked them up, even though all four carry the `DisableScan` counterparts. Velox only, since ClickHouse has never enabled `DisableProject`. 3. Seven Gluten-authored cases. Five trace to the commit that added the case to spark33 and skipped the newer modules (`399a91b12`, `de26ed2da`, `31adaf875`, `d2b50ac54`, `a662f3f75`). The other two are rewrite-style, where spark33 excludes Spark's own case and supplies a Gluten version in its place: `SPARK-35675` in `GlutenEnsureRequirementsSuite` and `default` in `GlutenLiteralExpressionSuite`. `SPARK-37652` belongs to that group too, so its exclude is added to spark34's ClickHouse settings, the only surviving version where that suite runs in exclude mode. Three things are deliberately not migrated. `SPARK-32932`. `VeloxAdaptiveQueryExecSuite` runs in include mode and `"SPARK-32932"` is already on the `includeByPrefix` list of all four surviving versions, while it is absent from spark33's, so Spark's own case has been running there all along and it runs through the Gluten path. The spark33 rewrite existed only to drop an assert on the child being an `AdaptiveSparkPlanExec`, and Spark 3.4 removed that assert itself while switching the match to `stripAQEPlan`, so the rewrite has no purpose on 3.4+. The three ANSI cast suites. They are enabled only on 3.3, but they never reach Velox in CI: `spark.gluten.sql.columnar.backend.velox.ansiFallback.enabled` defaults to true and the `spark-test-spark33` job does not turn it off, so `FallbackOnANSIMode` tags every node in the plan for fallback. `AnsiCast`, a 3.3-only class, is not in `ExpressionMappings` either. Nine `.sql` files that appear only in spark33's `VeloxSQLQueryTestSettings`. Each needs a UT run against Spark's own golden `.out` before it can be enabled, which does not fit a pure-addition change. Better handled as a separate issue. One gap in the inventory is worth recording here. This covers suite granularity, sql-file granularity, and Gluten-authored case granularity (8 out of 286 names). It does not cover vanilla `test(...)` cases inherited from Spark's own suites that are enabled on 3.3 but excluded in all four surviving modules' settings. spark33's settings carry hundreds of `.exclude` entries, and whether their complement is a subset of 3.4+ is still unanswered. That should be settled before the module is actually deleted. ## How was this patch tested? Compile only. `test-compile` passes on all four versions (`-Pbackends-velox` with `-Pspark-3.4`, with `-Pspark-3.5`, and with `-Pjava-17 -Pscala-2.13` plus `-Pspark-4.0` / `-Pspark-4.1`), and once more for `gluten-ut/spark35` under `-Pbackends-clickhouse -Pspark-3.5 -Pscala-2.13 -Pdelta`. `./dev/format-scala-code.sh check` is clean. No unit test was run locally, since this machine has neither a Velox native library nor `libch.so`, so every one of these cases depends on CI. Hard-coded expectations such as `numGlutenNodes` and the plan-shape assertions are copied verbatim from 3.3 and not pre-adjusted, because CI is a better source for the real values than guessing. If a case fails on one version, the intent is to adjust the expected value for that version, or to `excludeGlutenTest` it there with a reason, rather than withdraw the case from all four. One note for ClickHouse reviewers: `date_from_unix_date` feeds `-100000` and `200000`, which are outside ClickHouse's native Date32 range on purpose, and the case comment says that an implementation clamping to that range would diverge from Spark without failing. That case now runs on CH-3.5, so please take a look, and I will post `Run Gluten Clickhouse CI`. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude claude-opus-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
