baibaichen opened a new pull request, #11800: URL: https://github.com/apache/gluten/pull/11800
### What changes were proposed in this pull request? `GlutenTestsCommonTrait` does **not** configure `GlutenPlugin` or create a Gluten-enabled `SparkSession`. Test suites using this trait were running on vanilla Spark, falsely reporting "offload to gluten". This PR replaces `GlutenTestsCommonTrait` with the correct traits in 157 test files (spark40 + spark41): | Category | Count | Target Trait | Effect | |----------|:-----:|-------------|--------| | SQL tests (QueryTest/SharedSparkSession) | 105 | `GlutenSQLTestsTrait` / `GlutenSQLTestsBaseTrait` | SparkSession with GlutenPlugin via `sparkConf` override | | Expression + Unit tests | 73 | `GlutenTestsTrait` (spark41: `shim.GlutenTestsTrait`) | SparkSession with GlutenPlugin via `initializeSession()` | | Hive tests (TestHiveSingleton) | 7 | Kept as `GlutenTestsCommonTrait` | TestHive singleton cannot inject GlutenPlugin | | BroadcastJoinSuite | 1 | Kept as `GlutenTestsCommonTrait` | Already self-creates GlutenPlugin SparkSession | 47 suites (spark41) / 42 suites (spark40) are disabled with `// TODO: 4.x` markers after failing under the correct traits. 25 of these are newly discovered failures; the rest were already tracked in #11550. > [#11512](https://github.com/apache/gluten/pull/11512) introduced PlanStability test suites for Spark 4.0 and 4.1 by extending Spark's original suites with `GlutenTestsCommonTrait`. While the tests appeared to pass, they were **not actually loading the Gluten plugin** — they were effectively running vanilla Spark, which trivially passes golden file comparison against Spark's own approved plans. ### How was this patch tested? - Compiled both spark40 and spark41 (`test-compile` with mvnd) - Ran all 140+ enabled suites on spark41 via `run-scala-test.sh --mvnd` — all passed - Ran all 119+ enabled suites on spark40 — all passed - Ran the 40 spark41-excluded suites on spark40 to identify version-specific differences (5 suites pass on spark40 but fail on spark41) ### New disabled suites discovered by this PR These suites previously appeared to pass because they were not loading GlutenPlugin: #### catalyst.expressions | Suite | Spark 4.0 | Spark 4.1 | |-------|-----------|-----------| | `GlutenCastWithAnsiOnSuite` | 🔴 (4 failures) | 🔴 (10 failures) | | `GlutenCollationRegexpExpressionsSuite` | 🟢 | 🔴 (1 failure) | | `GlutenExpressionEvalHelperSuite` | 🔴 (2 failures) | 🔴 (2 failures) | | `GlutenObjectExpressionsSuite` | 🔴 (7 failures) | 🔴 (7 failures) | | `GlutenOrderingSuite` | 🟢 | 🔴 (2 failures) | | `GlutenScalaUDFSuite` | 🔴 (1 failure) | 🔴 (1 failure) | | `GlutenToPrettyStringSuite` | 🔴 (1 failure) | 🔴 (1 failure) | #### execution | Suite | Spark 4.0 | Spark 4.1 | |-------|-----------|-----------| | `GlutenCoalesceShufflePartitionsSuite` | 🔴 (ABORTED) | 🔴 (ABORTED) | | `GlutenColumnarRulesSuite` | 🟢 | 🔴 (1 failure) | | `GlutenDataSourceScanExecRedactionSuite` | 🔴 (2 failures) | 🔴 (2 failures) | | `GlutenDataSourceV2ScanExecRedactionSuite` | 🔴 (2 failures) | 🔴 (2 failures) | | `GlutenExternalAppendOnlyUnsafeRowArraySuite` | 🔴 (14 failures) | 🔴 (14 failures) | | `GlutenHiveResultSuite` | 🟢 | 🔴 (1 failure) | | `GlutenLogicalPlanTagInSparkPlanSuite` | 🔴 (ABORTED) | 🔴 (ABORTED) | | `GlutenProjectedOrderingAndPartitioningSuite` | 🔴 (6 failures) | 🔴 (6 failures) | | `GlutenRemoveRedundantProjectsSuite` | 🔴 (14 failures) | 🔴 (14 failures) | | `GlutenSQLExecutionSuite` | 🔴 (1 failure) | 🔴 (1 failure) | | `GlutenSQLJsonProtocolSuite` | 🔴 (1 failure) | 🔴 (1 failure) | | `GlutenShufflePartitionsUtilSuite` | 🔴 (1 failure) | 🔴 (1 failure) | | `GlutenSimpleSQLViewSuite` | 🔴 (1 failure) | 🔴 (2 failures) | | `GlutenUnsafeRowSerializerSuite` | 🔴 (1 failure) | 🔴 (1 failure) | #### execution.datasources.parquet | Suite | Spark 4.0 | Spark 4.1 | |-------|-----------|-----------| | `GlutenParquetCommitterSuite` | 🔴 (spark.test.home) | 🔴 (spark.test.home) | #### org.apache.spark.sql | Suite | Spark 4.0 | Spark 4.1 | |-------|-----------|-----------| | `GlutenJoinHintSuite` | 🔴 (1 failure) | 🔴 (1 failure) | | `GlutenRandomDataGeneratorSuite` | 🟢 | 🔴 (232 failures) | #### streaming | Suite | Spark 4.0 | Spark 4.1 | |-------|-----------|-----------| | `GlutenTransformWithStateClusterSuite` | 🔴 (spark.test.home) | 🔴 (spark.test.home) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
