voonhous opened a new pull request, #19921: URL: https://github.com/apache/hudi/pull/19921
### Describe the issue this Pull Request addresses Part of #19524. The Java CI wall clock is now set by the Scala SQL shards (about 40 min). scalatest has no fork count, so the only in-job lever is running suites concurrently in one JVM, which needs one shared SparkContext. This is the first step: make that shared lifecycle exist behind a switch and gate it serially. Concurrency itself is a later PR. ### Summary and Changelog - New pom property `hudi.spark.test.sharedSession` (default `false`), forwarded to the scalatest JVM. With it on, every suite on `HoodieSparkSqlTestBase` shares one never-stopped SparkContext and works in its own `SparkSession.newSession()` child: own SQL conf, temp views and HoodieCatalog; shared external catalog and warehouse. - Per-suite conf deltas (`extraConf`, `sparkConf()` overrides) are applied to the child session; `spark.hadoop.*` keys go to the shared Hadoop conf and are restored in `afterAll`. Any other context-level key fails loudly. - The per-test cleanup drops only the suite's own tables, since the catalog is shared. - The child session is pinned as the active session on the suite thread, with a fail-fast check that the session's HoodieCatalog is bound to it. - Switched on for `test-spark-java17-scala-dml-tests` only; those packages hold nothing but `HoodieSparkSqlTestBase` suites. `scala-other-tests` mixes in suites that create their own SparkContext and stays on the default. - `TestHoodieDataUtils` no longer closes its JavaSparkContext in `afterAll` (it stopped the context). With the property off, nothing changes: one session per suite, stopped in `afterAll`. <details> <summary>Why a switch, and what newSession() does and does not isolate</summary> `HoodieSparkClientTestHarness` does `new SparkContext`, which throws while another context is live, so a never-stopped context cannot be the default while a JVM mixes both suite families (scala-other-tests, any full module run). The switch keeps the change to the shard being measured. `newSession()` isolates SQL conf, temp views, UDFs and the per-session catalog plugin. It cannot isolate SparkConf entries, static SQL confs (warehouse dir, extensions) or SharedState (external catalog, global temp views, cache manager, listener bus). Hudi reads `SparkSession.active` in `HoodieCatalog` (captured at construction), `BaseProcedure` (per CALL), `HoodieInternalV2Table` and `SparkCatalogMetaStoreClient`, all on the calling thread; inside tasks `getActiveSession` is `None` and `SQLConf.get` reads task-local properties, unchanged. The write client reads `jsc.hadoopConfiguration()`, hence the Hadoop conf handling for `spark.hadoop.*`. </details> <details> <summary>Local verification (Spark 3.5 / Scala 2.12 / JDK 11 unless noted)</summary> Maven-free scalatest runs against the reactor-compiled worktree, plus one Maven run for the pom wiring. | run | mode | result | |---|---|---| | 12 SQL-base suites of `org.apache.spark.sql.hudi.common` | shared | 49 tests, 0 failures, one base session (WARN line once) | | TestClusteringBinaryCopyStrategy, TestShowCleansProcedures, TestCopyToTempViewProcedure, TestRepairsProcedure, TestCallProcedure, TestHelpProcedure, TestSparkCatalogSync, TestDropTable, TestInsertTable5 (sparkConf override, extraConf, newSession, global temp views, FileSystem close, procedures, catalog listing) | shared | 54 tests, 0 failures | | whole `org.apache.spark.sql.hudi.common` package, including TestHoodieInternalRowUtils which builds its own SparkContext | default | 61 tests, 16 suites, 0 failures | | `mvn test` of TestHoodieDataUtils with `-Dhudi.spark.test.sharedSession=true`, Spark 4.2 / Scala 2.13 / JDK 17, fresh reactor | shared | green; the WARN line appears in the Maven log, so the pom forwards the property | | whole `org.apache.spark.sql.hudi.common` package with the flag on | shared | aborts at TestHoodieInternalRowUtils with "Only one SparkContext should be running in this JVM": the documented reason the flag is limited to SQL-only shards | The full dml packages run on CI in shared mode through the wired shard; that run is the phase gate. </details> ### Impact Test infrastructure only. No production code changes. ### Risk Level low. Default behaviour is unchanged; the shared mode runs on one CI shard and is verified locally on the packages listed above. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
