voonhous opened a new pull request, #19926: URL: https://github.com/apache/hudi/pull/19926
### Describe the issue this Pull Request addresses Part of #19524, stacked on #19919 (the CI stack's commits are included until it merges). Supersedes the proof of concept #19921, which branched from master. The Java CI wall clock is set by the Scala SQL shards; scalatest has no fork count, so the lever is running suites concurrently in one JVM, which needs one shared SparkContext. This PR makes that shared lifecycle exist behind a property and runs it serially; #19923's successor removes the cross-suite state and the third PR turns the concurrency on. ### Summary and Changelog - New pom property `hudi.spark.test.sharedSession` (default `false`), forwarded to the scalatest JVM. With it on, every suite on `HoodieSparkSqlTestBase` shares one never-stopped SparkContext and works in its own `SparkSession.newSession()` child: own SQL conf, temp views and HoodieCatalog; shared external catalog and warehouse. - A suite's conf deltas (`extraConf`, `sparkConf()` overrides) are validated first, then applied to the child session; `spark.hadoop.*` keys go to the shared Hadoop conf and are restored in `afterAll`; any other context-level or static key is rejected before anything mutates. - The child session is pinned as the active session on the suite thread, with a fail-fast check that the session's HoodieCatalog is bound to it. The per-test cleanup drops only the suite's own tables. - Switched on for the three Java 17 Scala shards whose packages hold only `HoodieSparkSqlTestBase` suites: dml-1, dml-2, other-1. other-2 mixes in suites that create their own SparkContext and stays on the default. - `TestHoodieDataUtils` no longer closes its JavaSparkContext in `afterAll` (it stopped the context). - `ParquetUtils.readAvroRecords` reads on a private copy of the Hadoop configuration in both overloads; the schema-less one also drops any read schema or projection the copy carried. The schema overload used to set the read schema on the caller's configuration, usually the SparkContext-wide one, so a later schema-less read in the same JVM inherited it. One production file; callers see no change. With the property off, nothing changes. <details> <summary>Evidence from the proof of concept (#19921)</summary> On master's layout the dml job ran in shared mode, serially, at master's own time: 51:56 and 53:08 ([run 34676800123](https://github.com/apache/hudi/actions/runs/34676800123), attempts 1 and 2) against 52 to 56 minutes for the same job on seven recent master runs (41:22 on one fast runner), with the same 326 tests, 4 canceled and 1 ignored. Two review rounds on #19921 are folded in: the conf deltas are validated before any mutation, static SQL confs are rejected, and generateTableName builds on the prefix the cleanup relies on. </details> <details> <summary>Why the reader fix: other-1's first run in shared mode</summary> other-1 had never run in shared mode before this stack (the proof of concept flagged only the dml shard). Its first serial shared run failed the four `repair_deduplicate` tests with `Parquet/Avro schema mismatch: Avro field '_hoodie_commit_time' not found` ([job 103593400424](https://github.com/apache/hudi/actions/runs/34708708909/job/103593400424)). The hdfs parquet import procedure sets an Avro read schema on `jsc.hadoopConfiguration` (a no-op for its own Job, which had already copied the configuration) and the dedupe read built its reader on that same shared object, so it read Hudi base files with a schema that has no meta fields. Reproduced locally by running the import suite and then the repairs suite in one shared JVM (four failures in 17 s); the reverse order passes, and with the fix the same order passes ten of ten. </details> ### Impact Test infrastructure plus one reader fix in hudi-hadoop-common: `readAvroRecords` no longer mutates the caller's configuration. ### Risk Level low. Default behaviour is unchanged; the shared mode runs on three CI shards and was verified on #19921. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
