voonhous opened a new pull request, #19927: URL: https://github.com/apache/hudi/pull/19927
### Describe the issue this Pull Request addresses Part of #19524, stacked on the shared-session PR (successor of #19921). Supersedes the proof of concept #19923. The shared-session PR lets the Scala SQL suites share one SparkContext; this PR removes the state that would break once those suites run concurrently in a shard. Still serial: no scheduling change, no timing claim. ### Summary and Changelog - Literal table names (`blob_*`, `vec_*`, two more) and literal database names (`hudi_database`, `test_database`, `testdb`) in the SQL suites now come from `generateTableName`, so no two suites can name the same catalog object. Assertions that embedded a literal build from the same value. - `HoodieInMemoryHashIndex` (test-only, "ONLY USE FOR LOCAL TESTING") keeps one map per table instead of one for the JVM, and can clear a single table. The test base clears globally only in per-suite mode; in shared mode it clears a table's entry when the table is dropped. - New `ExclusiveSuite` mixin: a fair read/write lock where every suite holds the read side for its whole run and an exclusive suite holds the write side. Applied to the five suites that mutate JVM-wide state (shared Hadoop conf, persisted RDDs, metrics registry, Hudi's global properties). No effect until suites run in parallel. - `TestInsertTable4` scopes its stage listener to the jobs the test started (a local property, since Hudi overwrites the job group on its own jobs) and removes it in a `finally`. - `TestRepairsProcedure` no longer closes the JVM-cached `FileSystem`. <details> <summary>Why per-table keying in the index rather than serializing the INMEMORY suites</summary> Ten SQL suites across the shards use the INMEMORY index, and the base cleared the JVM-wide map after every test. Under concurrency one suite's clear, or a same-key write, misroutes another suite's upserts into wrong results, not a crash. Keying the map by base path removes the interaction at the source. `tagLocation`'s commit-time check is `containsOrBeforeTimelineStarts`, so it does not reject an entry left by an earlier table at the same path; the base clears a table's entry when it drops it, and a test that re-creates a table at a path it used before must clear that path itself. </details> <details> <summary>Evidence from the proof of concept (#19923)</summary> Green on both Scala shards with the same test counts as master: dml in shared mode 41:22 ([run 34686924099](https://github.com/apache/hudi/actions/runs/34686924099)) and 53:17 ([run 34693015636](https://github.com/apache/hudi/actions/runs/34693015636)), other-tests in default mode. One review round folded in: the read side of the suite lock covers a suite's whole run, and TestSqlConf became exclusive after a two-thread smoke showed its global properties turning a neighbour's query incremental. </details> ### Impact Test infrastructure plus one test-only index class. No user-facing change. ### Risk Level low. With the property off, the only behaviour change is the per-table keying of a test-only index, which the existing INMEMORY suites exercise on every CI run. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
