damccorm opened a new pull request, #41069: URL: https://github.com/apache/beam/pull/41069
## Summary Fixes a race condition in `IcebergIO Integration Tests` (`HiveCatalogIT > testWriteToDynamicNamespaces`, [failed run 42337942112](https://github.com/apache/beam/actions/runs/42337942112)): 1. **`RecordWriterManager.loadOrCreateTable`**: `TableCache.getAndRefreshIfStale` synchronizes per `(catalogConfig, TableIdentifier)`. When two threads in the same worker JVM concurrently write to two different tables (`_2.table_false` and `_2.table_true`) in the same dynamic namespace (`_2`), both can see `!supportsNamespaces.namespaceExists(namespace)` and call `supportsNamespaces.createNamespace(namespace)`. In Hive Metastore (`HMSHandler.create_database_core`), when the second thread fails to commit the database creation in Derby due to `UNIQUE_DATABASE` constraint violation after calling `wh.mkdirs(dbPath)`, its `finally` block executes `wh.deleteDir(dbPath, true)`, recursively deleting the namespace directory in GCS and wiping out the metadata file (`00000-...metadata.json`) just written by the first thread's table creation. Synchronizing the namespace check and creation on `NAMESPACE_LOCK` prevents concurrent threads in the same JVM from racing on `createNamespace`. 2. **`IcebergCatalogBaseIT.setUp`**: `OPTIONS.as(DirectOptions.class).setTargetParallelism(1)` was only mutating the static `OPTIONS` instance rather than `pipeline.getOptions()`, leaving `TestPipeline` running with default parallelism on `DirectRunner`. Also apply `setTargetParallelism(1)` to `pipeline.getOptions()`. ## Verification - Local unit tests and Checkstyle: `./gradlew :sdks:java:io:iceberg:spotlessApply :sdks:java:io:iceberg:test :sdks:java:io:iceberg:checkstyleMain :sdks:java:io:iceberg:checkstyleTest` - Triggered `IcebergIO Integration Tests` on this branch: https://github.com/apache/beam/actions/runs/42346576013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
