Indian-boult opened a new pull request, #20739:
URL: https://github.com/apache/hudi/pull/20739
### Describe the issue this Pull Request addresses
Closes #19893
`Registry.getRegistryOfClass` (hudi-io) logs an error and returns the
existing registry when the registry under the key has a different class.
`SparkHoodieBackedTableMetadataWriter#initRegistry` and its `TableVersionSix`
copy cast `Registry.getRegistry("HoodieMetadata", DistributedRegistry)` to
`DistributedRegistry`. The key `::HoodieMetadata` is JVM-wide, so once a writer
with metrics on and executor metrics off has created it as a `LocalRegistry`,
every later metadata writer with executor metrics on (and a reporter other than
INMEMORY) fails in its constructor:
```
java.lang.ClassCastException: class
org.apache.hudi.common.metrics.LocalRegistry cannot be cast to class
org.apache.hudi.metrics.DistributedRegistry
at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.initRegistry(SparkHoodieBackedTableMetadataWriter.java:122)
```
Reproduced on master with the order from the issue
(`-Dsurefire.runOrder=reversealphabetical
-Dtest='TestMetricsReporter,TestMetadataTableWithSparkSQL'`, one fork):
TestMetricsReporter passes, then both TestMetadataTableWithSparkSQL tests fail
with this exception.
The registry those lines create is never read: HUDI-7391 (#10635) moved
`HoodieMetadataMetrics` to `Metrics`, and the local `registry` variable has
been unused since.
### Summary and Changelog
The Spark metadata writers no longer create a registry they do not use, and
a registry class mismatch fails with a message naming the key and both classes
instead of returning a registry of the wrong class.
- `SparkHoodieBackedTableMetadataWriter`,
`SparkHoodieBackedTableMetadataWriterTableVersionSix`: `initRegistry` keeps
only the `HoodieMetadataMetrics` setup, as in
`JavaHoodieBackedTableMetadataWriter` and
`FlinkHoodieBackedTableMetadataWriter`. Four unused imports are removed.
- `Registry#getRegistryOfClass`: throws `HoodieException("Registry
<table>::<name> already exists with class <existing>, but class <requested> was
requested")` instead of logging and returning the existing registry.
- Callers of `getRegistryOfClass`, checked one by one:
- `HoodieEngineContext#getMetricRegistry` (LocalRegistry): no change.
- `HoodieSparkEngineContext#getMetricRegistry` casts the result to
`DistributedRegistry`. Over a table registry created earlier as a
`LocalRegistry` it failed with `ClassCastException`; it now fails with the
`HoodieException` above.
- `DistributedRegistryUtil#createWrapperFileSystemRegistries` with
executor metrics off asked for the class of the process-wide
`HoodieWrapperFileSystem` registry (always `LocalRegistry`), and got back the
table's `DistributedRegistry` when a writer of the same table with executor
metrics on had created one in this JVM. The strict check would fail that write
client, so this path now reuses the table's registry of any class and creates a
`LocalRegistry` otherwise. It no longer creates the process-wide
`HoodieWrapperFileSystem` and `HoodieWrapperFileSystemMetaFolder` registries,
which nothing reads.
- `Registry#getRegistry(name)` and `getRegistry(name, clazz)`: the
remaining callers (timeline service, tests) use one class per name.
- Tests:
- `TestRegistry#testGetRegistryOfClassRejectsAnotherClass`: the same class
returns the same instance; another class throws with the exact message and
leaves the registry in place. On master nothing is thrown.
-
`TestDistributedRegistry#testEngineContextRejectsRegistryOfAnotherClass`:
`HoodieSparkEngineContext#getMetricRegistry` over an existing `LocalRegistry`
throws the `HoodieException`. On master it throws `ClassCastException`.
-
`TestDistributedRegistry#testWrapperFileSystemRegistriesReuseTheTableRegistryOfAnyClass`:
after a config with executor metrics on, a config with them off reuses both
`DistributedRegistry` instances of the table. It passes on master; with the
strict check and the old `DistributedRegistryUtil` it fails with "Registry
testWrapperFileSystemRegistries::HoodieWrapperFileSystem already exists with
class org.apache.hudi.metrics.DistributedRegistry, but class
org.apache.hudi.common.metrics.LocalRegistry was requested".
### Impact
A JVM that runs Spark metadata writers both with and without executor
metrics no longer fails with `ClassCastException`, whatever order its test
classes or writers run in. `Registry#getRegistryOfClass` now throws on a class
mismatch. In Hudi that is reachable only through
`HoodieSparkEngineContext#getMetricRegistry`, when a writer with executor
metrics on follows a writer of the same table with them off in one JVM; that
case already failed, with a `ClassCastException`. No metric that a reporter
publishes changes: the removed registries were never written to.
### Risk Level
low. The removed code had no reader. The strict check changes one failure
from `ClassCastException` to `HoodieException`, and the only lenient caller in
Hudi keeps its behavior.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]