cchung100m commented on PR #11505: URL: https://github.com/apache/ozone/pull/11505#issuecomment-6311690367
Thanks to @chungen0126, I agree that the NPE is a symptom. I traced the call chains to see what this PR does and doesn't cover. ### What the PR covers The Jira stack trace matches he bootstrap state machine registered by `SCMRatisServerImpl.initialize()` (at that revision only called from `scmInit`). Ratis recovers every UUID-named directory in the storage dir, not only the configured group, so a leftover group's entries reach that state machine. The patch only skips them there. ### What it doesn't cover On normal start, a leftover group gets a full `SCMStateMachine`, but invokers are registered only for the expected group. In a unit test, applying an entry on a full state machine with no invoker terminates SCM (`invoker == null`). I checked this at the state-machine level only and have not reproduced a full SCM restart. Either way, you are right that the real problem is the misconfiguration. ### OM reference `OzoneManager#initializeRatisDirs` (HDDS-5546, relaxed in HDDS-7088, restored in HDDS-7553) fails fast: a directory in the Ratis storage dir that is not the expected group ID throws an `IOException` before the Ratis server is created. ### Proposal I suggest following the OM approach: before building the Ratis server, in both the SCM constructor and `initialize()`, throw an `IOException` if a UUID-named subdirectory differs from the cluster ID's group ID. The message would name the directory and point to `ozone.scm.ha.ratis.storage.dir`. The existing skip would stay as a safety net. If you prefer something smaller, failing only in `scm --init` is an alternative, but a normal start could still hit the problem. ### Questions - `scm --init` writes the VERSION file before initializing Ratis, so after a failed check, a rerun takes the "already initialized" branch and calls `initializeRatis` again with the same cluster ID. Is it fine to tell users to remove the stale directory and retry? - Do you know of any case where an SCM Ratis directory legitimately holds more than one group (e.g., after upgrading from the old shared `<meta>/ratis`)? Since HDDS-7553 OM refuses to share it, I expect a strict check to be consistent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
