byteayan opened a new pull request, #3379:
URL: https://github.com/apache/hugegraph/pull/3379
## Purpose of the PR
- close #3226
A restarted PD can start before the previous PD JVM has released the
metadata RocksDB `LOCK` (systemd restart, `stop` then `start`). Since #3368 a
failed metadata open fails startup instead of leaving a half-started PD, but it
fails immediately, so this restart race still needs an operator. #3368 left the
retry as a TODO pointing at #3226.
## Main Changes
- `HgKVStoreImpl.init()` retries the open for up to 30 seconds (the time
`stop-hugegraph-pd.sh` waits for the old JVM) while the failure is a lock
conflict, backing off from 500 ms to 4 s because every RocksDB open attempt
rotates the info `LOG`. After that, startup fails exactly as before
(`IllegalStateException` → `PDException` → `RocksDBException`).
- Lock conflicts are recognised by `IOError` plus RocksDB's lock text:
`While lock file` (another process) or `lock hold by current process` (this
JVM). Both strings are present in rocksdbjni 8.10.2. Other failures, including
a `LOCK` file that cannot be created, are not retried.
- Interruption restores the interrupt flag and fails startup with the lock
error.
- Snapshot reload keeps opening without a retry; it shares the same open
path with a zero wait.
- Docs: `hugegraph-pd/docs/configuration.md` (PD Core Settings) describes
the wait and warns never to delete `LOCK` files while the old process may be
alive; `README.md` links to it; `api-reference.md` no longer calls the retry
pending. The two TODOs pointing at #3226 are removed.
Known limitation: only the metadata store waits. PD opens it before the raft
log store today (`PDService.init`), and the shutdown order closes Raft first,
so the raft log `LOCK` is already free by then.
## Verifying these changes
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [ ] Already covered by existing tests, such as *(please modify tests
here)*.
- [x] Need tests and can be verified as follows:
- New `HgKVStoreImplTest` cases: a lock released after 1 s is waited
for; a lock never released fails after the (shortened) timeout with the RocksDB
cause; a non-lock failure is not retried.
- `mvn test -pl hugegraph-pd/hg-pd-test -am -P pd-core-test`: 193 tests,
0 failures, 2 skipped. `mvn editorconfig:check` on hg-pd-core, hg-pd-test,
hg-pd-service passes.
- Manual check with the built distribution (`bin/start-hugegraph-pd.sh
-d false`) while another process held `pd_data/rocksdb/LOCK` with `fcntl`:
- held for 8 s: PD logged `While lock file ...`, opened the store
once it was released, and `/v1/health` returned 200;
- held for 60 s: PD gave up after 30 s, startup failed with `Failed
to open PD metadata`, and the process exited with status 1.
## Does this PR potentially affect the following parts?
- [ ] Dependencies
- [ ] Modify configurations
- [ ] The public API
- [x] Other affects (typed here): PD startup can take up to 30 s longer
when the metadata `LOCK` is held
- [ ] Nope
## Documentation Status
- [ ] `Doc - TODO`: required documentation is pending; complete it before
merging.
- [x] `Doc - Done`: documentation is included here or linked below.
- [ ] `Doc - No Need`: no user-visible documentation is affected.
Documentation files in this PR or paired hugegraph-doc PR:
`hugegraph-pd/docs/configuration.md`, `hugegraph-pd/docs/api-reference.md`,
`hugegraph-pd/README.md`. No new configuration key; the website PD pages still
need to be checked for startup or `LOCK` guidance before merge.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
