SebastianGruza opened a new issue, #3226: URL: https://github.com/apache/hugegraph/issues/3226
### Bug Type (问题类型) others (availability) ### Before submit - [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents) ### Environment (环境信息) PD master `83ef9f3f` (the code path is unchanged since 1.7.0: `HgKVStoreImpl.init`), single PD, Java 17. Second occurrence: a 3-node PD cluster restarted by systemd during a load; the new JVMs started before the old ones released the RocksDB lock, all three came up half-initialised, stores lost the PD leader, servers timed out, and the cluster stood still until an operator deleted the `LOCK` files by hand. ### Expected & Actual behavior (期望与实际表现) **Expected:** a PD that cannot open its KV store either retries the open (the lock is transient during a restart race) or exits non-zero so the supervisor restarts it. **Actual** (lab, [`results/pd-halfalive-lock`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/pd-halfalive-lock)): ``` 15:19:10 ERROR HgKVStoreImpl - Failed to open RocksDB from ./pd_data/rocksdb/ org.rocksdb.RocksDBException: While lock file: ./pd_data/rocksdb//LOCK: Resource temporarily unavailable 15:19:10 ERROR HgKVStoreImpl - Failed to open data file,{} 15:19:13 INFO HugePDServer - Started HugePDServer in 5.856 seconds lock released at 15:19:50; three minutes later: GET /v1/health -> 200 GET /v1/ready -> 503 {"ready":false,"state":"STATE_UNINITIALIZED","isLeader":false} GET /v1/members -> 401 {"status":-1,"error":"Unauthorized"} (auth cannot read anything either) gRPC and raft ports listening; stores: Failed to get the PD leader "Failed to open RocksDB" logged exactly once: no retry, no exit ``` ### Analysis - `HgKVStoreImpl.init()` catches the `PDException` from `openRocksDB()` and only logs it (`HgKVStoreImpl.java:66-80`); `this.db` stays null. Spring Boot finishes startup, Tomcat, gRPC and the raft node come up, and every KV access later fails, surfacing as "Unauthorized" on REST and "leader not ready" for the stores. - The lock error is the normal outcome of a restart race: RocksDB's `LOCK` is an fcntl lock held until the previous JVM exits, and a supervisor that restarts on the shell wrapper (or a manual `start` right after `stop`) lands in that window. jraft's own log storage in `pd_raft/log` has the same lock; when that one is held as well the process eventually exits, so the half-alive state needs the raft side to open while the KV side fails. - With `/v1/health` hard-coded to 200 (see #3222) nothing external restarts such a PD; a 3-node cluster where all three hit the window stays down until an operator intervenes, and deleting `LOCK` files is the wrong fix when the old process is still alive. ### How to reproduce (lab, no systemd needed, [`repro_pd_lock.sh`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/results/pd-halfalive-lock/repro_pd_lock.sh)) 1. Copy a PD data directory; from another process hold an fcntl lock on `pd_data/rocksdb/LOCK` (Python `fcntl.lockf`) for 40 s. 2. Start PD on that directory while the lock is held. 3. Release the lock. PD never recovers: `/v1/ready` 503 `STATE_UNINITIALIZED`, REST 401, health 200, process alive. ### Proposal 1. `HgKVStoreImpl.init()` retries the open for a bounded time when the failure is a lock conflict (the previous instance is shutting down), then rethrows; the caller lets Spring startup fail, so the process exits non-zero and the supervisor restarts it cleanly. 2. Until the store is open, `/v1/health` answers 503 (the same direction as #3222: health should reflect a terminal error state, ready reflects leadership, #3185). 3. A note in the operations guide that the `LOCK` files must never be deleted while a previous JVM is alive. Related: #3222 (single-node PD stays leaderless after a snapshot error, health still 200), #3185, #3189. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
