SebastianGruza opened a new issue, #3226:
URL: https://github.com/apache/hugegraph/issues/3226

   ### Bug Type (问题类型)
   
   others (availability)
   
   ### Before submit
   
   - [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 
[FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have 
confirmed and searched that there are no similar problems in the historical 
issue and documents)
   
   ### Environment (环境信息)
   
   PD master `83ef9f3f` (the code path is unchanged since 1.7.0: 
`HgKVStoreImpl.init`), single PD, Java 17. Second occurrence: a 3-node PD 
cluster restarted by systemd during a load; the new JVMs started before the old 
ones released the RocksDB lock, all three came up half-initialised, stores lost 
the PD leader, servers timed out, and the cluster stood still until an operator 
deleted the `LOCK` files by hand.
   
   ### Expected & Actual behavior (期望与实际表现)
   
   **Expected:** a PD that cannot open its KV store either retries the open 
(the lock is transient during a restart race) or exits non-zero so the 
supervisor restarts it.
   
   **Actual** (lab, 
[`results/pd-halfalive-lock`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/pd-halfalive-lock)):
   
   ```
   15:19:10 ERROR HgKVStoreImpl - Failed to open RocksDB from ./pd_data/rocksdb/
            org.rocksdb.RocksDBException: While lock file: 
./pd_data/rocksdb//LOCK: Resource temporarily unavailable
   15:19:10 ERROR HgKVStoreImpl - Failed to open data file,{}
   15:19:13 INFO  HugePDServer - Started HugePDServer in 5.856 seconds
   lock released at 15:19:50; three minutes later:
     GET /v1/health   -> 200
     GET /v1/ready    -> 503 
{"ready":false,"state":"STATE_UNINITIALIZED","isLeader":false}
     GET /v1/members  -> 401 {"status":-1,"error":"Unauthorized"}   (auth 
cannot read anything either)
     gRPC and raft ports listening; stores: Failed to get the PD leader
     "Failed to open RocksDB" logged exactly once: no retry, no exit
   ```
   
   ### Analysis
   
   - `HgKVStoreImpl.init()` catches the `PDException` from `openRocksDB()` and 
only logs it (`HgKVStoreImpl.java:66-80`); `this.db` stays null. Spring Boot 
finishes startup, Tomcat, gRPC and the raft node come up, and every KV access 
later fails, surfacing as "Unauthorized" on REST and "leader not ready" for the 
stores.
   - The lock error is the normal outcome of a restart race: RocksDB's `LOCK` 
is an fcntl lock held until the previous JVM exits, and a supervisor that 
restarts on the shell wrapper (or a manual `start` right after `stop`) lands in 
that window. jraft's own log storage in `pd_raft/log` has the same lock; when 
that one is held as well the process eventually exits, so the half-alive state 
needs the raft side to open while the KV side fails.
   - With `/v1/health` hard-coded to 200 (see #3222) nothing external restarts 
such a PD; a 3-node cluster where all three hit the window stays down until an 
operator intervenes, and deleting `LOCK` files is the wrong fix when the old 
process is still alive.
   
   ### How to reproduce (lab, no systemd needed, 
[`repro_pd_lock.sh`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/results/pd-halfalive-lock/repro_pd_lock.sh))
   
   1. Copy a PD data directory; from another process hold an fcntl lock on 
`pd_data/rocksdb/LOCK` (Python `fcntl.lockf`) for 40 s.
   2. Start PD on that directory while the lock is held.
   3. Release the lock. PD never recovers: `/v1/ready` 503 
`STATE_UNINITIALIZED`, REST 401, health 200, process alive.
   
   ### Proposal
   
   1. `HgKVStoreImpl.init()` retries the open for a bounded time when the 
failure is a lock conflict (the previous instance is shutting down), then 
rethrows; the caller lets Spring startup fail, so the process exits non-zero 
and the supervisor restarts it cleanly.
   2. Until the store is open, `/v1/health` answers 503 (the same direction as 
#3222: health should reflect a terminal error state, ready reflects leadership, 
#3185).
   3. A note in the operations guide that the `LOCK` files must never be 
deleted while a previous JVM is alive.
   
   Related: #3222 (single-node PD stays leaderless after a snapshot error, 
health still 200), #3185, #3189.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to