bitflicker64 commented on PR #3185: URL: https://github.com/apache/hugegraph/pull/3185#issuecomment-5550492151
Logs and scripts from the two runs above, hosted on my fork (branch `logs/pr-3185`, [index](https://github.com/bitflicker64/hugegraph/tree/logs/pr-3185)): - [d3q-samples.log](https://raw.githubusercontent.com/bitflicker64/hugegraph/logs/pr-3185/d3q-samples.log) and [d3q-b-samples.log](https://raw.githubusercontent.com/bitflicker64/hugegraph/logs/pr-3185/d3q-b-samples.log): one line per 2 s on the surviving PD: epoch, `/v1/ready`, `/v1/health`, the `hg_raft_*` gauges, the Pod Ready condition. Run 2 appends `time_total` to the ready column as `code/seconds`. - [d3q.log](https://raw.githubusercontent.com/bitflicker64/hugegraph/logs/pr-3185/d3q.log) and [d3q-b.log](https://raw.githubusercontent.com/bitflicker64/hugegraph/logs/pr-3185/d3q-b.log): script output (pre-fault uids and IPs, T0, the Store gate, post-recovery checks, blocked-line counts). - [d3q.sh](https://raw.githubusercontent.com/bitflicker64/hugegraph/logs/pr-3185/d3q.sh) and [d3q-b.sh](https://raw.githubusercontent.com/bitflicker64/hugegraph/logs/pr-3185/d3q-b.sh): the scenario scripts, oracle declared before the fault. The stall, from run 2 (`ready=<code>/<seconds>`): ``` 1788593443.186162738 ready=200/0.003056 health=200 hg_raft_has_leader=1.0 podReady=True 1788593445.237568308 ready=200/0.002668 health=200 hg_raft_has_leader=1.0 podReady=True 1788593447.290482850 ready=200/0.003203 health=200 hg_raft_has_leader=1.0 podReady=True 1788593449.335970063 ready=503/9.792368 health=200 hg_raft_has_leader=1.0 podReady=True 1788593461.199044573 ready=200/0.002740 health=200 hg_raft_has_leader=1.0 podReady=True 1788593463.250381323 ready=200/0.002739 health=200 hg_raft_has_leader=1.0 podReady=True ``` T0 was 1788593447, the third line's timestamp. The fourth line's request went out at about T0+2.3 s and got its 503 at about T0+12.1 s, 9.79 s later; the gauge on that same line was read after the stall, once the replacements were back, which is why it already shows `has_leader 1.0`. The fifth line answered 200 in 2.7 ms. Run 1 shows the same window as `ready=000` (2 s timeouts) from T0+4.6 s to T0+22.8 s after one prompt 503 at T0+2.6 s with `has_leader 0.0`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
