SebastianGruza opened a new pull request, #3210: URL: https://github.com/apache/hugegraph/pull/3210
## Purpose of the PR - close #3203 With `usePD=true` the first hstore graph opened at startup (a local `conf/graphs` graph, or the system graph created on the first boot) needs `pd.initial-store-count` active stores, and the store client gives up after a fixed 10 retries (about 38 s). Stores that register later than that make the server exit 1 on a cold start; under Kubernetes the container is restarted and usually succeeds the second time, without an orchestrator the server stays down. ## Main Changes - New server option `pd.stores_wait_timeout` (`ServerOptions.PD_STORES_WAIT_TIMEOUT`, seconds, default 300, 0 = do not wait, i.e. today's behaviour). - `GraphManager`: with `usePD=true`, before the local graphs and the PD metadata are loaded, poll `PDClient.getActiveStores()` until the count reaches the required number (`min_store_count` from `getPDConfig()`, or `shard_count` when PD sends 0), log the progress every 5 s, and on timeout throw a `HugeException` that names the option. The loop is a static `waitForStores(IntSupplier, required, timeoutSeconds, pollSeconds)` so it can be tested without a PD. - No change to `NodeTxExecutor`: the 10-retry ceiling is generic and just went through #3204. ## Verifying these changes - [x] `unit/core/GraphManagerStoresWaitTest` (added to `UnitTestSuite`): enough stores at once; stores arriving over four polls with one failed query in between; timeout message naming the option. 3/3 on JDK 11. - [x] Reproduction before/after on 3 VMs (1 PD + 3 Store + 1 Server from a tarball, stores registering +5 s and +71 s after PD, `usePD=true`, first boot; same setup as in my comment on #3203, script `cluster/repro_coldstart.sh`, logs in `results/issue-3203/fix/` of https://github.com/SebastianGruza/hugegraph-validation): | server | result | |---|---| | master `1a15e762` | 18 × `error code = 105`, backoff 1,1,1,2,3,4,5,6,7,8, `upper limit : 10`, **exit 1 after 38 s** | | `1a15e762` + this PR | `PD needs 3 active store(s) (min_store_count=0, shard_count=3); waiting up to 300s`, 14 progress lines (`0/3 → 1/3 → 3/3`), `3 active store(s) in PD after 70s`, **0 × 105, exit 0 after 81 s**, REST 200 | | + this PR, `pd.stores_wait_timeout=20`, stores at +120 s | 5 progress lines, then `Timed out after 20s waiting for 3 active store(s) in PD (0 registered); start the stores first or raise pd.stores_wait_timeout`, exit 1 after 31 s | ## Notes - The wait sits in the `GraphManager` constructor, not in `createSysGraphIfNeed()`: with a local `conf/graphs` hstore graph that local graph is opened before `loadMetaFromPD()` and already trips the retry ceiling (a first version that waited only before the system graph did nothing on the VMs). - PD does not fill `min_store_count` in the `PDConfig` it serves (`ConfigService` builds it from `partition_count` and `shard_count` only), so the server falls back to `shard_count`. A one-line follow-up in `ConfigService` (plus the two PD tests that expect 0 today) would make the required count exact on clusters where `pd.initial-store-count` differs from the replica count. - The `[wait-storage]` gate in the image entrypoint from #3132 stays the right thing for containers; this change covers the tarball and any start without an orchestrator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
