SebastianGruza commented on issue #3203: URL: https://github.com/apache/hugegraph/issues/3203#issuecomment-5646766502
### Reproduced outside Kubernetes, same outcome I reproduced this on 3 VMs (1 PD + 3 Store + 1 Server, tarball built from master `60c8803`, no Docker) by starting the Server right after PD and delaying the Store registration by a fixed number of seconds. The results are deterministic: | Server config | stores register (after PD) | result | |---|---|---| | `usePD=true`, first boot of the cluster | 1 store at +5 s, the other two at +71 s | 10 attempts in 38 s (sleeps 1,1,1,2,3,4,5,6,7,8 s), `upper limit : 10`, **exit 1** | | `usePD=true`, first boot of the cluster | 1 store at +5 s, the other two at +26 s | 9 attempts, the 9th succeeds, Server starts | | local `conf/graphs`, no `usePD` | all stores at +20 / +45 / +60 s | Server up after 8 s, no error 105 at all; the missing stores only show up as code 102 in the `task-db-worker` thread | With zero registered stores PD answers 102 (*There is no any online store*), with one or two it answers 105; the client treats both the same way. Script and logs: [`cluster/repro_coldstart.sh`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/cluster/repro_coldstart.sh), [`results/issue-3203/`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/issue-3203). ### What the partition lookup on the main thread actually is The path that ends in `NodeTxSessionProxy.createTable` is the creation of the **system graph** `DEFAULT-~sys_graph`: `GraphManager.loadMetaFromPD → kvStoreInit → createSysGraphIfNeed`. That method calls `createGraph(..., init = true)` only when PD holds no system-graph config yet, and `initBackend()` on hstore is `HstoreStore.init → createTable`, i.e. the first partition lookup. This explains both observations at once: it can only happen on the very first boot of a cluster, and never on pod replacement, because by then the config is in PD and `init` is false. The trigger is `usePD=true` in `rest-server.properties`, which the entrypoint sets from `HG_SERVER_USE_PD`. Two small corrections to the "Related" section: `init-store` is not involved here, it skips hstore graphs since `b9a3dd9`; and on the PD side code 105 comes from `StoreNodeService.allocShards` and is thrown only while the shard group of a partition does not exist yet, so also only on a cold cluster. ### Stdout Confirmed: without `STDOUT_MODE=true`, `hugegraph-server.sh` redirects the JVM's stdout and stderr to `logs/hugegraph-server-stdout.log`, and the image's entrypoint does not enable that mode. All that reaches `docker logs` is the line from `start-hugegraph.sh`. ### Direction I would consider for a fix - Since this is a one-off bootstrap path, a gate "wait until PD reports at least `minStoreCount` active stores" right before `createSysGraphIfNeed` (or in `HstoreSessionsImpl.open`), with a configurable timeout and a log line every few seconds. `PDClient.getActiveStores()` and `getPDConfig()` already expose what is needed. - I would not change `NODE_MAX_RETRYING_TIMES` or the general retry loop for this: the same counter handles failures during normal operation, and #3204 touches that loop right now; two PRs changing its semantics at the same time would be hard to review. - `STDOUT_MODE=true` in the image's entrypoint as a separate one-line change. If you want, I can try to implement such a gate with a test and measure it with the same script; if you would rather do it yourself, I am happy to test your patch against the same reproduction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
