bitflicker64 commented on issue #3142:
URL: https://github.com/apache/hugegraph/issues/3142#issuecomment-5248859824
> My current test scenario is as follows: I have 5 Store nodes, and each
partition has 3 replicas. The service processes are running on the system
disks, while the data is stored on separate data disks. I simulated a situation
where the data disk of one Store node was suddenly disconnected, causing data
loss, while the system disk remained intact.
Thanks for explaining the setup. Could you clarify how the cluster is
deployed: Docker Compose, Kubernetes, VMs, or bare metal? Also, which HugeGraph
version or commit are you testing?
When you say the data disk was disconnected, was it temporarily unmounted
with the original data still available, or was the disk and its data
permanently lost?
> As a result, data queries may fail, but the node status still appears to
be normal. After I stopped the damaged node, its status changed from UP to
OFFLINE.
This suggests that the current Store liveness check only confirms that the
Store process is still sending heartbeats; it does not detect that the Store's
data path has become unavailable in this scenario.
Could you share the affected Store's logs after disconnecting the disk? It
would also be helpful to know whether all queries failed or only queries
involving partitions whose leader was on that Store.
> However, I understand that there is a maximum waiting period (48 hours by
default) before the system considers this node unavailable and triggers replica
migration.
In the current `master` code, `store.max-down-time` is declared as a
configuration option but is not used by any runtime logic. Therefore, the
documented 48-hour timeout does not currently trigger automatic replica
reallocation. A Store is marked `Offline` after its heartbeat expires, but no
timeout-based transition or migration is implemented afterward.
While reproducing the failure, please share the relevant PD configuration
and the output of `GET /v1/stores`, both before and after stopping the Store.
This will help us confirm the observed state changes.
> My question is: Is there any way to manually trigger replica migration? Or
is it possible to manually change the node status from OFFLINE to an
invalid/unavailable state, or any other state that can trigger replica
migration?
There is an API for changing a Store's state: `POST /v1/store/{storeId}`.
Setting the Store to `Tombstone` removes it from the active Store set, removes
its shards from their current shard groups, and starts shard reallocation.
However, `Tombstone` is a permanent decommissioning state for that Store ID,
not a temporary failure state. Once a Store is marked `Tombstone`, it cannot
register again using the same Store ID and must be reinitialized as a new Store.
The REST path also does not currently perform all the safety and capacity
checks present in the gRPC path. Because of that, I would not recommend using
the REST operation directly in a production cluster until we have verified and
hardened the complete recovery flow.
> My goal is to allow the replica migration to complete first, and then
restart the damaged node afterward.
>
> I could not find any available APIs related to this operation.
If the original data disk is recoverable, the expected recovery may simply
be to restore or remount the disk and restart the Store with its existing data
and identity.
If the disk and its data are permanently lost, the Store likely needs to be
decommissioned, its replicas rebuilt on healthy Stores, and the repaired
machine later initialized as a new Store. The deployment and disk-loss details
requested above will help us determine which case applies.
> Alternatively, could you please advise whether there is a standard
recovery procedure for handling a data disk failure scenario? For example, what
is the recommended process to recover the cluster after a Store node loses its
data disk?
This is a useful production-recovery question, but it is separate from the
Prometheus metrics bug tracked by this issue. Could you please open a new issue
containing the deployment details, HugeGraph version, PD configuration, Store
and PD logs, and exact reproduction steps?
We can then investigate and document the Store recovery procedure separately
without mixing it with the metrics fix. Thank you for testing this scenario and
reporting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
