cui2022 opened a new issue, #3150:
URL: https://github.com/apache/hugegraph/issues/3150

   ### Problem Type (问题类型)
   
   None
   
   ### Before submit
   
   - [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 
[FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have 
confirmed and searched that there are no similar problems in the historical 
issue and documents)
   
   ### Environment (环境信息)
   
   - Server Version: 1.7.0
   - Backend: Hstore
   - OS: CentOS 
   
   ### Your Question (问题描述)
   
   I pulled the latest source code and built a deployment package. Then I 
deployed a cluster on my servers, including **5 Store nodes, 3 PD nodes, and 3 
Server nodes**, with the replica number configured as **3**. The deployment was 
done directly by extracting the packages, without using Docker.
   
   I am currently simulating possible failures that may occur in a production 
environment and trying to determine a reasonable recovery procedure. The 
failure scenario I have simulated so far is as follows:
   
   I disconnected the data disk of the second Store node while keeping its 
system disk running normally (the Store process is running on the system disk). 
I found that the Store node was still shown as **UP**, but queries involving 
data stored on the second Store node would fail.
   
   This behavior is acceptable from my perspective because it indicates that 
the heartbeat mechanism only checks whether the Store process is alive and does 
not verify the integrity or availability of the underlying data. After that, I 
stopped the second Store process manually. The status of the second Store node 
changed from **UP → OFFLINE**, and cluster queries returned to normal.
   
   I then checked the replica status. The shards that were originally 
**leaders** on the second Store node were switched to **followers**, and the 
corresponding shard status became **OFFLINE**. At this point, I believe the 
behavior is expected and there are no issues.
   
   However, the next scenario I would like to simulate is a case where the data 
on the second Store node is permanently lost and cannot be recovered. In this 
situation, I expect to remove the offline second Store node from the current 
metadata and then restart this node as a new Store node (equivalent to 
replacing the failed node with a fresh node).
   
   However, I am not sure what the correct procedure is to achieve this.
   
   I have tried several approaches, but I was unable to make the replicas 
migrate from the failed second Store node to other available Store nodes.
   
   For example, I tried calling the previously mentioned API:
   
   ```
   ```
   
   ```
   /v1/store/{storeId}
   ```
   
   and setting the second Store node status to **Tombstone**.
   
   After changing the status to Tombstone, I can confirm through:
   
   ```
   ```
   
   ```
   /v1/stores
   ```
   
   that the Store status has changed to **Tombstone**.
   
   However, when checking:
   
   ```
   ```
   
   ```
   /v1/partitions
   ```
   
   I found that the shards are still assigned to the Tombstone Store node, and 
no shard migration or replica recovery has occurred.
   
   Therefore, I would like to know:
   
   1.  What is the recommended recovery workflow when a Store node's data is 
permanently lost? 
   2.  How should an offline/tombstone Store node be removed from PD metadata? 
   3.  How can the replicas on the failed Store node be automatically migrated 
to healthy Store nodes? 
   4.  If replacing the failed Store node with a new empty node, what is the 
correct procedure to trigger replica rebuilding? 
   
   Thank you very much for your help.
   
   
   ### Vertex/Edge example (问题点 / 边数据举例)
   
   ```javascript
   
   ```
   
   ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
   
   ```javascript
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to