cui2022 commented on issue #3150:
URL: https://github.com/apache/hugegraph/issues/3150#issuecomment-5252074460
<html>
<body>
<!--StartFragment--><html><head></head><body><p>英文翻译如下（适合提交给 HugeGraph 开发者 /
GitHub Issue / 邮件讨论）：</p><hr><p>I provided the source code of
<strong>pd-service</strong> to Claude Code and explained my issue. Claude Code
analyzed the code and believes that there may be a bug in the replica migration
logic when a Store node enters the <strong>Tombstone</strong> state. The issue
may cause the old Store information in the PD metadata to not be updated
correctly.</p><p>I am not sure whether this analysis can provide any useful
information, but I hope it can help with the investigation.</p><p>Below is the
analysis result from Claude Code:</p><hr><h1>Complete Analysis: Replica
Migration Failure During Offline → Tombstone Transition</h1><h2>Initial
Conditions</h2><p>Assume the cluster state is:</p><pre><code>partition
shardCount = 3
Active Stores:
Store1 (Up)
Store2 (Up)
Store3:
Previously Offline, about to be changed to Tombstone
ShardGroup 0:
[
Store1 (Leader),
Store2 (Follower),
Store3 (Follower)
]
</code></pre><p>Note:</p><p>When Store3 was changed to
<strong>Offline</strong>, <code inline="">updateStore()</code> only called
<code inline="">removeActiveStore()</code> (line 306), but did not call <code
inline="">storeTurnoff()</code>. Therefore, the ShardGroup stored in RocksDB
still contains Store3's shard information.</p><hr><h2>Phase 1: REST API
Entry</h2><p>User sends:</p><pre><code>POST /v1/store/3
{
"storeState": "Tombstone"
}
</code></pre><p><code inline="">StoreAPI.java:90-97</code></p><pre><code
class="language-java">Metapb.Store lastStore = pdRestService.getStore(storeId);
// Store3, current state = Offline
Metapb.Store newStore = ... setState(Tombstone);
pdRestService.updateStore(newStore);
</code></pre><p>Then the request enters the core update
logic.</p><hr><h2>Phase 2: updateStore()</h2><p><code
inline="">updateStore()</code> changes:</p><pre><code>lastStore.state = Offline
store.state = Tombstone
</code></pre><h3>State validation</h3><p>Because:</p><pre><code
class="language-java">store.getState() == Tombstone
</code></pre><p>is true.</p><p>However:</p><pre><code
class="language-java">lastStore.getState() == Up
</code></pre><p>is false because the previous state was already
Offline.</p><p>Therefore, the <code inline="">minStoreCount</code> validation
is skipped and the update succeeds.</p><hr><h2>Phase 3: Tombstone
Processing</h2><p>The Store metadata is updated:</p><pre><code
class="language-java">storeInfoMeta.updateStore(store);
</code></pre><p>The state transition is detected:</p><pre><code>Offline
-> Tombstone
</code></pre><p>Then:</p><pre><code
class="language-java">storeInfoMeta.removeActiveStore(store);
storeTurnoff(store);
</code></pre><p>is executed.</p><hr><h2>Phase 4: First Replica Migration
Attempt</h2><p>Inside:</p><pre><code>storeTurnoff(Store3)
</code></pre><p>The code finds:</p><pre><code>ShardGroup:
[
Store1,
Store2,
Store3
]
</code></pre><p>Then it removes Store3:</p><pre><code>[
Store1,
Store2
]
</code></pre><p>and calls:</p><pre><code
class="language-java">reallocShards(filteredShardGroup)
</code></pre><hr><h2>Potential Bug 1: reallocShards() Early
Return</h2><p>Inside:</p><pre><code>reallocShards()
</code></pre><p>Active stores:</p><pre><code>[
Store1,
Store2
]
</code></pre><p>Configured shard count:</p><pre><code>3
</code></pre><p>Then:</p><pre><code class="language-java">shardCount =
Math.min(3, activeStores.size());
</code></pre><p>becomes:</p><pre><code>shardCount = 2
</code></pre><p>The input shard list is already:</p><pre><code>[
Store1,
Store2
]
</code></pre><p>Therefore:</p><pre><code class="language-java">shardCount ==
shards.size()
</code></pre><p>and execution reaches:</p><pre><code
class="language-java">else {
return shards;
}
</code></pre><p>The function returns immediately.</p><p>As a
result:</p><ul><li><p><code inline="">storeInfoMeta.updateShardGroup()</code>
is <strong>not called</strong></p></li><li><p><code
inline="">fireChangeShard()</code> is <strong>not
called</strong></p></li><li><p>RocksDB still contains:</p></li></ul><pre><code>[
Store1,
Store2,
Store3
]
</code></pre><p>The migration attempt effectively does
nothing.</p><hr><h2>Phase 5: Store Status Listener
Triggered</h2><p>After:</p><pre><code>Offline -> Tombstone
</code></pre><p>the listener is triggered:</p><pre><code
class="language-java">PartitionService.storeOffline(store);
</code></pre><p>This starts another migration attempt.</p><hr><h2>Phase 6:
Second Migration Attempt</h2><p><code inline="">storeOffline()</code> reads the
ShardGroup from RocksDB:</p><pre><code>[
Store1,
Store2,
Store3
]
</code></pre><p>Because the previous migration did not update
metadata.</p><p>Then it calls:</p><pre><code
class="language-java">reallocShards(shardGroup);
</code></pre><hr><h2>Potential Bug 2: Wrong Replica
Removal</h2><p>Now:</p><pre><code>activeStores = [
Store1,
Store2
]
shards = [
Store1 Leader,
Store2 Follower,
Store3 Follower
]
</code></pre><p>Again:</p><pre><code class="language-java">shardCount =
Math.min(3,2)
</code></pre><p>so:</p><pre><code>shardCount = 2
</code></pre><p>The system needs to remove one replica.</p><p>The current
logic:</p><pre><code class="language-java">if (iterator.next().getRole() !=
Leader) {
iterator.remove();
}
</code></pre><p>only checks whether the shard is a Leader.</p><p>It does not
check:</p><ul><li><p>whether the Store is Tombstone</p></li><li><p>whether the
Store is Offline</p></li><li><p>whether the Store is
healthy</p></li></ul><p>Because the iteration order is:</p><pre><code>Store1
Leader
Store2 Follower
Store3 Follower
</code></pre><p>The result becomes:</p><pre><code>Store1 Leader (kept)
Store2 Follower (removed)
Store3 Tombstone (kept)
</code></pre><p>Therefore:</p><ul><li><p>A healthy Store2 is
removed.</p></li><li><p>The failed Tombstone Store3 remains in the replica
list.</p></li></ul><p>The metadata is then updated incorrectly:</p><pre><code>[
Store1,
Store3
]
</code></pre><p>and a conf change is
sent:</p><pre><code>fireChangeShard([Store1, Store3])
</code></pre><p>which still includes the unavailable Store3.</p><hr><h2>Root
Cause Summary</h2><p>There appear to be two potential bugs:</p>
Bug | Location | Problem
-- | -- | --
Bug 1 | reallocShards() | When the filtered shard list size equals the
calculated shard count, the function returns directly without updating PD
metadata. The removed Tombstone store remains in RocksDB.
Bug 2 | reallocShards() | Replica removal only avoids removing the Leader.
It does not consider Store state, so a healthy follower can be removed while a
Tombstone store remains.
<hr><h2>Trigger Condition</h2><p>The issue happens
when:</p><pre><code>activeStoreCount <= configuredShardCount
</code></pre><p>Because:</p><pre><code
class="language-java">Math.min(configuredShardCount, activeStoreCount)
</code></pre><p>reduces the target replica count.</p><p>When the filtered
shard list size equals this reduced value, the early return path is
triggered.</p><hr><h2>My Questions</h2><ol><li><p>Is this analysis consistent
with the expected behavior of HugeGraph-Store PD?</p></li><li><p>Is there any
known issue regarding Offline → Tombstone transition and replica
migration?</p></li><li><p>Is there a recommended workaround or recovery
procedure for this situation?</p></li><li><p>Is there any API or administrative
operation that can manually trigger replica migration or force PD to rebuild
the ShardGroup metadata?</p></li></ol><p>Thank you very much for your
help.</p><hr>
</body>
</html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
