SebastianGruza opened a new issue, #3224:
URL: https://github.com/apache/hugegraph/issues/3224

   ### Bug Type (问题类型)
   
   logic (逻辑设计问题) / upgrade
   
   ### Before submit
   
   - [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 
[FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have 
confirmed and searched that there are no similar problems in the historical 
issue and documents)
   
   ### Environment (环境信息)
   
   Before: official `apache-hugegraph-incubating-1.7.0` (PD, 3 Store, Server), 
`usePD=true`, 12 partitions per store. After: `lib/` of PD, Stores and Server 
replaced with jars built from master `83ef9f3f` (2026-09-20, includes #3153 and 
#3220), conf and data directories untouched. Lab: PD + 3 stores on three VMs, 
server on the PD node. Second occurrence on a separate deployment upgraded the 
same way (data loaded on 1.7.0 through batch PUT, snapshots taken during normal 
operation).
   
   ### Expected & Actual behavior (期望与实际表现)
   
   **Expected:** after upgrading a 1.7.0 HStore cluster to master in place, the 
data written by 1.7.0 stays readable, before and after the first write.
   
   **Actual** (lab, 
[`results/upgrade-170-to-master/cycle4-graphid`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/upgrade-170-to-master/cycle4-graphid),
 16 Gremlin checks compared with the 1.7.0 baseline): 1.7.0 batch load 29 
vertices / 63 edges; forced raft snapshot on every partition; a second batch of 
5 vertices: 34 / 63.
   
   | step | A: batch entries still in the raft log at the swap | B: snapshot 
again right before the stop |
   |---|---|---|
   | master start, **no client write** | **15 V / 23 E**; 8 allocations per 
store during startup, all from `PartitionStateMachine.onApply → doBatch → 
getKeyOrCreate` | 30 V / 56 E; 2 allocations per store |
   | first client batch on master (one vertex) | n/a | **23 V / 41 E**; 4 
allocations per store |
   | identical to the 1.7.0 baseline | 0/16 | 4/16 |
   | `GET :8520/fix/graph_ids/{pid}` | the 1.7.0 rows under `65534` with 
`"graph": "not found"`; the allocated id holds only what was replayed or 
written since | same |
   | rollback: 1.7.0 jars on the allocated mapping | not measured (graph not 
exposed within the wait) | 23 V / 41 E: 1.7.0 honours the mapping, rollback 
does not restore the view |
   
   No `Failed to parse entry` in the server log in either variant: with the 
schema visible the rows are simply not read. (With jars that predate #3220 the 
same run also logs parse errors, [cycle 
3](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/upgrade-170-to-master/cycle3-graphid);
 that is #3219, not this issue.) On the second deployment, upgraded the same 
way from a 1.7.0 loader import with snapshots taken during normal operation, 
most of the data was gone after the upgrade and get-by-id answered "does not 
exist" for ids present in the store.
   
   ### Analysis
   
   - Physical key: `[2B graph id][logical key][2B code]`. In 1.7.0 the batch 
PUT/MERGE path used `getKey()`, which returns the sentinel `0xFFFE` when the 
graph has no mapping in the partition (#3095). A 1.7.0 graph whose first write 
in a partition was a batch (the normal loader and REST batch case) therefore 
holds all its rows under `0xFFFE`. Our lab SST files start with `fffe a7 …` in 
every partition.
   - #3153 (master since 2026-08-26) switches the batch path to 
`getKeyOrCreate()`; its description states that it does not repair already 
affected data.
   - Read path unchanged: `GraphIdManager.getGraphId()` falls back to 
`maxGraphID` (`0xFFFE`) when there is no mapping (`GraphIdManager.java:63-77`), 
so a freshly upgraded cluster reads its 1.7.0 rows correctly. This is why an 
upgrade test that only reads passes.
   - First allocation on master, two triggers: (a) the store's own startup, 
when jraft replays the log entries since the last snapshot through 
`PartitionStateMachine.onApply → doBatch → getKeyOrCreate`, which now allocates 
(variant A, no client involved); (b) otherwise the first batch write from a 
client (variant B). `GraphIdManager.getGraphIdOrCreate()` persists `graph → id` 
in the partition metadata; from then on every read and write uses that id and 
the `0xFFFE` range is never consulted again. Visible after the upgrade is 
exactly what the log replayed; everything captured by snapshots (default 
`snapshotInterval` 1800 s in 1.7.0, so any cluster older than half an hour) is 
unreachable.
   - Rolling the jars back to 1.7.0 does not restore the view (measured, 
variant B: 23 V / 41 E on 1.7.0): its `getGraphId()` reads the same persisted 
mapping.
   - The condition is the common one, not a corner case: any 1.7.0 graph loaded 
with the loader or with batch REST, upgraded in place.
   
   ### How to reproduce (lab, [`cycle4_graphid.sh 
A|B`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/results/upgrade-170-to-master/cycle4-graphid/cycle4_graphid.sh))
   
   1. 1.7.0 PD + 3 Stores + Server, `usePD=true`; create the schema; load 
through `POST /graph/vertices/batch` and `/graph/edges/batch`.
   2. `GET http://<store>:8520/test/snapshot` on every store (or wait for the 
periodic snapshot, 1800 s by default); optionally one more batch.
   3. Stop everything without wiping; replace `lib/` in PD, Stores and Server 
with master jars; start. Do not write anything.
   4. Read: only the rows replayed from the raft log are visible. If the log 
was empty, reads are still complete (the `0xFFFE` fallback); then one batch 
write from a client, and the 1.7.0 rows disappear. Store logs: `Graph ID N is 
allocated for graph …`; `/fix/graph_ids/{pid}`: the 1.7.0 rows under `65534`, 
"not found".
   
   ### Ways out, to be discussed before any code
   
   1. **Adopt `0xFFFE` at first allocation.** In `getGraphIdOrCreate()`, when 
the graph has no mapping and the partition already holds rows under `0xFFFE` 
and no other graph is mapped in that partition, persist `graph → 0xFFFE` 
instead of allocating a new id. Zero data movement, transparent upgrade for the 
single-graph-per-store case (every 1.7.0 deployment I know), and it covers both 
triggers because the raft replay goes through the same allocation. Ambiguous 
when several graphs shared `0xFFFE` (the #3095 case): then refuse to allocate 
and require step 2.
   2. **Migration endpoint.** `FixGraphIdController` already has the pieces 
(`graph_ids`, `check`, `update_graph_id`, `delete_graph_id`); add 
`migrate/{partition}/{graph}` that rewrites the `0xFFFE` range of `g+v`, 
`g+oe`, `g+ie`, `g+index` to the graph's id (idempotent, resumable, per 
partition), plus a dry-run count. Needed anyway for clusters that already 
allocated. Open question: should it run through raft (each replica rewrites its 
own copy today, the controller writes locally).
   3. **Upgrade guard.** A store started with master on data that still has a 
non-empty `0xFFFE` range and no mapping logs a WARN with the affected 
partitions before replaying the raft log; refusing writes alone is not enough, 
since the replay itself allocates. Plus an upgrade note in 
`hugegraph-store/docs/operations-guide.md` and in the 1.8 release notes.
   
   ### Workaround for operators who already hit it (exercised on the lab, both 
variants)
   
   Map the graph back to `0xFFFE` on every partition of every store: `POST 
http://<store>:8520/fix/update_graph_id/{partition}` with body 
`{"<graphspace>/<graph>/g": 65534}`. On the lab this brought the graph back to 
34 V / 63 E immediately, 16/16 checks identical to the 1.7.0 baseline; the 
mapping is persisted (`has_slot_id: true`), survives a store restart without a 
new allocation, and a write under it lands under `0xFFFE` as well 
(`getGraphIdOrCreate()` honours a persisted mapping equal to the sentinel, 
re-reading the key on every call). Caveats: rows written under the allocated id 
in between become unreachable in turn (they reappear only if their raft entries 
are replayed after the remap); the endpoint writes local metadata outside raft, 
so it has to be run on each replica; it only makes sense when the store holds 
one graph (with several graphs under `0xFFFE` the #3095 collision comes back).
   
   Related: #3095, #3153, #3223 (rolling-upgrade and rollback requirements: 
this is requirement (d), upgrade and rollback with data written by the other 
version). I am not proposing a PR yet; the direction (1 vs 2 vs both) deserves 
a decision first. The harness above is ready to verify any fix: the acceptance 
bar is 16/16 after the in-place upgrade without the remap.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to