SebastianGruza opened a new issue, #3224: URL: https://github.com/apache/hugegraph/issues/3224
### Bug Type (问题类型) logic (逻辑设计问题) / upgrade ### Before submit - [x] 我已经确认现有的 [Issues](https://github.com/apache/hugegraph/issues) 与 [FAQ](https://hugegraph.apache.org/docs/guides/faq/) 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents) ### Environment (环境信息) Before: official `apache-hugegraph-incubating-1.7.0` (PD, 3 Store, Server), `usePD=true`, 12 partitions per store. After: `lib/` of PD, Stores and Server replaced with jars built from master `83ef9f3f` (2026-09-20, includes #3153 and #3220), conf and data directories untouched. Lab: PD + 3 stores on three VMs, server on the PD node. Second occurrence on a separate deployment upgraded the same way (data loaded on 1.7.0 through batch PUT, snapshots taken during normal operation). ### Expected & Actual behavior (期望与实际表现) **Expected:** after upgrading a 1.7.0 HStore cluster to master in place, the data written by 1.7.0 stays readable, before and after the first write. **Actual** (lab, [`results/upgrade-170-to-master/cycle4-graphid`](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/upgrade-170-to-master/cycle4-graphid), 16 Gremlin checks compared with the 1.7.0 baseline): 1.7.0 batch load 29 vertices / 63 edges; forced raft snapshot on every partition; a second batch of 5 vertices: 34 / 63. | step | A: batch entries still in the raft log at the swap | B: snapshot again right before the stop | |---|---|---| | master start, **no client write** | **15 V / 23 E**; 8 allocations per store during startup, all from `PartitionStateMachine.onApply → doBatch → getKeyOrCreate` | 30 V / 56 E; 2 allocations per store | | first client batch on master (one vertex) | n/a | **23 V / 41 E**; 4 allocations per store | | identical to the 1.7.0 baseline | 0/16 | 4/16 | | `GET :8520/fix/graph_ids/{pid}` | the 1.7.0 rows under `65534` with `"graph": "not found"`; the allocated id holds only what was replayed or written since | same | | rollback: 1.7.0 jars on the allocated mapping | not measured (graph not exposed within the wait) | 23 V / 41 E: 1.7.0 honours the mapping, rollback does not restore the view | No `Failed to parse entry` in the server log in either variant: with the schema visible the rows are simply not read. (With jars that predate #3220 the same run also logs parse errors, [cycle 3](https://github.com/SebastianGruza/hugegraph-validation/tree/master/results/upgrade-170-to-master/cycle3-graphid); that is #3219, not this issue.) On the second deployment, upgraded the same way from a 1.7.0 loader import with snapshots taken during normal operation, most of the data was gone after the upgrade and get-by-id answered "does not exist" for ids present in the store. ### Analysis - Physical key: `[2B graph id][logical key][2B code]`. In 1.7.0 the batch PUT/MERGE path used `getKey()`, which returns the sentinel `0xFFFE` when the graph has no mapping in the partition (#3095). A 1.7.0 graph whose first write in a partition was a batch (the normal loader and REST batch case) therefore holds all its rows under `0xFFFE`. Our lab SST files start with `fffe a7 …` in every partition. - #3153 (master since 2026-08-26) switches the batch path to `getKeyOrCreate()`; its description states that it does not repair already affected data. - Read path unchanged: `GraphIdManager.getGraphId()` falls back to `maxGraphID` (`0xFFFE`) when there is no mapping (`GraphIdManager.java:63-77`), so a freshly upgraded cluster reads its 1.7.0 rows correctly. This is why an upgrade test that only reads passes. - First allocation on master, two triggers: (a) the store's own startup, when jraft replays the log entries since the last snapshot through `PartitionStateMachine.onApply → doBatch → getKeyOrCreate`, which now allocates (variant A, no client involved); (b) otherwise the first batch write from a client (variant B). `GraphIdManager.getGraphIdOrCreate()` persists `graph → id` in the partition metadata; from then on every read and write uses that id and the `0xFFFE` range is never consulted again. Visible after the upgrade is exactly what the log replayed; everything captured by snapshots (default `snapshotInterval` 1800 s in 1.7.0, so any cluster older than half an hour) is unreachable. - Rolling the jars back to 1.7.0 does not restore the view (measured, variant B: 23 V / 41 E on 1.7.0): its `getGraphId()` reads the same persisted mapping. - The condition is the common one, not a corner case: any 1.7.0 graph loaded with the loader or with batch REST, upgraded in place. ### How to reproduce (lab, [`cycle4_graphid.sh A|B`](https://github.com/SebastianGruza/hugegraph-validation/blob/master/results/upgrade-170-to-master/cycle4-graphid/cycle4_graphid.sh)) 1. 1.7.0 PD + 3 Stores + Server, `usePD=true`; create the schema; load through `POST /graph/vertices/batch` and `/graph/edges/batch`. 2. `GET http://<store>:8520/test/snapshot` on every store (or wait for the periodic snapshot, 1800 s by default); optionally one more batch. 3. Stop everything without wiping; replace `lib/` in PD, Stores and Server with master jars; start. Do not write anything. 4. Read: only the rows replayed from the raft log are visible. If the log was empty, reads are still complete (the `0xFFFE` fallback); then one batch write from a client, and the 1.7.0 rows disappear. Store logs: `Graph ID N is allocated for graph …`; `/fix/graph_ids/{pid}`: the 1.7.0 rows under `65534`, "not found". ### Ways out, to be discussed before any code 1. **Adopt `0xFFFE` at first allocation.** In `getGraphIdOrCreate()`, when the graph has no mapping and the partition already holds rows under `0xFFFE` and no other graph is mapped in that partition, persist `graph → 0xFFFE` instead of allocating a new id. Zero data movement, transparent upgrade for the single-graph-per-store case (every 1.7.0 deployment I know), and it covers both triggers because the raft replay goes through the same allocation. Ambiguous when several graphs shared `0xFFFE` (the #3095 case): then refuse to allocate and require step 2. 2. **Migration endpoint.** `FixGraphIdController` already has the pieces (`graph_ids`, `check`, `update_graph_id`, `delete_graph_id`); add `migrate/{partition}/{graph}` that rewrites the `0xFFFE` range of `g+v`, `g+oe`, `g+ie`, `g+index` to the graph's id (idempotent, resumable, per partition), plus a dry-run count. Needed anyway for clusters that already allocated. Open question: should it run through raft (each replica rewrites its own copy today, the controller writes locally). 3. **Upgrade guard.** A store started with master on data that still has a non-empty `0xFFFE` range and no mapping logs a WARN with the affected partitions before replaying the raft log; refusing writes alone is not enough, since the replay itself allocates. Plus an upgrade note in `hugegraph-store/docs/operations-guide.md` and in the 1.8 release notes. ### Workaround for operators who already hit it (exercised on the lab, both variants) Map the graph back to `0xFFFE` on every partition of every store: `POST http://<store>:8520/fix/update_graph_id/{partition}` with body `{"<graphspace>/<graph>/g": 65534}`. On the lab this brought the graph back to 34 V / 63 E immediately, 16/16 checks identical to the 1.7.0 baseline; the mapping is persisted (`has_slot_id: true`), survives a store restart without a new allocation, and a write under it lands under `0xFFFE` as well (`getGraphIdOrCreate()` honours a persisted mapping equal to the sentinel, re-reading the key on every call). Caveats: rows written under the allocated id in between become unreachable in turn (they reappear only if their raft entries are replayed after the remap); the endpoint writes local metadata outside raft, so it has to be run on each replica; it only makes sense when the store holds one graph (with several graphs under `0xFFFE` the #3095 collision comes back). Related: #3095, #3153, #3223 (rolling-upgrade and rollback requirements: this is requirement (d), upgrade and rollback with data written by the other version). I am not proposing a PR yet; the direction (1 vs 2 vs both) deserves a decision first. The harness above is ready to verify any fix: the acceptance bar is 16/16 after the in-place upgrade without the remap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
