mrproliu opened a new pull request, #1187: URL: https://github.com/apache/skywalking-banyandb/pull/1187
## What The `migration` tool now migrates **index-mode** measure groups (e.g. SkyWalking's `sw_metadata`) across all three subcommands — `copy`, `verify`, and `analyze`. A single plan may mix normal measure, index-mode measure, and stream groups; each group is routed to its catalog and its in-catalog mode automatically. Index-mode groups were previously **refused** up front. ## Why An index-mode measure stores no columnar parts: every data point is a single inverted document under the segment-level `seg/sidx/`, upserted by series per segment (one max-version doc per `(series, segment)`). The existing copy path builds and broadcasts a per-group union secondary index, which is correct for normal measures but corrupts index-mode data (its values live inside the sidx). So index-mode groups had no offline migration path — operators re-gridding or relocating a cluster had to leave `sw_metadata` behind. ## How - **Classification & routing.** Each measure group is classified as normal or index-mode (a group whose only non-`_top_n_result` measures all carry `IndexMode: true`). Normal groups keep the union-sidx path; index-mode groups take a dedicated sidx rebuild / byte-copy path and are excluded from union-sidx broadcast via an optional `SkipUnionSidx` executor capability. - **Copy.** For each source segment's sidx, the tool either byte-copies the whole directory (fast path: every doc aligns to one not-yet-written target segment with no other source feeding it) or rebuilds doc-by-doc (slow path: the source splits across target segments, or several sources merge into one). On rebuild, stored tag fields are carried over verbatim, and the index-only entity fields (`_im_name` / `_im_entity_tag_*`, which the write path never stores) are regenerated from each doc's `_id`, with `Analyzer` / `NoSort` restored from the catalog's index rules so the migrated sidx stays searchable. Re-gridding onto a coarser/realigned target merges redundant same-series docs keeping the highest version, so the target doc count is `≤` source while every distinct series is preserved. - **Bad-data guard.** A valid index-mode doc always carries a checked, non-zero `_timestamp` (the write path stamps it), so a doc read back with `Timestamp == 0` means the field is missing or undecodable — corrupt source data. The reader (the single chokepoint shared by copy/verify/analyze) now aborts with the offending sidx and a sample of series IDs instead of silently routing the doc to a guessed segment. - **Shard-part guard.** An index-mode group that also holds a non-`_top_n_result` normal measure with actual on-disk parts is refused. The guard keys off real part directories, not bare `shard-*` directory existence — the segment controller can create an empty `shard-0/` (or a dataless `_top_n_result` leaves one behind) without any normal-measure data, which must not block the copy. - **Verify.** For index-mode groups, `verify` reconciles sidx doc counts, distinct doc-id (series) counts, per-`(segment, series)` full-field value digests, and per-segment 1:1 alignment, instead of the part-row counts used for normal measures. - **Analyze.** For index-mode groups, `analyze` reports `(series, timestamp)` version-duplicate and value-conflict keys, which explain why the target doc count is `≤` source. - **Docs.** `README.md` and `MIGRATION.md` are corrected — they previously stated index-mode was refused and `sw_metadata` was out of scope — with an "Index-mode specifics" section, the index-mode segment layout (sidx only, no `shard-N/`), and a note that the runbook still keeps `sw_metadata` in place for operational reasons even though the tool can migrate it. ## Tests - **Disk-level unit tests** cover doc rebuild (stored tags + regenerated index-only fields), the fast/slow path split, multi-source max-version merge, the `ts == 0` abort, the empty-`shard-*` guard, searchability preservation (the migrated sidx answers term queries with source-matching hit counts), and the verify/analyze reconciliation. - **Service-level end-to-end test** drives the full path through the live measure service: it registers an index-mode group + measure + an index rule (+ binding), writes points across two segments, re-grids at migration so the two source segments **merge into one** target segment (forcing the rebuild/slow path — asserted both structurally and by `Totals.Bytes == 0`, which only the byte-copy path increments), then re-opens the target and asserts the migrated data answers a full-scan and an indexed-tag-filter query **identically to the source**, comparing every tag value, every field value, and the timestamp field-for-field. - **Stability.** Every asynchronous assertion waits on an explicit readiness signal rather than a fixed timeout — in particular, writes are gated on the index rule actually being **applied** (`GetIndexRules()` reflecting the same index schema the write path consults), since the binding event arrives on a separate channel after the measure resolves and a point written in that window would land un-indexed permanently. The e2e passes repeatedly under `GOMAXPROCS=1`. - `make test` and `make lint` pass. - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Fixes apache/skywalking#<issue number>. - [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
