mrproliu opened a new pull request, #1145:
URL: https://github.com/apache/skywalking-banyandb/pull/1145
## Why
The lifecycle migration moves expired data from one stage to the next (hot
→ warm → cold). Each stage may define a **different `segment_interval`**. The
original migration copied whole part files into
a single target segment chosen from the part's timestamp (the
"chunk-sync" path). That is only correct when every source part falls entirely
inside one target segment.
When the source and target segment intervals are **not multiples of each
other** (their day-grids don't align — e.g. a 5-day source interval migrating
into 7-day target segments), a single source
part's rows can **straddle a target segment boundary**. Chunk-copying such
a part dumps **all** of its rows into one target segment, so the rows that
belong to the adjacent segment end up in the wrong
place: the migrated part's `[minTimestamp, maxTimestamp]` escapes its
target segment's `[start, start+interval)` bounds. This produces mis-segmented
data and breaks per-segment assumptions for later
queries and retention.
## What
Detect source parts that cross a target segment boundary and migrate them
with **row-level replay** instead of a file copy:
- **Part fully inside one target segment** → unchanged fast **chunk-sync**
(copies the part file).
- **Part spanning ≥2 target segments** → **row-replay**: re-publish each
row so the receiving node routes it to the target segment that matches the
row's own timestamp. No row lands in the wrong
segment.
### Supporting changes
- **Per-part flush-and-confirm**: each migrated part is flushed and its
server-side acks confirmed before it is marked complete (the batch publisher is
closed/reopened per part), so a part is only
recorded as migrated once it is durably delivered; on failure it is
recorded as an error instead of silently lost.
- **Source-segment completion key**: migration progress is keyed on the
*source* segment instead of the target, fixing a collision where multiple
source segments mapping to one target segment could be
wrongly treated as already-migrated.
- **`raw_element_id` consistency (stream)**: a new `raw_element_id` is
carried through row-replay so an element migrated via both chunk-sync and
row-replay keeps one storage element id; the warm-stage
query dedup then converges to a single row instead of duplicating it.
- **Migration report** now breaks down `chunk_sync` vs `row_replay`
parts/rows per group so the path taken is observable.
## Tests
- New `row_replay_*` unit tests for the per-type replayers and progress
tracking.
- Distributed e2e (`TestPropertyLifecycle`): added measure / stream /
trace cross-segment specs that seed boundary-straddling data, run the
migration, and verify (a) the **physical** layout — every
migrated part stays within its own target segment, segments split
correctly, exact segment count + contiguous boundaries — and (b) the **gRPC
query** results from the warm stage (per-timestamp,
same-segment, and cross-segment aggregate). Stream specs also assert
`raw_element_id` dedup; trace adds a `cross_segment_timestamp` index rule so
warm-stage queries are verifiable.
- `make lint`, lifecycle unit tests, and the distributed e2e (5/5) all
pass.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace
the issue number. Fixes apache/skywalking#<issue number>.
- [x] Update the [`CHANGES`
log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
