hanahmily opened a new pull request, #1210: URL: https://github.com/apache/skywalking-banyandb/pull/1210
## What Add trace **finalization sampling** (`PIPELINE_EVENT_FINALIZE`) — a best-effort background backstop for the in-merge trace retention filter (`PIPELINE_EVENT_MERGE`). Part of apache/skywalking#13634. ## Why The in-merge sampler only drops traces during opportunistic merges of parts older than `merge_grace`. When hot writing rotates to the next segment, the just-cooled segment can be left with parts the in-merge sampler never revisited (no new writes → no merge pressure). Finalization sampling sweeps those cooled parts through the **same** registered sampler chain. ## Design - A single node-wide, **concurrency-1 scanner** periodically sweeps cooled segments (`segEnd < now − finalize_grace`) and force-merges each shard's un-finalized parts through the group's sampler chain, **reusing the hot merge path** (`mergeParts` + per-sidx `Merge` + the shard's introducer loop) so it never acquires the hot merge semaphore. - **Per-part `finalizeGen` stamp**: min-propagated on ordinary merges, written to disk *before* the part metadata so a crash cannot double-sample on replay (the next round's predicate `finalizeGen < gNext` excludes already-stamped parts). - **Per-shard `finalize.json`** state (generation, cooldown clock, round count, terminal flag, arrival-based unsampled-bytes counter incremented O(1) on the flush path). - **Bounded re-rounds**: `FLOOR` (8 MiB) / `RATIO` (0.10 of segment size) / cooldown (= `finalize_grace`) / hard `max_finalize_rounds` (8, then terminal). - **No reopen storm**: a `storage.PeekSegments` API lets the scanner inspect per-shard on-disk state *without* reopening a segment, so terminal / in-cooldown / max-rounds segments are skipped and only segments that may warrant a round are reopened. - **Concurrency-safe pinning** between the finalize scanner and the hot merge dispatcher (atomic check-and-pin on both sides). - Gated per group via `enabled_events` containing `PIPELINE_EVENT_FINALIZE`; settling window is `finalize_grace` (default 5m, node default `--trace-pipeline-finalize-grace-default`). Finalize compute yields under memory pressure and a disk-headroom gate. **Coverage is best-effort**: misses (e.g. a segment TTL-deleted before finalization) are accepted, not guaranteed — the pass never impacts the hot write/merge/query path. ## Tests Unit + integration coverage: per-part generation propagation (4 crash-safety cases, on-disk re-read), a running-tsTable finalize round, the DD6 replay-exclusion case, `MERGE`+`FINALIZE` composition on one shard, the scanner pre-filter / `PeekSegments` seam, threshold (FLOOR/RATIO/cooldown/terminal), and per-shard state round-trip. Full `banyand/trace/...` and `banyand/internal/storage/...` suites pass; `make lint` clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
