hanahmily commented on PR #1144: URL: https://github.com/apache/skywalking-banyandb/pull/1144#issuecomment-4564833629
Force-pushed a simplification: **`StageEvent` enum and `apply_at` field removed**; per-stage filtering now fires only at migration-out. The `merge_grace` field, the §7.1 in-merge filter chapter, and the §5.1 compaction-rewrites section are gone as natural consequences. **Two-anchored-points architecture (down from three):** - **Tail-sampling gate at finalization** (`tail_sampling`, gated by `finalize_grace` watermark) — decides whether a trace is retained at all once its segment is settled. - **Per-stage retention at migration-out** (`StageRule` predicates — `min_duration` / `keep_errors` / `keep_tag_rules`) — decides what migrates to the next stage. Routine LSM compaction is now intentionally outside the pipeline's scope; it stays byte-for-byte lossless. This removes the entire merge-time complexity (the `merge_grace` per-trace maturity gate, the in-merge filter hook in `mergeBlocks`, the lossy-merge correctness rules) — none of it was load-bearing once stage filtering is anchored to migration boundaries (a segment is already ≥ stage TTL old at migration, so completeness concerns are already satisfied without a separate merge-time grace). **Proto deltas** vs the previous PR push: - Removed `enum StageEvent`, `StageRule.apply_at`, `TracePipelineConfig.merge_grace`. - Renumbered `StageRule` fields (`stage`=1, `min_duration`=2, `keep_errors`=3, `keep_tag_rules`=4) and `TracePipelineConfig.finalize_grace` (7). - `buf lint` + `buf build` pass. **Doc deltas:** - §5 collapses from 3 subsections to 2 (`§5.1` Migration + `§5.2` Eviction); the former §5.1 Compaction Rewrites is gone. - §7 collapses from 3 subsections to 2 (`§7.1` Pre-Migration Rewrite + `§7.2` Scheduled Finalization); the former §7.1 LSM Compaction Merge Phase Loop is gone. - §1.2, §2.1, §2.2, §2.3 (tuple now `(Group, Schema, Stage)`, no `StageEvent`), §2.4 (no `merge_grace` bullet), §3.1 (no "Provisional during LSM merge" bullet), §4.2, §6.x JSONs (no `apply_at` lines, no `merge_grace`), §7.3-now-§7.2 (single-pass tail_sampling, no FINALIZE-targeted StageRule composition), §8 step-by-step (gating then migration-out only), and backlog all updated. - Section numbering renumbered to stay contiguous: §1-§7 with no gaps. **Net diff:** 3 commits, 4 files, +876 / 0 vs `apache/main` (down from +971 — ~95 lines removed by the simplification). `make generate` / `buf lint` / `buf build` all pass. 3 commits on the PR: - `767588e7` Add post-trace pipeline design (per-stage retention + tail-sampling) - `c8ea4da2` Add trace_pipeline.proto for banyandb.pipeline.v1 - `ca0e9d4d` Preserve deferred weighted-scoring model under design/backlog -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
