Copilot commented on code in PR #1144: URL: https://github.com/apache/skywalking-banyandb/pull/1144#discussion_r3449311225
########## api/proto/banyandb/pipeline/v1/trace_pipeline.proto: ########## @@ -0,0 +1,271 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +syntax = "proto3"; + +package banyandb.pipeline.v1; + +import "banyandb/common/v1/common.proto"; +import "google/api/annotations.proto"; +import "google/protobuf/duration.proto"; +import "google/protobuf/struct.proto"; +import "protoc-gen-openapiv2/options/annotations.proto"; +import "validate/validate.proto"; + +option go_package = "github.com/apache/skywalking-banyandb/api/proto/banyandb/pipeline/v1"; +option java_package = "org.apache.skywalking.banyandb.pipeline.v1"; +option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {base_path: "/api"}; + +// PipelineEvent identifies a pipeline-wide event that can be independently +// enabled. Per-stage retention (StageRule.sampler_plugin) fires implicitly at +// the stage's migration-out boundary and is not toggleable via this enum. +enum PipelineEvent { + PIPELINE_EVENT_UNSPECIFIED = 0; + // In-merge filter during Hot-phase LSM compaction merges (Warm/Cold + // compactions stay lossless). Per-trace drops are gated by `merge_grace` so + // partial traces are not destroyed prematurely. Cheap, runs often; verdicts + // wait for trace maturity (see §7.1). + PIPELINE_EVENT_MERGE = 1; + // Tail-sampling gate at Hot-phase segment finalization, after the segment + // has settled (event-time watermark past `segment.End + finalize_grace`). + // Heavy but authoritative; sees the complete trace (see §7.3). + PIPELINE_EVENT_FINALIZE = 2; +} Review Comment: The PR description says the proto adds declarative tail-sampling/per-stage rule messages (e.g. TailSampling, TagSamplingRule, TagMatcher, StringList) and a StageEvent enum (COMPACTION/FINALIZE/MIGRATION_OUT), but this proto defines only the plugin-based SamplerPlugin model and a PipelineEvent enum (MERGE/FINALIZE). Please align the PR description (and/or the proto) so reviewers/users aren’t misled about the actual API surface being introduced. ########## docs/design/post-trace-pipeline.md: ########## @@ -0,0 +1,542 @@ +# BanyanDB Storage-Node Post-Trace Pipeline & Storage Tier Mapping Specification + +This document presents the complete technical design for the **BanyanDB Storage-Node Post-Trace Pipeline** and its integration with physical hardware storage tiers and time-aging schedulers. + +Unlike traditional streaming-based telemetry engines that perform span assembly inside active, memory-heavy windowing pipelines, BanyanDB relies on a **native trace model**. Spans are stored, sorted, and indexed by `trace_id` directly within the storage engine layout on individual data nodes. + +By decoupling trace assembly from the active ingestion path, the post-trace pipeline executes asynchronous tail-sampling, per-stage retention filtering, and data reduction natively during storage lifecycle events on the data nodes. This design is grounded in recent academic advancements in post-hoc retroactive tracing and storage-level file merge analysis. + +## Architectural Concept: Decoupled Gating and Per-Stage Retention + +To maximize storage efficiency and compute performance on the BanyanDB data nodes, trace evaluation is separated into two logical phases, which subsequently feed into an automated **time-aging system** for partition-level migration: + +```mermaid +flowchart TD + GA["Grouped Trace Assembly"] + GA --> G["1. Gating (sampler_plugin, at merge/finalize)<br/>Goal: decide whether a trace is retained at all<br/>Criteria: native Go plugin verdict (operator-defined)"] + G -->|"Retain"| R["2. Per-Stage Retention (StageRule.sampler_plugin)<br/>Goal: at each stage's migration-out event, keep/drop per the stage's plugin verdict<br/>Criteria: per-stage native Go plugin verdict (operator-defined)"] + G -->|"Drop / Purge"| D["Discard Block<br/>Reclaim Space"] + R --> T["3. Time-Aging System (Stage-Stepped Migration Engine)<br/>Goal: each stage's StageRule decides which traces migrate to the next stage;<br/>routine Hot-phase compaction may also drop traces when PIPELINE_EVENT_MERGE is on (default)<br/>Stage Migration: Hot → Warm → Cold (LifecycleStage order)<br/>RULE: medium is a node-group concern (may be heterogeneous); the per-stage plugin governs retention, not medium selection"] +``` + +### 1.1 Gating (`sampler_plugin`) + +- **Operation Type**: A native Go plugin (`sampler_plugin`) delivers the keep/drop verdict; the plugin owns the entire decision. It is the sole gating mechanism, invoked at whichever of the toggleable events is enabled. + +- **Responsibility**: Gating runs **only in the Hot phase**, at whichever of the toggleable events is enabled — `PIPELINE_EVENT_MERGE` (in-merge on Hot compactions, §7.1; default-on) and/or `PIPELINE_EVENT_FINALIZE` (on settled Hot segments, §7.3); Warm and Cold compactions are byte-for-byte lossless. It determines whether a freshly-assembled trace block **survives** at all — passing into the stage lifecycle — or is **purged** to reclaim storage space. + +- **Compute Profile**: Operator-defined. The plugin receives a vectorized batch of traces and, via its `Project()` declaration, materializes only the columns it needs — the named tag columns and, only when requested, the heavy span bodies (§2.5). Verdicts are expected to be pure in their inputs (the `TraceBatch` plus the frozen `config`), so they are deterministic in `trace_id` and stable across re-evaluation once the trace is mature per `merge_grace` or settled per `finalize_grace`. + +- **Contract**: A native Go plugin (a `.so` loaded in-process via the Go `plugin` package) that owns the entire keep/drop verdict, invoked by the same enabled events at the same maturity/settling gates. Its contract — a vectorized columnar batch in, a boolean keep-mask out, with up-front column projection — is specified in §2.5. + +### 1.2 Per-Stage Retention (`StageRule.sampler_plugin`) + +- **Operation Type**: Per-stage keep/drop verdict by a native Go plugin. Each `StageRule` carries its own `sampler_plugin`; a trace it drops is omitted from the part written for the next stage at the stage's migration-out boundary. + +- **Responsibility**: Decides **which traces survive each stage** as data ages. Each `StageRule` fires once per segment lifetime, at the stage's **migration-out** boundary (when the segment migrates to the next stage). The "rising bar" effect — Hot keeps more, Cold keeps less — is expressed by tightening each stage plugin's config at successive stages. The pipeline does **not** choose the physical storage medium; that is a node-group / `LifecycleStage` placement concern (§4.1). Routine Hot-phase LSM compaction is governed independently by `PIPELINE_EVENT_MERGE` (on by default, applies the plugin gating policy with a per-trace `merge_grace` gate; see §5.1 / §7.1). With merge disabled — and at every Warm/Cold compaction — LSM compaction stays byte-for-byte lossless. + +- **Compute Profile**: Operator-defined — the same vectorized contract as the gating plugin (§2.5): the stage plugin receives a batch of traces, projects only the columns it declares via `Project()`, and returns a boolean keep-mask. `MinTS`/`MaxTS` are free from block metadata; tag columns are decoded only when the plugin's projection requests them. + +## Protobuf Message Design + +The pipeline configuration is a single trace-typed message, `TracePipelineConfig`. Rather than a parallel abstract metadata layer, it reuses the existing catalog identifiers — Group (via `metadata`), lifecycle stage names, and schema names — and adds only the trace-specific gating (a native sampler plugin) and per-stage retention rules. General stream parsing and metrics aggregation configurations are excluded to maintain a strict focus on trace-centric analytical operations. + +### 2.1 Targeting Model: Reuse Existing Catalog Identifiers + +This design deliberately does **not** introduce an abstract `Pipeline` resource or an `ExecutionTrigger` enum. Both would re-declare targeting that the storage model already owns and let the two drift. A trace pipeline is instead expressed entirely with identifiers that already exist: + +- **Group** — named by the config's own `metadata.group` (`common.v1.Metadata`). A `TracePipelineConfig` lives in, and applies to, that Group, exactly as every other schema resource does. The Group already fixes the `catalog`, so catalog is never repeated. +- **Lifecycle stages** — a `StageRule` per targeted stage, each naming a stage from the Group's `ResourceOpts.stages` (e.g. `"hot"`, `"warm"`, `"cold"`) and carrying that stage's **retention `sampler_plugin`** (the same native-plugin mechanism as gating). Each rule fires at the stage's migration-out boundary. Stage names are the same vocabulary queries already accept (`trace/v1/query.proto`'s `stages`). All retention policy is configured here, per stage and per pipeline — there is no hardcoded retention logic in the engine. +- **Schema selector** — an explicit `schema_names` list (exact match on `common.v1.Metadata.name`) plus a `schema_name_regex` (RE2). A schema matches if it is listed OR matches the regex; both empty targets every schema in the Group. + +The former `ExecutionTrigger` (COMPACTION / MIGRATION / SCHEDULED) is replaced by three anchored events: the **in-merge filter at LSM compaction** (§7.1, toggleable via `PIPELINE_EVENT_MERGE` — on by default), the **plugin gating pass at finalization** (§7.3, toggleable via `PIPELINE_EVENT_FINALIZE`), and the **per-stage retention pass at migration-out** (§7.2, always-on when any `StageRule` carries a `sampler_plugin`). The first two run the gating plugin; the third runs the per-stage `StageRule` plugins. This keeps the trace-specific rules out of the generic, catalog-agnostic `common.v1.LifecycleStage` (which stream and measure groups share) while still binding them to stages by name. + +Multiple `TracePipelineConfig`s may coexist in a Group only when their effective coverage is disjoint; the schema registry enforces this with a per-tuple uniqueness rule (§2.3). + +### 2.2 Trace Pipeline Specification (`trace_pipeline.proto`) + +The full message definitions live in the proto source at `api/proto/banyandb/pipeline/v1/trace_pipeline.proto` (package `banyandb.pipeline.v1`); they are not duplicated here. `TracePipelineConfig` is the single root resource: it carries its own identity (`metadata`), the targeting fields from §2.1, and the trace-specific gating (a native sampler plugin) and per-stage retention rules. There is no embedded abstract `Pipeline` and no `ExecutionTrigger`; the anchored filter points are the plugin gating at merge/finalization and per-stage retention at migration-out. + +The message set is: + +- **`TracePipelineConfig`** — root resource: `metadata`, `enabled`, the per-stage `stages` rules, the `schema_names` / `schema_name_regex` selector, the gating policy as a native-plugin `sampler_plugin`, the `enabled_events` list (defaults to `[PIPELINE_EVENT_MERGE]`), and the two grace windows `merge_grace` (§5.1 / §7.1) and `finalize_grace` (§7.3) — each consulted only when its corresponding event is enabled. +- **`PipelineEvent`** — enum of pipeline-wide events: `PIPELINE_EVENT_MERGE` (in-merge filter at LSM compaction, §7.1) and `PIPELINE_EVENT_FINALIZE` (plugin gating pass at segment finalization, §7.3). Per-stage retention (`StageRule`) fires implicitly at migration-out and is not in this enum. +- **`StageRule`** — binds the pipeline to one lifecycle stage and carries that stage's retention `sampler_plugin`: a `stage` name plus a `SamplerPlugin`. The rule fires at the stage's migration-out boundary, where the stage plugin's keep-mask decides which traces migrate; a `StageRule` with no `sampler_plugin` has no filtering effect (see §4.2). +- **`SamplerPlugin`** — the sole gating mechanism: `path` (the `.so` within the trusted plugin dir), `symbol` (constructor, default `NewSampler`), `abi_version` (checked against the host at load), and a structured `config` (`google.protobuf.Struct`) set directly in the pipeline config — the engine serializes it to JSON and hands it to the plugin, which unmarshals it into its own typed config. The vectorized-batch / projection / verdict contract is the Go SDK, not the proto (§2.5). +- **`TracePipelineRegistryService`** — the CRUD registry surface (`Create` / `Update` / `Delete` / `Get` / `List` / `Exist`, with HTTP mappings under `/v1/trace-pipeline/schema`), mirroring every other schema resource's `*RegistryService`. `Create`/`Update` are where the admission and conflict checks of §2.3 / §2.4 run. Without this service a `TracePipelineConfig` would be an orphaned, un-writable resource. + Review Comment: This design doc includes TracePipelineRegistryService as part of the proto/API surface (“CRUD registry surface … mirroring every other schema resource”), which conflicts with the PR description’s statement that the registry RPC is intentionally not part of this PR. Please align the design doc and PR description (either document it as deferred, or acknowledge it is included here). ########## api/proto/banyandb/pipeline/v1/trace_pipeline.proto: ########## @@ -0,0 +1,271 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +syntax = "proto3"; + +package banyandb.pipeline.v1; + +import "banyandb/common/v1/common.proto"; +import "google/api/annotations.proto"; +import "google/protobuf/duration.proto"; +import "google/protobuf/struct.proto"; +import "protoc-gen-openapiv2/options/annotations.proto"; +import "validate/validate.proto"; + +option go_package = "github.com/apache/skywalking-banyandb/api/proto/banyandb/pipeline/v1"; +option java_package = "org.apache.skywalking.banyandb.pipeline.v1"; +option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {base_path: "/api"}; + +// PipelineEvent identifies a pipeline-wide event that can be independently +// enabled. Per-stage retention (StageRule.sampler_plugin) fires implicitly at +// the stage's migration-out boundary and is not toggleable via this enum. +enum PipelineEvent { + PIPELINE_EVENT_UNSPECIFIED = 0; + // In-merge filter during Hot-phase LSM compaction merges (Warm/Cold + // compactions stay lossless). Per-trace drops are gated by `merge_grace` so + // partial traces are not destroyed prematurely. Cheap, runs often; verdicts + // wait for trace maturity (see §7.1). + PIPELINE_EVENT_MERGE = 1; + // Tail-sampling gate at Hot-phase segment finalization, after the segment + // has settled (event-time watermark past `segment.End + finalize_grace`). + // Heavy but authoritative; sees the complete trace (see §7.3). + PIPELINE_EVENT_FINALIZE = 2; +} + +// TracePipelineConfig is the root configuration for a storage-node trace pipeline. +// It reuses existing catalog identifiers (group via metadata, stage names, schema +// names) for targeting instead of declaring a parallel metadata model. +// +// The pipeline has up to three filter points: +// 1. PIPELINE_EVENT_MERGE — in-merge filter during Hot-phase LSM compaction (default). +// 2. PIPELINE_EVENT_FINALIZE — tail-sampling gate at Hot-phase finalization. +// 3. Per-stage retention via StageRule.sampler_plugin, applied at the +// stage's migration-out boundary (when the segment migrates to the next +// stage). Always implicit when any StageRule carries a sampler_plugin. +// Events 1 and 2 are toggleable via `enabled_events`. The gating policy those +// events evaluate is a native Go plugin (`sampler_plugin`). +message TracePipelineConfig { + // Identity and revision tracking; metadata.group is the Group this pipeline + // lives in and applies to, consistent with every other schema resource. + // Required: every config needs a name/group for registry handling. + common.v1.Metadata metadata = 1 [(validate.rules).message = {required: true}]; + // Active status of the pipeline. + bool enabled = 2; + // Per-stage retention rules: which lifecycle stages this pipeline acts on, + // with the retention sampler_plugin for each. Each rule fires at its stage's + // migration-out boundary regardless of `enabled_events`. Empty means the + // only filters are the `enabled_events` events (no per-stage drop). + repeated StageRule stages = 3; + // Explicit schema names to target within the Group (exact match on Metadata.name). + // Each entry must be non-empty; cross-element uniqueness is enforced server-side. + repeated string schema_names = 4 [(validate.rules).repeated.items.string.min_len = 1]; + // RE2 regular expression matched against schema names. A schema is targeted if it + // is listed in schema_names OR matches this pattern. When both are empty, every + // schema in the Group is targeted. + string schema_name_regex = 5; + // Gating policy: a user-supplied native Go plugin (.so) loaded in-process + // that owns the entire keep/drop verdict, evaluated by any enabled event + // (PIPELINE_EVENT_MERGE and/or PIPELINE_EVENT_FINALIZE). Unset means the only + // retention is the per-stage StageRule sampler_plugin(s) at migration-out. + SamplerPlugin sampler_plugin = 6; + // Pipeline-wide events to run. Empty defaults to [PIPELINE_EVENT_MERGE] — + // the in-merge filter is on, the finalization gate is off. To enable the + // finalization gate, include PIPELINE_EVENT_FINALIZE; to disable the merge + // filter, list only [PIPELINE_EVENT_FINALIZE]; the explicit empty default + // value is also acceptable to mean "merge only". Each element must be a + // defined, non-UNSPECIFIED value; duplicates are normalized to a set + // server-side. + repeated PipelineEvent enabled_events = 7 [(validate.rules).repeated.items.enum = { + defined_only: true + not_in: [0] + }]; + // Per-trace maturity window for the in-merge filter (§7.1). A trace is + // eligible for dropping during an LSM compaction merge only once its latest + // span timestamp is older than `now - merge_grace`; younger traces pass + // through the merge unchanged. Bounds the expected intra-trace span arrival + // spread (typically seconds). Used iff `enabled_events` contains + // PIPELINE_EVENT_MERGE. Strictly positive if set; engine default 30s if unset. + google.protobuf.Duration merge_grace = 8 [(validate.rules).duration = { + gt: {seconds: 0} + }]; + // Per-segment settling window for the scheduled finalization pass (§7.3). A + // segment is treated as settled, and the authoritative final filter runs, + // once the event-time watermark exceeds `segment.End + finalize_grace`. + // Bounds segment-wide late arrival (typically minutes). Used iff + // `enabled_events` contains PIPELINE_EVENT_FINALIZE. Strictly positive if + // set; engine default 5m if unset. + google.protobuf.Duration finalize_grace = 9 [(validate.rules).duration = { + gt: {seconds: 0} + }]; +} + +// StageRule binds the pipeline to one lifecycle stage of the targeted Group +// and declares that stage's retention sampler. The rule fires at the stage's +// migration-out boundary (i.e. when a segment migrates from this stage to the +// next stage); routine compaction is governed by PIPELINE_EVENT_MERGE on +// TracePipelineConfig, not by StageRule. +// +// Per-stage retention uses the SAME native-plugin mechanism as gating: each +// stage's `sampler_plugin` owns the keep/drop verdict for traces leaving that +// stage. A StageRule with no `sampler_plugin` set has no filtering effect — +// every trace at this stage migrates unchanged. The "rising bar" across stages +// (Hot keeps more, Cold keeps less) is expressed by each stage plugin's config +// (see §4.2 of the design doc), not by a fixed predicate vocabulary. +message StageRule { + // Stage name from the Group's ResourceOpts.stages (e.g. "hot", "warm", "cold"). + // Must be non-empty; an empty stage name cannot match any lifecycle stage. + string stage = 1 [(validate.rules).string.min_len = 1]; + // Per-stage retention sampler: the native Go plugin that decides keep/drop + // for traces leaving this stage at its migration-out boundary. Unset means no + // per-stage drop (every trace migrates). Same contract as the gating plugin + // (see SamplerPlugin). + SamplerPlugin sampler_plugin = 2; +} + +// SamplerPlugin configures a user-supplied native Go plugin (a .so loaded +// in-process via the Go `plugin` package) that owns a keep/drop verdict over a +// vectorized batch of traces. It is the single sampler mechanism at every +// filter point: as TracePipelineConfig.sampler_plugin it is the gating verdict +// at the enabled PipelineEvent(s) (merge / finalization); as +// StageRule.sampler_plugin it is the per-stage retention verdict at a stage's +// migration-out boundary. +// +// The full plugin↔engine contract (the vectorized batch type, the projection +// handshake, and the verdict shape) lives in the pinned Go SDK module +// `pkg/pipeline/sdk`, not in this proto; this message only locates and admits +// the plugin. Three properties of that contract: +// - Strong compatibility: the boundary exchanges only stdlib/primitive types +// defined in the pinned SDK, so no third-party struct version is pinned +// across the .so boundary. The plugin must be built with the SAME Go +// toolchain, build tags, and flags (-trimpath, CGO) and the SAME pinned SDK +// as the running data node; `abi_version` is checked at load. +// - Vectorized input: the plugin's Decide is called once per columnar batch of +// traces, not once per trace. +// - Projection: the plugin declares the columns it needs (SDK Project → +// Projection{Tags, SpanIDs, Spans}); the engine materializes only those tag +// columns and, only when requested, the spans stream — like a query +// projection. +// +// Operational constraints (Go `plugin`): Linux/macOS only; plugins cannot be +// unloaded, so changing one requires a node restart (no hot-reload); a plugin +// panic is contained with recover() and fails open (the whole batch is +// retained). Loading arbitrary code is operator-only and gated behind a server +// flag plus a trusted plugin directory. +message SamplerPlugin { + // Plugin .so filename, resolved within the data node's trusted plugin + // directory. The engine rejects any path that escapes that directory. + string path = 1 [(validate.rules).string.min_len = 1]; + // Constructor symbol the engine looks up; defaults to "NewSampler" if empty. + string symbol = 2; + // ABI version the plugin was built against. The engine refuses to load the + // plugin unless this equals its own compiled sdk.ABIVersion. + uint32 abi_version = 3 [(validate.rules).uint32 = {gte: 1}]; + // Plugin-defined configuration, set directly in the pipeline config as a + // structured object. The engine does not interpret its keys: it serializes + // the Struct to canonical JSON and hands the bytes to the plugin's + // constructor (SDK NewSampler([]byte)), which unmarshals them into the + // plugin's own typed config and validates them — a malformed config fails + // the load. Optional: a plugin that needs no configuration leaves it unset. + google.protobuf.Struct config = 4; +} + +message TracePipelineRegistryServiceCreateRequest { + TracePipelineConfig trace_pipeline_config = 1 [(validate.rules).message = {required: true}]; +} + +message TracePipelineRegistryServiceCreateResponse { + int64 mod_revision = 1; +} + +message TracePipelineRegistryServiceUpdateRequest { + TracePipelineConfig trace_pipeline_config = 1 [(validate.rules).message = {required: true}]; +} + +message TracePipelineRegistryServiceUpdateResponse { + int64 mod_revision = 1; +} + +message TracePipelineRegistryServiceDeleteRequest { + common.v1.Metadata metadata = 1 [(validate.rules).message = {required: true}]; +} + +message TracePipelineRegistryServiceDeleteResponse { + bool deleted = 1; + // delete_time is the server-assigned tombstone timestamp in unix nanos. + int64 delete_time = 2; + // mod_revision is the etcd revision of the tombstone; zero if the server did not record one. + int64 mod_revision = 3; +} + +message TracePipelineRegistryServiceGetRequest { + common.v1.Metadata metadata = 1 [(validate.rules).message = {required: true}]; +} + +message TracePipelineRegistryServiceGetResponse { + TracePipelineConfig trace_pipeline_config = 1; +} + +message TracePipelineRegistryServiceExistRequest { + common.v1.Metadata metadata = 1 [(validate.rules).message = {required: true}]; +} + +message TracePipelineRegistryServiceExistResponse { + bool has_group = 1; + bool has_trace_pipeline_config = 2; +} + +message TracePipelineRegistryServiceListRequest { + string group = 1 [(validate.rules).string.min_len = 1]; +} + +message TracePipelineRegistryServiceListResponse { + repeated TracePipelineConfig trace_pipeline_config = 1; +} + +// TracePipelineRegistryService manages TracePipelineConfig resources, mirroring +// the registry services of every other schema resource. Create/Update run the +// admission and conflict checks of §2.3/§2.4 of the design. +service TracePipelineRegistryService { + rpc Create(TracePipelineRegistryServiceCreateRequest) returns (TracePipelineRegistryServiceCreateResponse) { + option (google.api.http) = { + post: "/v1/trace-pipeline/schema" + body: "*" + }; + } + + rpc Update(TracePipelineRegistryServiceUpdateRequest) returns (TracePipelineRegistryServiceUpdateResponse) { + option (google.api.http) = { + put: "/v1/trace-pipeline/schema/{trace_pipeline_config.metadata.group}/{trace_pipeline_config.metadata.name}" + body: "*" + }; + } + + rpc Delete(TracePipelineRegistryServiceDeleteRequest) returns (TracePipelineRegistryServiceDeleteResponse) { + option (google.api.http) = {delete: "/v1/trace-pipeline/schema/{metadata.group}/{metadata.name}"}; + } + + rpc Get(TracePipelineRegistryServiceGetRequest) returns (TracePipelineRegistryServiceGetResponse) { + option (google.api.http) = {get: "/v1/trace-pipeline/schema/{metadata.group}/{metadata.name}"}; + } + + rpc List(TracePipelineRegistryServiceListRequest) returns (TracePipelineRegistryServiceListResponse) { + option (google.api.http) = {get: "/v1/trace-pipeline/schema/lists/{group}"}; + } + + // Exist doesn't expose an HTTP endpoint. Please use HEAD method to touch Get instead. + rpc Exist(TracePipelineRegistryServiceExistRequest) returns (TracePipelineRegistryServiceExistResponse); +} Review Comment: The PR description explicitly calls out that a TracePipelineRegistryService is intentionally NOT included, but this proto adds TracePipelineRegistryService (with HTTP mappings) and the generated docs now expose it. Please either remove the service from this PR or update the PR description to match what’s being introduced. ########## docs/design/post-trace-pipeline.md: ########## @@ -0,0 +1,542 @@ +# BanyanDB Storage-Node Post-Trace Pipeline & Storage Tier Mapping Specification + +This document presents the complete technical design for the **BanyanDB Storage-Node Post-Trace Pipeline** and its integration with physical hardware storage tiers and time-aging schedulers. Review Comment: The PR description lists an additional new file `docs/design/backlog/post-trace-scoring.md`, but that file/directory isn’t present in this change set. Please either add the deferred scoring doc as described, or update the PR description/file list so it matches the actual contents of the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
