hanahmily opened a new pull request, #1190: URL: https://github.com/apache/skywalking-banyandb/pull/1190
## Summary Three layered vectorized-trace-query performance improvements (one commit each). The headline: with the native columnar wire frame, the vectorized engine now **beats** the row engine on the distributed `trace_tag_filter` path across the full cardinality sweep, where it was previously at parity-or-worse. ## Changes 1. **`perf(trace)`: push offset+limit into the vectorized SIDX scan** — the ordered SIDX (tag-filter) path built Phase-1 with no trace cap, so it carried forward and materialized span data for *every* matching trace before the outer `traceLimit` truncated to the limit (O(matches) vs the row path's O(limit)). It now passes `maxRows` (= offset+limit) to `BuildMergePhase1`, stopping after the top-N distinct trace IDs in sorted-merge order (via the existing `ErrLimitExhausted`). This removed a 2.3–3.6× `trace_tag_filter` regression vs row at 1M–2M and aligns the SIDX path with the static traceID path, which already truncated. 2. **`perf(sidx)`: stop the sync query scan at MaxBatchSize distinct results** — `QuerySync` (the vec trace path's sole caller) scanned and decoded every matching block, while the streaming path stops early via consumer backpressure. `processSyncLoop` now bounds on distinct results (`mergeSync` dedups by data = trace ID within a batch; a cross-batch counter extends that) and stops the scan once `MaxBatchSize` distinct are collected. The sync path is vec-only, so there is no blast radius; `MaxBatchSize<=0` keeps the full scan. 3. **`perf(trace)`: native columnar wire frame for distributed query results** — the data↔liaison result hop marshalled `tracev1.InternalQueryResponse` via protobuf (~25% of query CPU at 2M `trace_tag_filter`, dominated by per-span message-slice/oneof/reflection — not byte copies). Behind the vectorized flag, the distributed data node now emits a columnar frame aligned to the vec `model.TraceResult` (no `InternalTrace`/`TagValue` materialization on the internal hop); the liaison decodes it and merges on `model.TraceResult` directly. The encode buffer is pre-sized exactly to avoid realloc churn. The frame is gated to the distributed data-node path (`distributed` flag on `traceQueryProcessor`); standalone keeps the proto body, and flag-off is byte-identical proto. ## Benchmark (distributed, Docker 4cpu/8g) `trace_tag_filter`, vec/row ratio (`<1.00x` = vec faster/lighter): | metric @ 2M | proto | native columnar | | --- | ---: | ---: | | p50 | 0.97x | **0.82x** | | QPS | 1.04x | **1.17x** | | CPU sec | 1.00x | **0.87x** | | Mallocs/query | 1.02x | **0.87x** | | Bytes/query | 1.01x | **0.93x** | vec beats row at every cardinality 1k–2M (p50 0.66–0.87x), and mallocs/row stays 0.83–0.87x across both `trace_by_id` and `trace_tag_filter` (the structural win from not materializing `InternalTrace`/`TagValue` on the internal hop). Profile: internal-hop `proto.Unmarshal` 16.5%→7.5%, `proto.Marshal` 8.6%→0.7%. ## Testing - `make check` + `make lint` — clean. - Unit: `api/data`, `banyand/internal/sidx`, `banyand/trace`, `pkg/query/logical/trace` — pass, incl. new frame round-trip / exact-pre-size / sidx early-stop tests. - Integration: `test/integration/distributed/query` **428/428** and `test/integration/standalone/query` **670/670**. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
