GitHub user leborchuk added a comment to the discussion: [Ideas] Add instrumentation and latency metrics to the Anser subsystem
> I've been working on something close to Change 1 (since July), but from the > executor side: per-node wait timing for cross-slice blocking. > > My original motivation was a bit different from performance analysis — I > wanted a way to catch queries that get stuck (or fail outright) because of a > cross-slice bug, where the optimizer and the executor disagree about slice > assignment and producer/consumer locality is broken. That was the subject of > my talk at Community Over Code Asia, "When the Optimizer Lies: Debugging > Cross-Slice Execution in Apache Cloudberry": a Shared Scan over a CTE on a > replicated table, hidden behind a scalar SubPlan, ends up hanging or failing > with temporary file errors, and from the outside there is nothing to look at. > Latency instrumentation turned out to be the practical way to make those > cases visible — a consumer waiting forever on a producer that will never > publish looks exactly like an unbounded wait on one node. > > So I implemented it in the core as per-node wait statistics. In open-gpdb I > added wait stats for cross-slice ShareInputScan > ([open-gpdb/gpdb#405](https://github.com/open-gpdb/gpdb/pull/405)): the > consumer side measures how long it blocks waiting for the producer slice, and > the elapsed time is reported in three ways — EXPLAIN ANALYZE (Cross-slice > wait: N ms max (segK), M ms avg x P workers), the stats collector > (cross_slice_wait_ms as a per-query aggregate), and the error context when a > query is cancelled while blocked (e.g. by statement_timeout), which is what > makes the pathological cases traceable after the fact. > > It also makes wait skew across segments visible: the max/avg split per worker > shows whether one segment is holding everything up or the wait is uniform. > > The mechanism is generic enough that the same shape (per-node accumulator → > max/avg per segment → EXPLAIN VERBOSE + a query-level aggregate) should fit > the Anser producer/consumer waits directly. If the community is interested, > I'd be happy to port it to Cloudberry — either as a general per-node > wait-timing facility that Change 1 can build on, or just as input for whoever > picks up Change 1. I can prepare pull request to cloudberry if the community > is interested in it. Thank you, it will be quite usefull. I do not think it should be part of Anser - it's more general data, needed to DBA regardless of whether Anser is enabled or not. You should definitely finish your work and let collect this statistics. GitHub link: https://github.com/apache/cloudberry/discussions/1958#discussioncomment-18333382 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
