GitHub user Alena0704 edited a comment on the discussion: [Ideas] Add instrumentation and latency metrics to the Anser subsystem
I've been working on something close to Change 1 (since July), but from the executor side: per-node wait timing for cross-slice blocking. My original motivation was a bit different from performance analysis — I wanted a way to catch queries that get stuck (or fail outright) because of a cross-slice bug, where the optimizer and the executor disagree about slice assignment and producer/consumer locality is broken. That was the subject of my talk at Community Over Code Asia, "When the Optimizer Lies: Debugging Cross-Slice Execution in Apache Cloudberry": a Shared Scan over a CTE on a replicated table, hidden behind a scalar SubPlan, ends up hanging or failing with temporary file errors, and from the outside there is nothing to look at. Latency instrumentation turned out to be the practical way to make those cases visible — a consumer waiting forever on a producer that will never publish looks exactly like an unbounded wait on one node. So I implemented it in the core as per-node wait statistics. In open-gpdb I added wait stats for cross-slice ShareInputScan (open-gpdb/gpdb#405): the consumer side measures how long it blocks waiting for the producer slice, and the elapsed time is reported in three ways — EXPLAIN ANALYZE (Cross-slice wait: N ms max (segK), M ms avg x P workers), the stats collector (cross_slice_wait_ms as a per-query aggregate), and the error context when a query is cancelled while blocked (e.g. by statement_timeout), which is what makes the pathological cases traceable after the fact. It also makes wait skew across segments visible: the max/avg split per worker shows whether one segment is holding everything up or the wait is uniform. The mechanism is generic enough that the same shape (per-node accumulator → max/avg per segment → EXPLAIN VERBOSE + a query-level aggregate) should fit the Anser producer/consumer waits directly. If the community is interested, I'd be happy to port it to Cloudberry — either as a general per-node wait-timing facility that Change 1 can build on, or just as input for whoever picks up Change 1. I can prepare pull request to cloudberry if the community is interested in it. GitHub link: https://github.com/apache/cloudberry/discussions/1958#discussioncomment-18302309 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
