palashc opened a new pull request, #2506: URL: https://github.com/apache/phoenix/pull/2506
### What changes were proposed in this pull request? Decouples the `cdcIndexUpdateLag` metric from batch completion. A new package-private helper `IndexCDCConsumerProgress` holds the consumer's monotonic effective freshness watermark, advanced by: - a successful own-partition batch (sets `lastProcessedTimestamp`), or - an empty own-partition CDC poll (proves caught-up to `pollEnd − timestampBufferMs`). A new `sleepWithLagSampling(...)` helper in `IndexCDCConsumer` replaces every consumer-thread `Thread.sleep` site (run loop, startup wait, parent-progress wait, failure backoff). It chunks every sleep into `lagSampleIntervalMs` slices (default 1000ms, configurable via new property `phoenix.index.cdc.consumer.lag.sample.interval.ms`) and emits one `cdcIndexUpdateLag` sample per slice. The previous per-batch `updateCdcLag` calls are removed so the sampler is the single emitter, and parent-replay timestamps no longer pollute the per-table histogram. Files changed: - `phoenix-core-server/.../IndexCDCConsumer.java` — modified - `phoenix-core-server/.../IndexCDCConsumerProgress.java` — new - `phoenix-core/src/test/.../IndexCDCConsumerProgressTest.java` — new (9 unit tests) - `phoenix-core/src/it/.../IndexCDCConsumerLagIT.java` — new ### Why are the changes needed? `cdcIndexUpdateLag` is the primary freshness SLO signal for eventually consistent secondary indexes, but pre-fix it only emits inside `if (!batchMutations.isEmpty())` blocks (`IndexCDCConsumer.java:988` and `:1109`). This produces three distinct bugs: 1. **Silent during idle / sustained failure / startup.** Histograms have no new samples; existing percentiles age out and dashboards look healthy while the EC index is silently falling behind. 2. **Silent during post-split parent replay.** While `replayAndCompleteParentRegions(...)` runs (which can take hours), the child region's own writes accumulate unprocessed but the lag metric reports nothing. 3. **Mis-attribution.** The per-batch emit also fires from `processPartitionToCompletion` during replay, so ancestor-partition timestamps pollute the per-data-table histogram with stale values. Net effect: freshness alerts produce false negatives during the operational windows that matter most. ### Does this PR introduce _any_ user-facing change? **Yes** — `cdcIndexUpdateLag` behaviour changes (no name/shape change; histogram, same per-table fanout). | Scenario | Before | After | |---|---|---| | Active traffic | Sample per batch | Sample per ~1s (configurable) | | Idle table | **Silent** | Continuous samples, value ≈ `timestampBufferMs` baseline | | Sustained failures | **Silent** | Continuous, value growing — visibly stuck | | Parent-region replay | **Silent + polluted with ancestor timestamps** | Continuous, value reflects child-region wall-clock lag | | Cold start | **Silent until first batch** | Reports `now − consumerStartTime` | One operational note for release notes: the metric now legitimately grows during post-split parent replay (the EC index *is* stale during that window). Alerts on `cdcIndexUpdateLag` should use a long-enough window (e.g. > 30 min) until a follow-up subtask ships a `cdcParentReplayActiveGauge` for suppression. New config: `phoenix.index.cdc.consumer.lag.sample.interval.ms` (default 1000, floor 50). ### How was this patch tested? **Unit tests** — `IndexCDCConsumerProgressTest`, 9 deterministic tests, all passing. Covers cold-start floor, monotonicity of both signals, empty-poll watermark math, processed-vs-empty interaction, idle bounded growth, negative-lag clamp, and pre-buffer empty-poll edge case. **Integration test** — `IndexCDCConsumerLagIT`, 1 test, passing in ~11s. Verifies on a real MiniCluster that the sampler keeps emitting samples during a 5s idle window (binary flow check — pre-fix delta would be 0). Uses `awaitMinCount(1, 120s)` rather than fixed sleeps for startup so the test is robust to slow CI / GC jitter. Numerical value-correctness is left to the unit tests, which are deterministic. **Regression coverage on existing EC index ITs:** - `MultiTenantEventualIndexIT#testBasicMultiTenantEventualIndex` — PASS - `MultiTenantEventualIndexGenerateIT#testBasicMultiTenantEventualIndex` — PASS (covers the `processCDCBatchGenerated` path) - `ConcurrentMutationsCoveredEventualIT` — 3/3 PASS (exercises sibling metrics `cdcBatchProcessTime` / `cdcBatchCount` / `cdcMutationCount` that are deliberately preserved in the same blocks) ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude (Anthropic) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
