lokiore opened a new pull request, #2609: URL: https://github.com/apache/phoenix/pull/2609
### What changes were proposed in this pull request? This PR makes the ZK-less HA client's HA/CRR/failover metrics sliceable per HA group, and fills gaps in the client-side HA metric coverage. It builds on (and supersedes) #2605, which is folded in here as its own PHOENIX-7872 commit. **1. Per-HA-group metrics tag.** A new per-group Hadoop Metrics2 source, `HAGroupClientMetricsSource`, registers one metrics2 source per HA group. Each instance: - appends `,haGroup=<ObjectName.quote(name)>` to its JMX context so every group registers as a distinct source / MBean, and - stamps a `haGroup` tag carrying the (unquoted) group name so the series can be sliced per HA group downstream. `HAGroupMetricsManager` is a small registry (one source per group name) wired into `HighAvailabilityGroup` lifecycle: a source is created on group init and detached on group close. Emission is **dual**: the existing JVM-global `GLOBAL_HA_*` counters continue to emit unchanged, and the same event is additionally recorded on the per-group source. The tag key lives in a new module-neutral `org.apache.phoenix.metrics.MetricConstants` (`HA_GROUP_TAG_NAME = "haGroup"`), referenced by both the new client source and the server-side `HAGroupStoreMetricsSource` so both sides share one tag key. The per-group set is intentionally limited to metrics attributable to a single HA group (each emission site has a `HighAvailabilityGroup` in scope). The JVM-shared parallel-executor pool metrics and the `HA_CRR_CACHE_AGE_MS` gauge are excluded. **2. Missing client counters.** New `MetricType`s and their `GLOBAL_HA_*` wrappers, emitted (dual) at the relevant client sites: - `HA_FAILOVER_CONNECTION_CREATED_COUNTER` — a `FailoverPhoenixConnection` was successfully created against the active cluster. - `HA_FAILOVER_CONNECTION_FAILED_COUNTER` — a connect-to-active attempt threw (no active cluster, demoted mid-connect, or connect error). *(from #2605 / PHOENIX-7872)* - `HA_ROLE_TRANSITION_FAILED_COUNTER` — a cluster-role-transition dispatch failed while applying a new CRR. - `CRR_TRANSITION_COUNT` — a CRR transition was applied per HA policy, including transitions into a no-active state (distinct from `HA_FAILOVER_COUNT`, which counts only transitions that establish/move an ACTIVE cluster). Existing counters (`HA_FAILOVER_COUNT`, `HA_FAILOVER_DURATION_MS`, `HA_STALE_CRR_DETECTED_COUNT`, `HA_MUTATION_BLOCKED_COUNT`, `HA_CRR_REFRESH_COUNT`, the `HA_PARALLEL_*` connection/task counters, and the poller-tick counters) are additionally recorded per-group at their existing emission sites. The #2605 change (record `HA_FAILOVER_DURATION_MS` on the CRR-write path rather than the dead `failover()` path, and add the connection-failed counter) is preserved as its own PHOENIX-7872-attributed commit at the base of this branch; #2605 will be closed as superseded by this PR. ### Why are the changes needed? On the ZK-less HA client the JVM-global `GLOBAL_HA_*` counters aggregate across every HA group in the process, so a JVM serving more than one HA group cannot attribute failover / stale-CRR / mutation-blocked / transition activity to a specific group. Tagging each series with `haGroup` makes per-group dashboards and alerting possible while keeping the existing global counters intact for backward compatibility. The additional counters close observability gaps around connection creation/failure and role-transition outcomes that had no client-side metric. ### Does this PR introduce _any_ user-facing change? No behavioral change. New client-side metrics are added (new `MetricType` entries and their `GLOBAL_HA_*` wrappers) and a new per-group `haGroup`-tagged metrics2 source is registered; the existing global HA counters are unchanged. No SQL, API, or wire-format change. ### How was this patch tested? New unit tests (run under surefire): - `HAGroupClientMetricsSourceTest` — the `haGroup` tag carries the unquoted group name; the JMX context is quoted per group; increment/update are per-counter; unknown metric types are ignored; each group is a distinct registered source; unregister frees the source name for reuse. - `HAGroupMetricsManagerTest` — `getOrCreate` is idempotent and registers a source; null/empty group names are a no-op; two groups never cross-count; each group gets a distinct tagged source; `update` accumulates per group; `remove` detaches the source; re-create after remove rebuilds a fresh source. - `HighAvailabilityGroupTest` additions cover the CRR-write-path duration/connection-failed emission (from #2605 / PHOENIX-7872). `mvn spotless:apply` was run before pushing; the full module build compiles clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
