Lokesh Khurana created PHOENIX-7995:
---------------------------------------
Summary: Tag client-side HA metrics with the HA group and add
missing HA/CRR/failover client metrics on the ZK-less HA client
Key: PHOENIX-7995
URL: https://issues.apache.org/jira/browse/PHOENIX-7995
Project: Phoenix
Issue Type: Sub-task
Reporter: Lokesh Khurana
Assignee: Lokesh Khurana
Client-side HA/CRR/failover metrics are currently emitted JVM-global only, so a
JVM connected to more than one HA group cannot attribute a metric to the group
that produced it. The server side already solves this with an ha_group metrics2
tag on the HAGroupStore / replication-log sources (apache/phoenix #2606). This
ticket brings the same treatment to the client, and fills in client-side HA
observability that the ZK-less HA client is currently missing.
h2. Part 1 – Per-HA-group tagging
Emit the group-attributable client HA metrics through a per-group Hadoop
Metrics2 source that stamps an ha_group tag (value = the HA group name), so the
series can be sliced per group – mirroring the server-side ha_group tag
mechanism. The existing JVM-global counters remain unchanged (dual-emit); a
per-group source registers under a distinct JMX context/MBean and is detached
on HA-group close.
h2. Part 2 – Additional client-side HA/CRR/failover metrics
Add the following client counters, applying only those relevant to a ZK-less
client:
- Failover connections: created, failed.
- CRR lifecycle: transition count, transition duration (ms),
stale-CRR-detected,
CRR-deserialize-failure, CRR-refresh/notification count. On the ZK-less client
these are driven by the master-endpoint CRR fetch / reconcile path rather than
a ZooKeeper node-cache notification, so the emission points differ from the
ZK-based client – semantics adapted accordingly.
- Role-transition failed.
- Parallel-connection policy: fallback, created, error, task-timeout.
- CRR-missing-fallback (JVM-global; fires on the static path before any group
instance exists).
h2. Notes
- These observability gaps were identified while adding the equivalent metrics
to the ZooKeeper-based HA client; this ticket ports the applicable subset to
the ZK-less client.
- A shared tag-name constant should live in a module-neutral metrics package
so client and server reference the same ha_group tag key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
