Andrew Purtell created HBASE-22660:
--------------------------------------
Summary: Precise end to end tracking of cross cluster replication
latency
Key: HBASE-22660
URL: https://issues.apache.org/jira/browse/HBASE-22660
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
ageOfLastShippedOp tracks replication latency forward from the point where a
source process tailing a WAL has found an edit to ship. This is not an end to
end measure.
To achieve a holistic end to end measure we should have an active process that
periodically injects sentinel values at commit time adjacent to the WALedits
carrying application data at the source and records when they are finally
processed at the sink, using a timestamp embedded in the sentinel to measure
true end to end latency for the adjacent commit. This could be done for a
configurable (and small) percentage of commits so would give a probabilistic
measure with confidence controlled by sample rate. It should be done this way
rather than by passively sampling cell timestamps because cell timestamps can
be set by the user and may not correspond to wall clock time.
We could introduce a new type of synthetic WALedit, a new global metric, and
because the adjacent commit from which we build the sentinel contains table
information we could track that too and add a per table metric.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
