hadoopkandy opened a new pull request, #4509: URL: https://github.com/apache/flink-cdc/pull/4509
### What is the purpose of the change
This PR adds a new metric `currentBinlogPositionLag` for the MySQL binlog
reader, which measures the lag between the current consumed binlog offset and
the latest master
binlog offset.
Unlike the existing `currentFetchEventTimeLag` metric (which only updates
when there are events flowing), this metric is meaningful even during idle
periods — it reflects how
far behind the reader is from the MySQL master at the position level.
### Brief change log
- Added `BinlogLagCalculator` to compute binlog position lag, supporting
both GTID mode and file-position mode.
- In `BinlogSplitReader`, periodically (every 10s) fetch the master's
current binlog offset via `SHOW MASTER STATUS` and store it in a shared
`AtomicReference<BinlogOffset>`.
- In `MySqlRecordEmitter`, periodically (every 10s) read the shared master
offset and calculate the lag against the current consumed offset, then report
it via
`MySqlSourceReaderMetrics`.
- Registered the new gauge metric `currentBinlogPositionLag` in
`MySqlSourceReaderMetrics`.
### Lag calculation strategy
| Mode | Lag meaning | Calculation |
|------|-------------|-------------|
| GTID | Transaction count difference | Sum of (master max txn ID -
current max txn ID) per server UUID |
| File-position (same file) | Byte offset difference | master position -
current position |
| File-position (cross file) | Estimated byte difference | file sequence
diff × 1,000,000 + master position |
### Verifying this change
This change added tests:
- `BinlogLagCalculatorTest`: unit tests covering GTID mode
(single/multiple UUIDs, disjoint intervals, starts-from-middle), file-position
mode (same file, cross file, edge
cases).
- `MySqlRecordEmitterTest#testBinlogPositionLagMetricIsUpdated`: verifies
that the lag metric is updated correctly during record emission.
### Does this pull request potentially affect one of the following parts
- Dependencies: no
- The public API: no
- The serializers: no
- The runtime per-record code path: yes (lightweight gauge update during
binlog phase only)
- Anything that affects determine of shard: no
### Documentation
- [ ] Does this pull request introduce a new feature? yes — a new
monitoring metric
- [ ] If yes, how is the feature documented? JavaDoc on the metric
constant; needs user-facing docs update for the metric name and semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
