luoyuxia opened a new issue, #3464: URL: https://github.com/apache/fluss/issues/3464
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Motivation The current lake tiering bucket metric `timestampLag` is calculated from the timestamp difference between the latest local log record and the latest lake-tiered log record. This can produce a pulse-like false positive after an idle period: 1. A table writes records and lake tiering catches up. 2. The table stays idle for a while. 3. A new batch is written after the idle period. 4. `timestampLag` immediately becomes large because it includes the idle time between the previous lake-tiered record timestamp and the new local record timestamp. In this case, lake tiering is not necessarily delayed. The metric is large because new data arrived after a quiet period. This makes `timestampLag` hard to use directly for freshness alerts. A similar distinction exists in MySQL replication/binlog systems. MySQL exposes replication lag through `Seconds_Behind_Source` in `SHOW REPLICA STATUS`, and replication heartbeats can be configured with `SOURCE_HEARTBEAT_PERIOD` so an idle source can still keep the replica connection and progress observable without confusing idleness with backlog. References: - MySQL `SHOW REPLICA STATUS` and `Seconds_Behind_Source`: https://dev.mysql.com/doc/refman/8.4/en/show-replica-status.html - MySQL `SOURCE_HEARTBEAT_PERIOD`: https://dev.mysql.com/doc/refman/8.4/en/change-replication-source-to.html Fluss should expose a metric that answers the same operational question for lake tiering: if there are records pending to be tiered to lake, how long has the oldest pending committed record been waiting? ### Solution Add a new bucket-level lake tiering gauge metric: ```text pendingRecordLag ``` Suggested semantics: ```text if there are no committed records pending lake tiering: pendingRecordLag = 0 else: pendingRecordLag = current_time_ms - timestamp(oldest committed record pending lake tiering) ``` More concretely: ```text highWatermark = log high watermark firstPendingOffset = lakeLogEndOffset >= 0 ? lakeLogEndOffset : localLogStartOffset firstPendingOffset = max(firstPendingOffset, localLogStartOffset) if firstPendingOffset >= highWatermark: return 0 return max(currentTimeMillis - timestamp(record at firstPendingOffset), 0) ``` If there are pending records but the timestamp cannot be read from the local log, return `-1` to indicate that the metric is unavailable. This metric should be registered under the existing `table_bucket_lakeTiering` metric group together with `pendingRecords` and `timestampLag`. The existing `timestampLag` metric should be kept unchanged for compatibility. Users can use the new metric for freshness alerting, for example: ```text pendingRecords > 0 AND pendingRecordLag > table.datalake.freshness ``` ### Anything else? This avoids alert spikes caused by idle periods followed by new writes. The metric measures the age of the actual lake-tiering backlog instead of the timestamp distance between the latest local record and latest lake-tiered record. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
