Vinish Reddy created HUDI-3373:
----------------------------------
Summary: Add zero value counter metrics because of Prometheus
PushGateway stale metrics problem
Key: HUDI-3373
URL: https://issues.apache.org/jira/browse/HUDI-3373
Project: Apache Hudi
Issue Type: Task
Reporter: Vinish Reddy
Prometheus [Pushgateway|https://prometheus.io/docs/practices/pushing/] never
forgets series pushed to it and will expose them to Prometheus forever unless
those series are manually deleted via the Pushgateway's API. When deltastreamer
finds that there are no new records to be read from the source since the last
checkpoint it doesn't emit a metric because the write path is not triggered and
leads to a graph like below in case of PushGateway metrics reporter. For other
metrics reporter it would be a null value and no data will be shown in the
graph.
{{22/02/07 11:23:10 INFO DeltaSync: No new data, source checkpoint has not
changed. Nothing to commit. Old checkpoint=(Option\{val=20220207062847034}).
New Checkpoint=(20220207062847034)}}
{{!https://user-images.githubusercontent.com/16958856/152831831-7c4baab1-8b01-4100-8b64-40ec702e9749.png!}}
To give a better representation of the actual metrics for Prometheus
PushGateway, the counter metrics have to be updated with zero value so that
it's better than having stale data which was ingested as part of the last
checkpoint.
!https://user-images.githubusercontent.com/16958856/152831933-7238666b-b6b4-4ac6-9962-399b6a4cf81b.png!
{{}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
