[
https://issues.apache.org/jira/browse/HUDI-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinish Reddy updated HUDI-3373:
-------------------------------
Component/s: metrics
> Add zero value counter metrics because of Prometheus PushGateway stale
> metrics problem
> --------------------------------------------------------------------------------------
>
> Key: HUDI-3373
> URL: https://issues.apache.org/jira/browse/HUDI-3373
> Project: Apache Hudi
> Issue Type: Task
> Components: metrics
> Reporter: Vinish Reddy
> Priority: Minor
>
> Prometheus [Pushgateway|https://prometheus.io/docs/practices/pushing/] never
> forgets series pushed to it and will expose them to Prometheus forever unless
> those series are manually deleted via the Pushgateway's API. When
> deltastreamer finds that there are no new records to be read from the source
> since the last checkpoint it doesn't emit a metric because the write path is
> not triggered and leads to a graph like below in case of PushGateway metrics
> reporter. For other metrics reporter it would be a null value and no data
> will be shown in the graph.
> {{22/02/07 11:23:10 INFO DeltaSync: No new data, source checkpoint has not
> changed. Nothing to commit. Old checkpoint=(Option\{val=20220207062847034}).
> New Checkpoint=(20220207062847034)}}
> {{!https://user-images.githubusercontent.com/16958856/152831831-7c4baab1-8b01-4100-8b64-40ec702e9749.png!}}
>
> To give a better representation of the actual metrics for Prometheus
> PushGateway, the counter metrics have to be updated with zero value so that
> it's better than having stale data which was ingested as part of the last
> checkpoint.
> !https://user-images.githubusercontent.com/16958856/152831933-7238666b-b6b4-4ac6-9962-399b6a4cf81b.png!
>
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
