Jasper Knulst created ATLAS-4602:
------------------------------------
Summary: Report Lag for consuming from ATLAS_HOOK
Key: ATLAS-4602
URL: https://issues.apache.org/jira/browse/ATLAS-4602
Project: Atlas
Issue Type: Improvement
Components: atlas-core
Affects Versions: 2.2.0
Reporter: Jasper Knulst
Fix For: trunk
Attachments: image-2022-05-11-17-42-12-250.png
Currently the 'Stats' webUI function shows some details about the consumption
from the ATLAS_HOOK Kafka topic where changes from Hive Metastore arrive.
!image-2022-05-11-17-42-12-250.png!
By far the most important metric is not available though; the lag the atlas
server consumer-group has in consuming Hive updates.
Monitoring the lag is very important as trust in Atlas is greatly undermined
when changes are not reflected in Atlas within seconds. I have had numerous
occasions where ATLAS_HOOK consumption was slowing down silently and atlas was
behind tens of thousands (or 2 days) worth of messages.
There should be a new metric for the lag on the stats page to quickly identify
a possible reason for slow Atlas updates
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
