Edward Zhang created EAGLE-2:
--------------------------------
Summary: watch message process backlog in Eagle UI
Key: EAGLE-2
URL: https://issues.apache.org/jira/browse/EAGLE-2
Project: Eagle
Issue Type: Improvement
Environment: production
Reporter: Edward Zhang
Message latency is a key factor for Eagle to enable realtime security
monitoring. For hdfs audit log monitoring, kafka is used as datasource. So
there is always some gap between current max offset in kafka and processed
offset in eagle. The gap is the backlog which eagle should consume quickly as
much as quickly. If the gap can be sampled for every minute or 20 seconds, then
we understand if eagle is catching up or is lagging behind more.
The command to get current max offset in kafka is
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list xxxx --topic
hdfs_audit_log --time -1
and Storm-kafka spout would store processed offset in zookeeper, in the
following znode:
/consumers/hdfs_audit_log/eagle.hdfsaudit.consumer/partition_0
So technically we can get the gap and write that to eagle service then in UI we
can watch the backlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
