[
https://issues.apache.org/jira/browse/METRON-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jasper Knulst updated METRON-1213:
----------------------------------
Description:
The insight that is really missing in Metron is an overview of Kafka topic
sizes and flow metrics (topic influx and outflux)
It is hard to answer questions like:
-Is the enrichments topic (lag) growing over time?
-What parser topics feed into enrichments, at what rate?
-Is the enrichment topology keeping up with the projected influx from all
parsers?
-Same questions, but then for indexing topic
When the capacity of the chained topologies is not aligned bad things can
happen, like a serious threat not reaching ES/Kibana fast enough to react upon.
Metron can only be a realtime alerter if there is no congestion along the way.
This would take merging metrics from various Metron moving parts like Storm
API, kafka-offset-checker (state/snapshots and flow metrics units/sec)
was:
The insight that is really missing in Metron is an overview of Kafka topic
sizes and flow metrics (topic influx and outflux)
It is hard to answer questions like:
-Is the enrichments topic growing over time?
-What parser topics feed into enrichments, at what capacity?
-Is the enrichment topology keeping up with the projected influx from parsers?
-Same questions, but then for indexing topic
When the capacity of the chained topologies is not aligned bad thing can
happen, like a serious threat not reaching ES fast enough to react upon. Metron
can only be a realtime alerter if there is no congestion along the way.
This would take merging metrics from various Metron moving parts like Storm
API, kafka-offset-checker (state/snapshots and flow metrics units/sec)
> Flow diagram for Metron Kafka topics
> ------------------------------------
>
> Key: METRON-1213
> URL: https://issues.apache.org/jira/browse/METRON-1213
> Project: Metron
> Issue Type: New Feature
> Affects Versions: 0.4.1
> Environment: Management UI
> Reporter: Jasper Knulst
> Fix For: Next + 1
>
>
> The insight that is really missing in Metron is an overview of Kafka topic
> sizes and flow metrics (topic influx and outflux)
> It is hard to answer questions like:
> -Is the enrichments topic (lag) growing over time?
> -What parser topics feed into enrichments, at what rate?
> -Is the enrichment topology keeping up with the projected influx from all
> parsers?
> -Same questions, but then for indexing topic
> When the capacity of the chained topologies is not aligned bad things can
> happen, like a serious threat not reaching ES/Kibana fast enough to react
> upon. Metron can only be a realtime alerter if there is no congestion along
> the way.
> This would take merging metrics from various Metron moving parts like Storm
> API, kafka-offset-checker (state/snapshots and flow metrics units/sec)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
