Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202805609
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the
context of the error
(e.g. stacktrace) and original message causing the error and sent to an
`error` queue. Invalid messages as determined by global validation
functions are also treated as errors and sent to an `error` queue.
+
+Multiple sensors can be aggregated into a single Storm topology. When this
is done, there will be
+multiple Kafka spouts, but only a single parser bolt which will handle
delegating to the correct
--- End diff --
This PR gives us the ability to group the parsers into a single topology if
we so desire. You would still write through to kafka. So, the topology in the
example would have 3 kafka spouts:
* One for monitoring `pix_syslog_router` (the syslog parser aka the routing
parser)
* One for monitoring `cisco-5-304`
* One for monitoring `cisco-6-302`
There would be one parser bolt, though, which would handle parsing all 3
sensor types. That is the contribution of this PR, the ability to determine
the parser and filter and field transformations from the input kafka topic and
use the appropriate one to parse the messages. There is not, however, any code
here that would bypass the intermediate kafka write (e.g. from the router
topology to the individual `cisco-5-304` or `cisco-6-302` topics). That's a
current gap.
---
