Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202805243
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -82,6 +82,12 @@ topology in kafka. Errors are collected with the
context of the error
(e.g. stacktrace) and original message causing the error and sent to an
`error` queue. Invalid messages as determined by global validation
functions are also treated as errors and sent to an `error` queue.
+
+Multiple sensors can be aggregated into a single Storm topology. When this
is done, there will be
+multiple Kafka spouts, but only a single parser bolt which will handle
delegating to the correct
--- End diff --
Essentially the flow in the syslog case is:
1. Syslog message comes in through the wrapper's topic.
2. The spout for syslog passes it through to the aggregate bolt.
3. The aggregated bolt does its job unwrapping and outputs the inner
messages to the appropriate topics for each.
4. The spouts for the inner messages then pick up from the now populated
topics.
5. All spouts are now passing messages as they receive them to the
aggregate bolt.
6. All messages get delegated and handled as needed.
This implementation doesn't do anything clever around avoiding output
topics if the aggregate parser is already contained here.
---
