[
https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547785#comment-16547785
]
ASF GitHub Bot commented on METRON-1657:
----------------------------------------
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1099
Happy to help explain it, and it's really helpful to get feedback on it
because it led to improved docs, so hopefully it makes sure things are clearer
to anyone looking at this later.
I created tickets for:
- [Management UI changes for sensor
aggregation](https://issues.apache.org/jira/browse/METRON-1678)
- [REST changes to support parser
aggregation](https://issues.apache.org/jira/browse/METRON-1679)
- [Allow the option for intermediate kafka topics to be removed in
aggregated sensors](https://issues.apache.org/jira/browse/METRON-1682)
- [Allow Ambari to accept quoted parser
groups](https://issues.apache.org/jira/browse/METRON-1680)
- [Decouple the ParserBolt from the Parse execution
logic](https://issues.apache.org/jira/browse/METRON-1681)
I held off on creating a ticket for the routing process applying a
transform. @ottobackwards Would you mind creating that, to make sure the use
case and examples or assumption, etc. are what you expect?
> Parser aggregation in storm
> ---------------------------
>
> Key: METRON-1657
> URL: https://issues.apache.org/jira/browse/METRON-1657
> Project: Metron
> Issue Type: Bug
> Reporter: Justin Leet
> Assignee: Justin Leet
> Priority: Major
>
> Currently our parsing solution requires one storm topology per sensor. It has
> been complained that this may be wasteful of resources and that, rather than
> one storm topology per sensor, it would be advantageous to have multiple
> sensors in the same topology. The benefit to this is that it would require
> fewer storm slots.
> The issue with this is that whenever we've aggregated functionality like this
> before, we've run into issues appropriately being able to scale storm (e.g.
> batch vs random access indexing in the same topology). The main point in
> addressing this is to recommend that parsers with similar velocities and
> complexity are grouped together.
> Particularly for a first cut, leave the configuration mostly as-is, while
> allowing for comma separated lists of sensors in start_parser_topology.sh
> (e.g. bro,yaf creates a aggregated parser consisting of those two).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
