[ https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540579#comment-16540579 ]
ASF GitHub Bot commented on METRON-1657: ---------------------------------------- Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1099 I ran through https://github.com/apache/metron/blob/master/use-cases/parser_chaining/README.md, but spinning it up as ``` $METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s cisco-6-302,cisco-5-304,pix_syslog_router ``` This results in indices (noting that I'd pushed the data to the topic a couple times, so the numbers won't line up directly if you run it): ``` [root@node1 ~]# curl -X GET "localhost:9200/_cat/indices?v" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open cisco-5-304_index_2018.07.11.18 z-MyPPEJSN6cur7FJbFORA 5 1 45 0 340.8kb 340.8kb yellow open cisco-6-302_index_2018.07.11.18 vAFrok4sRpW49_RYt9RqbQ 5 1 660 0 1.4mb 1.4mb ... ```` > Parser aggregation in storm > --------------------------- > > Key: METRON-1657 > URL: https://issues.apache.org/jira/browse/METRON-1657 > Project: Metron > Issue Type: Bug > Reporter: Justin Leet > Assignee: Justin Leet > Priority: Major > > Currently our parsing solution requires one storm topology per sensor. It has > been complained that this may be wasteful of resources and that, rather than > one storm topology per sensor, it would be advantageous to have multiple > sensors in the same topology. The benefit to this is that it would require > fewer storm slots. > The issue with this is that whenever we've aggregated functionality like this > before, we've run into issues appropriately being able to scale storm (e.g. > batch vs random access indexing in the same topology). The main point in > addressing this is to recommend that parsers with similar velocities and > complexity are grouped together. > Particularly for a first cut, leave the configuration mostly as-is, while > allowing for comma separated lists of sensors in start_parser_topology.sh > (e.g. bro,yaf creates a aggregated parser consisting of those two). -- This message was sent by Atlassian JIRA (v7.6.3#76005)