ASF GitHub Bot commented on METRON-1657:

Github user justinleet commented on the issue:

    I ran through 
 but spinning it up as
    $METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s 
    This results in indices (noting that I'd pushed the data to the topic a 
couple times, so the numbers won't line up directly if you run it):
    [root@node1 ~]# curl -X GET "localhost:9200/_cat/indices?v"
    health status index                           uuid                   pri 
rep docs.count docs.deleted store.size pri.store.size
    yellow open   cisco-5-304_index_2018.07.11.18 z-MyPPEJSN6cur7FJbFORA   5   
1         45            0    340.8kb        340.8kb
    yellow open   cisco-6-302_index_2018.07.11.18 vAFrok4sRpW49_RYt9RqbQ   5   
1        660            0      1.4mb          1.4mb

> Parser aggregation in storm
> ---------------------------
>                 Key: METRON-1657
>                 URL: https://issues.apache.org/jira/browse/METRON-1657
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Justin Leet
>            Assignee: Justin Leet
>            Priority: Major
> Currently our parsing solution requires one storm topology per sensor. It has 
> been complained that this may be wasteful of resources and that, rather than 
> one storm topology per sensor, it would be advantageous to have multiple 
> sensors in the same topology. The benefit to this is that it would require 
> fewer storm slots.
> The issue with this is that whenever we've aggregated functionality like this 
> before, we've run into issues appropriately being able to scale storm (e.g. 
> batch vs random access indexing in the same topology).  The main point in 
> addressing this is to recommend that parsers with similar velocities and 
> complexity are grouped together.
> Particularly for a first cut, leave the configuration mostly as-is, while 
> allowing for comma separated lists of sensors in start_parser_topology.sh 
> (e.g. bro,yaf creates a aggregated parser consisting of those two). 

This message was sent by Atlassian JIRA

Reply via email to