John, Thanks so much for your interest. I feel like you have a pretty good understanding of how things fit together. The existing project structure is left over from the legacy OpenSOC implementation so it¹s not as clear and organized as it could be. We are currently working on reorganizing Metron-Streaming into projects and packages that will make it easier to navigate and understand so expect some changes. Look for a pull request to be submitted within a few weeks. We would love to hear your feedback and ideas.
In the meantime, check out the architecture documentation on the wiki if you have not already: https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture Ryan Merriman On 3/31/16, 11:38 PM, "John" <[email protected]> wrote: >Hello Dev@Metron, > >I've been thinking about getting more involved with Metron. I've already >submitted a couple very simple PRs that got approved and one is now merged >into master. The ansible and vagrant scripts have made it super easy to >spend up a 10-node setup in AWS or a local VM setup for testing. So now >I'm >diving into the Metron-Streaming modules to try and figure out what roles >each of play. I haven't dug super deep yet, so based on little I've seen, >plus the individual README's -- this is what I've gathered so far at a >high-level... > > - *Metron-Pcap_Service* : Example service that grab packets and stores > them to HBase. > - *Metron-DataServices* : How the messages(/events) get into the > pipeline. > - *Metron-MessageParsers* : Takes raw messages (which can be binary > formats) and converts them to a common format of source/destination > ip/port/protocol w/ timestamp+message. Looks like a couple of the >parsing > patterns forked from Logstash. > - *Metron-EnrichmentAdapters* : As the messages come in, extra metadata > can be added, like geo, whois, etc. So I guess the parsed message + any > enrichment adapters you have enabled would be "the model". > - *Metron-DataLoads* : How to get the enrichment data into the system. > - *Metron-Alerts* : Sends the message onto the message stream like > normal, but will also send it to the alert stream. > - *Metron-Indexing* : This is the main output of the streaming system, > which is currently Elasticsearch/Kibana(v3)Š but looks like you're in >the > middle of adding Solr support too. > - *Metron-Topologies* : To configure all this stuff to meet your needs > (ex. which telemetries you want to collect). > - *Metron-Testing* : To test this whole thing without needing servers >or > data. > - *Metron-Common* : Dev tools/packages shared across modules. > >Totally not looking for someone to blow a bunch of time on a super >detailed >response; just curious if I'm totally off based on any of these modules or >if I missed something super big. > > >Thanks! >John
