Hello Dev@Metron, I've been thinking about getting more involved with Metron. I've already submitted a couple very simple PRs that got approved and one is now merged into master. The ansible and vagrant scripts have made it super easy to spend up a 10-node setup in AWS or a local VM setup for testing. So now I'm diving into the Metron-Streaming modules to try and figure out what roles each of play. I haven't dug super deep yet, so based on little I've seen, plus the individual README's -- this is what I've gathered so far at a high-level...
- *Metron-Pcap_Service* : Example service that grab packets and stores them to HBase. - *Metron-DataServices* : How the messages(/events) get into the pipeline. - *Metron-MessageParsers* : Takes raw messages (which can be binary formats) and converts them to a common format of source/destination ip/port/protocol w/ timestamp+message. Looks like a couple of the parsing patterns forked from Logstash. - *Metron-EnrichmentAdapters* : As the messages come in, extra metadata can be added, like geo, whois, etc. So I guess the parsed message + any enrichment adapters you have enabled would be "the model". - *Metron-DataLoads* : How to get the enrichment data into the system. - *Metron-Alerts* : Sends the message onto the message stream like normal, but will also send it to the alert stream. - *Metron-Indexing* : This is the main output of the streaming system, which is currently Elasticsearch/Kibana(v3)… but looks like you're in the middle of adding Solr support too. - *Metron-Topologies* : To configure all this stuff to meet your needs (ex. which telemetries you want to collect). - *Metron-Testing* : To test this whole thing without needing servers or data. - *Metron-Common* : Dev tools/packages shared across modules. Totally not looking for someone to blow a bunch of time on a super detailed response; just curious if I'm totally off based on any of these modules or if I missed something super big. Thanks! John
