A few other points of interest... *Sample PCAP* - The sample pcap data that is replayed is installed by the 'sensor-test-mode' that lives at `metron-deployment/roles/sensor-test-mode`. If you don't want to use that, just don't turn off 'sensor-test-mode'.
*ES Templates* - These live under `metron-deployments/roles/metron_elasticsearch_templates`. Imagine that. :) On Tue, Sep 27, 2016 at 12:31 PM, Nick Allen <[email protected]> wrote: > Hi Otto - > > I would agree with you. We do not have documentation that describes how > to 'permanently install' a new parser.. Your contribution would be highly > appreciated in this area. > > With the Ansible-based deployment of today, most likely you will have to > touch some of Metron's Ansible source code. An alternative would be to > mimic portions of Metron's deployment code, and manage that in its own > project, which would deploy your new parser. But of course, if we can find > ways to make this task easier, we will. > > You may not have to touch each of these areas, but they at least will > provide you with a better understanding of how everything is stitched > together. > > *Monit *- The Monit integration lives in `metron-deployment/roles/monit`. > You can follow the pattern of > metron-deployment/roles/monit/templates/monit/parsers.monit > to add your own parser definition to Monit. > > *Parsers* - The start script in `metron-platform/metron- > parsers/src/main/scripts/start_parser_topology.sh` will give you good > hooks into how each of the parsers are started. > > *Setup* - There are various setup tasks for the streaming functionality > that live under `metron-deployment/roles/metron_streaming`. To > understand that process, start at `tasks/main.yml`. > > I probably missed something, but let me know if you have questions. > > > > > On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <[email protected]> > wrote: > >> My wish, is that when I do an ansible-playbook -v -i {my configuration} >> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that >> my parser / topology is deployed, started and monitored the same way as >> the >> current bro, snort, and yaf parsers are. >> >> I might be misunderstanding something however. I seems to me that all the >> examples of adding other parsers are temporary and not permanent because >> they do not have the full deployment, kind of push the config and run the >> script and you are going. Am I missing something? Would the squid sample >> steps result in a parser topology that would survive restarts / reboots >> etc? >> >> On September 27, 2016 at 12:06:44, James Sirota ([email protected]) >> wrote: >> >> Just so I completely understand what you are asking for...you want to know >> how to create a new parser topology with the JSON parser and plug it into >> Monit so you can monitor and restart it on demand? >> >> 27.09.2016, 09:03, "Otto Fowler" <[email protected]>: >> > Thanks James, >> > >> > I want to deploy an instance of the JSONMapParser into my POC cluster >> and >> vagrant. I’m trying to work out exactly how to add a new configured >> parser >> instance to the deployment. I think these instructions would be a good >> extension to the squid stuff that is already there. If I could get that >> going and add a new parser all the way through, then maybe I can >> contribute >> something in that area. The ability to do this will also enable some of >> the other work you mentioned. >> > >> > On September 27, 2016 at 11:51:41, James Sirota ([email protected]) >> wrote: >> > >> >> There are three types of parsers you can have currently. Our preferred >> way is to use Grok parser. The only thing you need to do there is to >> define >> your Grok statement and the parser will uptake it and do the rest. That is >> what most of our documentation reflect. The second type of parser that we >> have is a java parser, where you actually extend a parser class and define >> your own custom parsing logic. We intend this type of parser for high >> velocity feeds that require custom parsing logic that is not easily >> attainable by Grok. The third type of parser is the one you have been >> working on, a Json parser. This is a parser designed to take pre-parsed >> JSON for sensors that either log in JSON format natively or have been >> pre-parsed for us by some system upstream. >> >> >> >> Parsers don't integrate with Monit by default. We can come up with some >> instructions for you on how to do that. >> >> >> >> I should also note there are 2 additional parser types that are on the >> road map. METRON-295 (scripting bolt), which is a parser that allows you >> to >> uptake something like javascript, lua, etc., for doing the parsing. There >> is also METRON-288, which is a XSL parser designed to parse XML documents. >> If either of these are of interest to you we would welcome this >> contribution and we can work with you to get you started. >> >> >> >> 26.09.2016, 10:35, "Otto Fowler" <[email protected]>: >> >>> Are all the steps required to add a parser documented anywhere? The >> squid >> >>> document starts the topology, but I don’t think that integrates it in >> with >> >>> monit for example. Or does that actually happen? >> >> >> >> ------------------- >> >> Thank you, >> >> >> >> James Sirota >> >> PPMC- Apache Metron (Incubating) >> >> jsirota AT apache DOT org >> >> ------------------- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org >> > > > > -- > Nick Allen <[email protected]> > -- Nick Allen <[email protected]>
