Thanks for doing the work, Otto. We'll take a look 01.10.2016, 21:00, "Otto Fowler" <[email protected]>: > I have been able to add a new parser to the the deployment, and have the > cluster fully deploy successfully. After I was able to push data to kafka > from HDF and get it all indexed. > > Unlike quick dev and full, no problems getting the storm ports correct > after deployment to my small cluster config. > > It looks to me that the steps I took to integrate the parser worked, but I > still may have missed something. One thing that I know I missed was > modifying the dashboard - adding the saved searches and integrating them > with the visualizations. > > Here is a gist off a patch of my changes. The patch in the gist has been > modified - so I don’t think it will apply for you. I removed proprietary > field names ( the stellar config, the enrichment hosts, es index template ). > > I hope what is there is enough for you to verify, correct what I have done. > > https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84 > > On September 27, 2016 at 13:42:51, Otto Fowler ([email protected]) > wrote: > > Thanks Nick, > That is some of the stuff that I have found trying to track down the deploy > bits of the existing parsers, but I don’t want to miss anything, so I’d > like some guidence. Right now, I’m OK with doing it all inside the metron > ansible base. I expect that once I get it working and wrap my head around > it I’ll have some ideas that I’ll float around improving this area, some > use cases to propose that would possibly be external to the main > deployment, or additive. First thing first is understanding all the points > in doing it the hard way ;) > > -- > > Sent with Airmail > > On September 27, 2016 at 12:41:10, Nick Allen ([email protected]) wrote: > > Hi Otto - > > I would agree with you. We do not have documentation that describes how to > 'permanently install' a new parser.. Your contribution would be highly > appreciated in this area. > > With the Ansible-based deployment of today, most likely you will have to > touch some of Metron's Ansible source code. An alternative would be to > mimic portions of Metron's deployment code, and manage that in its own > project, which would deploy your new parser. But of course, if we can find > ways to make this task easier, we will. > > You may not have to touch each of these areas, but they at least will > provide you with a better understanding of how everything is stitched > together. > > *Monit *- The Monit integration lives in `metron-deployment/roles/monit`. > You can follow the pattern of > metron-deployment/roles/monit/templates/monit/parsers.monit to add your own > parser definition to Monit. > > *Parsers* - The start script in > `metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh` > will give you good hooks into how each of the parsers are started. > > *Setup* - There are various setup tasks for the streaming functionality > that live under `metron-deployment/roles/metron_streaming`. To understand > that process, start at `tasks/main.yml`. > > I probably missed something, but let me know if you have questions. > > On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <[email protected]> > wrote: > >> My wish, is that when I do an ansible-playbook -v -i {my configuration} >> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that >> my parser / topology is deployed, started and monitored the same way as > > the >> current bro, snort, and yaf parsers are. >> >> I might be misunderstanding something however. I seems to me that all the >> examples of adding other parsers are temporary and not permanent because >> they do not have the full deployment, kind of push the config and run the >> script and you are going. Am I missing something? Would the squid sample >> steps result in a parser topology that would survive restarts / reboots >> etc? >> >> On September 27, 2016 at 12:06:44, James Sirota ([email protected]) >> wrote: >> >> Just so I completely understand what you are asking for...you want to know >> how to create a new parser topology with the JSON parser and plug it into >> Monit so you can monitor and restart it on demand? >> >> 27.09.2016, 09:03, "Otto Fowler" <[email protected]>: >> > Thanks James, >> > >> > I want to deploy an instance of the JSONMapParser into my POC cluster > > and >> vagrant. I’m trying to work out exactly how to add a new configured parser >> instance to the deployment. I think these instructions would be a good >> extension to the squid stuff that is already there. If I could get that >> going and add a new parser all the way through, then maybe I can > > contribute >> something in that area. The ability to do this will also enable some of >> the other work you mentioned. >> > >> > On September 27, 2016 at 11:51:41, James Sirota ([email protected]) >> wrote: >> > >> >> There are three types of parsers you can have currently. Our preferred >> way is to use Grok parser. The only thing you need to do there is to > > define >> your Grok statement and the parser will uptake it and do the rest. That is >> what most of our documentation reflect. The second type of parser that we >> have is a java parser, where you actually extend a parser class and define >> your own custom parsing logic. We intend this type of parser for high >> velocity feeds that require custom parsing logic that is not easily >> attainable by Grok. The third type of parser is the one you have been >> working on, a Json parser. This is a parser designed to take pre-parsed >> JSON for sensors that either log in JSON format natively or have been >> pre-parsed for us by some system upstream. >> >> >> >> Parsers don't integrate with Monit by default. We can come up with some >> instructions for you on how to do that. >> >> >> >> I should also note there are 2 additional parser types that are on the >> road map. METRON-295 (scripting bolt), which is a parser that allows you > > to >> uptake something like javascript, lua, etc., for doing the parsing. There >> is also METRON-288, which is a XSL parser designed to parse XML documents. >> If either of these are of interest to you we would welcome this >> contribution and we can work with you to get you started. >> >> >> >> 26.09.2016, 10:35, "Otto Fowler" <[email protected]>: >> >>> Are all the steps required to add a parser documented anywhere? The >> squid >> >>> document starts the topology, but I don’t think that integrates it in >> with >> >>> monit for example. Or does that actually happen? >> >> >> >> ------------------- >> >> Thank you, >> >> >> >> James Sirota >> >> PPMC- Apache Metron (Incubating) >> >> jsirota AT apache DOT org >> >> ------------------- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org > > -- > Nick Allen <[email protected]>
------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
