Your changes to add the 'laa' sensor look "sensible." Nice. On Sat, Oct 1, 2016 at 11:59 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:
> I have been able to add a new parser to the the deployment, and have the > cluster fully deploy successfully. After I was able to push data to kafka > from HDF and get it all indexed. > > Unlike quick dev and full, no problems getting the storm ports correct > after deployment to my small cluster config. > > It looks to me that the steps I took to integrate the parser worked, but I > still may have missed something. One thing that I know I missed was > modifying the dashboard - adding the saved searches and integrating them > with the visualizations. > > Here is a gist off a patch of my changes. The patch in the gist has been > modified - so I don’t think it will apply for you. I removed proprietary > field names ( the stellar config, the enrichment hosts, es index template ). > > I hope what is there is enough for you to verify, correct what I have done. > > https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84 > > > > On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Thanks Nick, > That is some of the stuff that I have found trying to track down the > deploy bits of the existing parsers, but I don’t want to miss anything, so > I’d like some guidence. Right now, I’m OK with doing it all inside the > metron ansible base. I expect that once I get it working and wrap my head > around it I’ll have some ideas that I’ll float around improving this area, > some use cases to propose that would possibly be external to the main > deployment, or additive. First thing first is understanding all the points > in doing it the hard way ;) > > > -- > > Sent with Airmail > > On September 27, 2016 at 12:41:10, Nick Allen (n...@nickallen.org) wrote: > > Hi Otto - > > I would agree with you. We do not have documentation that describes how to > 'permanently install' a new parser.. Your contribution would be highly > appreciated in this area. > > With the Ansible-based deployment of today, most likely you will have to > touch some of Metron's Ansible source code. An alternative would be to > mimic portions of Metron's deployment code, and manage that in its own > project, which would deploy your new parser. But of course, if we can find > ways to make this task easier, we will. > > You may not have to touch each of these areas, but they at least will > provide you with a better understanding of how everything is stitched > together. > > *Monit *- The Monit integration lives in `metron-deployment/roles/monit`. > You can follow the pattern of > metron-deployment/roles/monit/templates/monit/parsers.monit to add your > own > parser definition to Monit. > > *Parsers* - The start script in > `metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh` > will give you good hooks into how each of the parsers are started. > > *Setup* - There are various setup tasks for the streaming functionality > that live under `metron-deployment/roles/metron_streaming`. To understand > that process, start at `tasks/main.yml`. > > I probably missed something, but let me know if you have questions. > > > > > On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > My wish, is that when I do an ansible-playbook -v -i {my configuration} > > metron_full_install.yml to my cluster - or do the full_dev-> vagrant that > > my parser / topology is deployed, started and monitored the same way as > the > > current bro, snort, and yaf parsers are. > > > > I might be misunderstanding something however. I seems to me that all the > > examples of adding other parsers are temporary and not permanent because > > they do not have the full deployment, kind of push the config and run the > > script and you are going. Am I missing something? Would the squid sample > > steps result in a parser topology that would survive restarts / reboots > > etc? > > > > On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org) > > wrote: > > > > Just so I completely understand what you are asking for...you want to > know > > how to create a new parser topology with the JSON parser and plug it into > > Monit so you can monitor and restart it on demand? > > > > 27.09.2016, 09:03, "Otto Fowler" <ottobackwa...@gmail.com>: > > > Thanks James, > > > > > > I want to deploy an instance of the JSONMapParser into my POC cluster > and > > vagrant. I’m trying to work out exactly how to add a new configured > parser > > instance to the deployment. I think these instructions would be a good > > extension to the squid stuff that is already there. If I could get that > > going and add a new parser all the way through, then maybe I can > contribute > > something in that area. The ability to do this will also enable some of > > the other work you mentioned. > > > > > > On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org) > > wrote: > > > > > >> There are three types of parsers you can have currently. Our preferred > > way is to use Grok parser. The only thing you need to do there is to > define > > your Grok statement and the parser will uptake it and do the rest. That > is > > what most of our documentation reflect. The second type of parser that we > > have is a java parser, where you actually extend a parser class and > define > > your own custom parsing logic. We intend this type of parser for high > > velocity feeds that require custom parsing logic that is not easily > > attainable by Grok. The third type of parser is the one you have been > > working on, a Json parser. This is a parser designed to take pre-parsed > > JSON for sensors that either log in JSON format natively or have been > > pre-parsed for us by some system upstream. > > >> > > >> Parsers don't integrate with Monit by default. We can come up with > some > > instructions for you on how to do that. > > >> > > >> I should also note there are 2 additional parser types that are on the > > road map. METRON-295 (scripting bolt), which is a parser that allows you > to > > uptake something like javascript, lua, etc., for doing the parsing. There > > is also METRON-288, which is a XSL parser designed to parse XML > documents. > > If either of these are of interest to you we would welcome this > > contribution and we can work with you to get you started. > > >> > > >> 26.09.2016, 10:35, "Otto Fowler" <ottobackwa...@gmail.com>: > > >>> Are all the steps required to add a parser documented anywhere? The > > squid > > >>> document starts the topology, but I don’t think that integrates it in > > with > > >>> monit for example. Or does that actually happen? > > >> > > >> ------------------- > > >> Thank you, > > >> > > >> James Sirota > > >> PPMC- Apache Metron (Incubating) > > >> jsirota AT apache DOT org > > > > ------------------- > > Thank you, > > > > James Sirota > > PPMC- Apache Metron (Incubating) > > jsirota AT apache DOT org > > > > > > -- > Nick Allen <n...@nickallen.org> > > -- Nick Allen <n...@nickallen.org>
