Yes, METRON-777 is the first step to resolving that. The follow on, that I’ll be putting up as a follow up soon is METRON-942 ( Rest interface for installing, querying and uninstalling extensions ). As of METRON-777, every parser is a plugin ( or extension as I am calling them ) and has been refactored as such. The only parsers that are not extensions, are the root type parsers ( JSON, CSV, GROK ).
When both of them are landed we will be able to: Use an archetype to create a new parser extension. That archetype will include: * Sample parser * Configurations for indexing, enrichment, parser * Configuration for elasticsearch * grok patterns dir if you choose. * unit and integration tests for the sample to expand on You can have multiple parsers in an extension library. You can delete the java for the parser, and keep the tests ( if you look at Yaf in METRON-777 you can see ) and have a configuration only parser, but still have tests. The product of building the archetype produced project is a tar.gz assembly, that contains the configurations and the ‘bundle’, which is the code/jar package. Using the rest-api ( swagger let’s say ) - You can then install the extension, and all the bits will be played out. You can then use the rest api to create the kafka topic, the storm job etc and run the parsers. You can edit the configuration that are installed after that using the rest api ( the management ui may need some work as a follow on ). The elasticsearch templates are installed with the extension configuration in ZK, the idea being that down the road we will allow editing and deployment of those configurations. At the moment though ( or in the future when this lands initially ) the installation of elasticsearch templates is not handled. I am trying to set up what will be required to have a Install -> configure -> deploy flow. Also, the rest api does not talk to ambari, so if you want your parser to survive a reboot etc and start, you need to go into the ambari configuration and add it to the parsers list. ( if you don’t want to use rest, you could install the extension assembly, then go into ambari and add it to the list and restart everything as well ). There is a lot to review here, and it has not been reviewed yet. I expect there to be a lot of great feedback and possible changes however. There has been a few people on the list asking around this type of thing. Hopefully we are close to having it go through review and revise to get it going. Do you have any questions I may try to answer? On May 9, 2017 at 12:00:44, Christian Tramnitz ([email protected]) wrote: Correct me if I’m wrong, but we currently have a lot of hard-coded configuration files with the standard telemetry (yaf, bro, snort) included, but no consistent way to add additional telemetry configuration where needed?! I.e. we have - /usr/metron/<version>/config/zookeeper/[enrichments|parsers|indexing]/<telemetry-name>.json for topology configuration on the FS and a matching layout in zk - ES templates - Kibana configuration (both for index names and within some searches and vis elements) - Kafka topic creation and configuration The above exist for yaf, bro and snort only and it’s hard to get an additional telemetry source added consistenly. I.e. if I make manual adjustments to ES templates and Kibana configuration any “publish templates” action from Ambari will overwrite those changes. Wouldn’t it make sense to “register” telemetry sources centrally and then have the required configuration derived from there? Instead of starting with an exported/imported .kibana index it could be dynamically created (at runtime, not at build time), keeping user-defined values and only overwriting specific configuration statements. Is metron_service.py (METRON-777) supposed to solve some of those issues? If yes, would every telemetry source be treated as plugin? BR, Christian
