Correct me if I’m wrong, but we currently have a lot of hard-coded configuration files with the standard telemetry (yaf, bro, snort) included, but no consistent way to add additional telemetry configuration where needed?! I.e. we have - /usr/metron/<version>/config/zookeeper/[enrichments|parsers|indexing]/<telemetry-name>.json for topology configuration on the FS and a matching layout in zk - ES templates - Kibana configuration (both for index names and within some searches and vis elements) - Kafka topic creation and configuration The above exist for yaf, bro and snort only and it’s hard to get an additional telemetry source added consistenly. I.e. if I make manual adjustments to ES templates and Kibana configuration any “publish templates” action from Ambari will overwrite those changes.
Wouldn’t it make sense to “register” telemetry sources centrally and then have the required configuration derived from there? Instead of starting with an exported/imported .kibana index it could be dynamically created (at runtime, not at build time), keeping user-defined values and only overwriting specific configuration statements. Is metron_service.py (METRON-777) supposed to solve some of those issues? If yes, would every telemetry source be treated as plugin? BR, Christian
