First of all - very much looking forward to this approach. I'm not very familiar with management packs, but I did read some of the documentation in the link you sent.
Not sure if this is already included in a "minimum viable product," but at some point I think there needs to be a method of specifying proxies and/or internal package repos. I recently did a Metron 0.2.0 install behind a proxy (hence METRON-409 <https://issues.apache.org/jira/browse/METRON-409>) and it look me a semi-lengthy amount of time to (1) find all of the destinations I needed to request openings for in the proxy, and (2) modify the ambari scripts to appropriately use my proxies in the correct way. I also have a bit of a concern with upgrades and customizations in general (Not just how it would work with mpacks). I have not done any of this to date, but I have rebuilt and redeployed a couple of times and I needed to modify some of the metron code itself before build/deploy (because of my concern with it getting overwritten on upgrade if I just did it directly on the cluster). I would like to see a method of putting in install-specific files that modify or overwrite parts of the core metron stack, like changes to znodes, parsers, etc. Regarding not managing sensors with Ambari, I agree. I run a large bro cluster and it is maintained via Puppet and various other mechanisms - no need for Ambari to bleed over in my case. Thanks for the great work. Jon On Thu, Sep 15, 2016 at 5:10 PM Casey Stella <ceste...@gmail.com> wrote: > Hi Everyone, > > I wanted to solicit some discussion around a feature that is fast > approaching. A major pain point in using Metron is installation. Thus far > our only approach to installation has been driven by the developer's needs > to construct a virtual environment to test out changes, which lead us to > either an ansible installation or a manual installation. > > Because we want to make sure that the installation of Metron is as easy as > possible, we have had some great contributions of an additional approach, > installation via Apache Ambari directly. Our ansible scripts currently > rely on Ambari blueprints to set up Hadoop on the cluster that it is > deploying on, so it is not a new dependency, but we're working toward a > full Ambari management pack > <https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs> > that will lay down the relevant topologies (parser, enrichment, indexing), > configs, bits and their infrastructural dependencies (ES and mysql) and > allow the topologies to be started and stopped as minimum viable product. > > The beginnings of this have started with: > > - Ambari Service Definitions for the Parser topologies > <https://github.com/apache/incubator-metron/pull/218> > - Ambari Service Definition for the Indexing Topology > <https://github.com/apache/incubator-metron/pull/222> > - Ambari Service Definition for Elasticsearch > <https://github.com/apache/incubator-metron/pull/223> > > There will be more to come in the near-term to realize that vision, but we > wanted to get some reactions. Past minimum viable product, what do you > guys think we should have and how should it look? > > Currently we are treating the domain of the ambari installation as from > kafka to the indexes, which leaves the sensors unmanaged via ambari. Is > that a good decision? > > Are there other pain points that you have had around installation that > you'd like to see addressed? > > The purpose of this discussion thread is to let you guys know that we will > soon have a new way to install metron, but also to understand what the > future requirements are so we, as a community, can address them. > > Best, > > Casey > -- Jon
