> There is no equivalent usable ‘Metron’ application. That is absolutely part of the problem here. We are a platform on a platform, whereas many of the other projects in the Hadoop stack have very loose coupling/integrations with the other projects. HDF, which NiFi is a part of, would perhaps be more similar. Again, I don't believe the HDF MPack/Ambari work is in its own Apache project - that is a commercial bundle afaik.
On Thu, Sep 12, 2019 at 3:13 PM Otto Fowler <ottobackwa...@gmail.com> wrote: > I need a little while to think about this, but I would just like to say > that NiFi is a really bad example to compare Metron too. > > Nifi can be deployed as a single zip file and run from wherever it is > unzipped. It is more of an application that can be clustered with other > instances of that application. Metron is pretty different from that. > > There is no equivalent usable ‘Metron’ application. > > Can we think of a better example for inspiration? > > > > > On September 12, 2019 at 16:15:06, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'd like to discuss Metron's installation and management. We have used > Ambari for some time now, with and without an MPack for Metron (and > Elasticsearch). While this mechanism has proved useful to the project, it > is not without cost. This makes us an outlier among Apache projects in > terms of what we support as part of our installation and management > mechanism. It greatly expands the scope of responsibility for the Metron > community in terms of feature coverage that is not core to the > cybersecurity platform. > > We are currently undertaking an upgrade ( > https://issues.apache.org/jira/browse/METRON-2088) to beat some product > EOL > deadlines for Storm and the HDP binaries we depend on for deployment with > Ambari. This upgrade is critical for the Metron community. Ambari does not > prepackage any Hadoop binaries (see for example > https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so > for convenience we had been leveraging a commercial vendor's (Hortonworks) > binaries for HDP to fill this gap. There are a couple issues with this. > > First, and most dire, is that in the recent Cloudera earnings call > (Hortonworks and Cloudera have merged, if you're unaware), they have > publicly stated that *binaries will no longer be freely available to the > public *(see footnote 1). This means that our development environment is > tightly coupled to deployment artifacts that may cease to exist, and > subsequently break over night. We need to act on this immediately. > > Second, there is currently a bug in the latest versions of Ambari that > supports HDP 3.1 and our MPack and we are still working out if there is a > fix we can implement on our end to get around it - see > https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a > blocker for an upgrade if we cannot find a workaround (there has been one > failed attempt, another is under way so fingers crossed). > > Many other projects simply provide binaries and basic instructions for > deploying and using their products. See NiFi, for example - > > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi > . > > The product is developed in an a flexible and extensible way, and > individuals and vendors are free to expand on that basic functionality. > > I propose we take a similar approach and move to deprecate and replace > Ambari in one fell swoop as part of our 1.0 Metron release initiative. In > addition to the reasons listed above, this would also simplify our > development process, ease onboarding of new contributors and committers in > the community, and decrease the overall burden to the community for new > features and releases. The replacement for the development environment > could be a combination of Ansible and shell/Python scripts for managing > laying down the Metron bits. One option that has been discussed in the past > is to leverage open source versions of Docker containers for our service > dependencies. Recent conversations I've had with community devs suggests > this is still a popular solution. I agree. Quite a bit of Metron system > management is currently duplicated between Ambari and the management UI, > and we could maintain or expand on the current feature set in our UI as > desired. For general Hadoop/cluster management, we would no longer provide > a prepackaged option - BYOD as it were. There are numerous OSS and > commercial solutions available today, and users and vendors alike would be > free to choose one that works best for their specific use case(s) and > customer(s). > > Thanks everyone. This is not a vote, but opening the floor for discussion. > > Best, > Mike Miklavcic > Metron PMC > > *Notes:* > > (1) From the Cloudera earnings call - > > https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx > > "Secondly, the distribution of our compiled software, the binaries, will be > > limited. Specifically, access to these binaries, as well as support > > services and technical expertise will be available only with the current > > subscription agreement. The binaries contain Cloudera-specific > intellectual > > property, the testing, securing, and integration of various Open Source > > projects into an enterprise grade system that meets the requirements of > our > > customers. > > > > > Consistent with the Red Hat model, our binaries will no longer be freely > > available to non-paying customers. These changes to licensing and > > distribution will affect all subsequent versions of our current products > as > > well as new products, ensuring their future development by Cloudera and > the > > community is better protected." >
