Otto - Also, in reference to the Slack channel note about Ansible - the installation/commands would not be managed by Ansible. We'd likely only use it as needed to do very basic calls out to the install/setup scripts that we would provide for Metron. There's probably some difference between how this looks for our dev environment (as mentioned, Docker for the Hadoop services) and how a user might choose to deploy and manage their hadoop cluster. Here are Apache instructions for Bigtop, for example - https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+1.2.0 .
I absolutely hear what you're saying regarding a single zip file with NiFi. We're actually not *that* different. I think the only material way we deviate from that is when distributing our client jaas for Kerberos - each node needs some basic Metron artifacts. Even so, providing a script for running a distributed scp given a list of hostnames doesn't seem too big a challenge. It's possible we should also consider how Debs and RPMs fit in this mix. I think the only reason we do that currently is for integration with Ambari. We fundamentally could bundle up the entire application in one fell swoop as a zip and explode it out onto a node. On Thu, Sep 12, 2019 at 3:22 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > > There is no equivalent usable ‘Metron’ application. > > That is absolutely part of the problem here. We are a platform on a > platform, whereas many of the other projects in the Hadoop stack have very > loose coupling/integrations with the other projects. HDF, which NiFi is a > part of, would perhaps be more similar. Again, I don't believe the HDF > MPack/Ambari work is in its own Apache project - that is a commercial > bundle afaik. > > On Thu, Sep 12, 2019 at 3:13 PM Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> I need a little while to think about this, but I would just like to say >> that NiFi is a really bad example to compare Metron too. >> >> Nifi can be deployed as a single zip file and run from wherever it is >> unzipped. It is more of an application that can be clustered with other >> instances of that application. Metron is pretty different from that. >> >> There is no equivalent usable ‘Metron’ application. >> >> Can we think of a better example for inspiration? >> >> >> >> >> On September 12, 2019 at 16:15:06, Michael Miklavcic ( >> michael.miklav...@gmail.com) wrote: >> >> I'd like to discuss Metron's installation and management. We have used >> Ambari for some time now, with and without an MPack for Metron (and >> Elasticsearch). While this mechanism has proved useful to the project, it >> is not without cost. This makes us an outlier among Apache projects in >> terms of what we support as part of our installation and management >> mechanism. It greatly expands the scope of responsibility for the Metron >> community in terms of feature coverage that is not core to the >> cybersecurity platform. >> >> We are currently undertaking an upgrade ( >> https://issues.apache.org/jira/browse/METRON-2088) to beat some product >> EOL >> deadlines for Storm and the HDP binaries we depend on for deployment with >> Ambari. This upgrade is critical for the Metron community. Ambari does not >> prepackage any Hadoop binaries (see for example >> https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so >> for convenience we had been leveraging a commercial vendor's (Hortonworks) >> binaries for HDP to fill this gap. There are a couple issues with this. >> >> First, and most dire, is that in the recent Cloudera earnings call >> (Hortonworks and Cloudera have merged, if you're unaware), they have >> publicly stated that *binaries will no longer be freely available to the >> public *(see footnote 1). This means that our development environment is >> tightly coupled to deployment artifacts that may cease to exist, and >> subsequently break over night. We need to act on this immediately. >> >> Second, there is currently a bug in the latest versions of Ambari that >> supports HDP 3.1 and our MPack and we are still working out if there is a >> fix we can implement on our end to get around it - see >> https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a >> blocker for an upgrade if we cannot find a workaround (there has been one >> failed attempt, another is under way so fingers crossed). >> >> Many other projects simply provide binaries and basic instructions for >> deploying and using their products. See NiFi, for example - >> >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi >> . >> >> The product is developed in an a flexible and extensible way, and >> individuals and vendors are free to expand on that basic functionality. >> >> I propose we take a similar approach and move to deprecate and replace >> Ambari in one fell swoop as part of our 1.0 Metron release initiative. In >> addition to the reasons listed above, this would also simplify our >> development process, ease onboarding of new contributors and committers in >> the community, and decrease the overall burden to the community for new >> features and releases. The replacement for the development environment >> could be a combination of Ansible and shell/Python scripts for managing >> laying down the Metron bits. One option that has been discussed in the >> past >> is to leverage open source versions of Docker containers for our service >> dependencies. Recent conversations I've had with community devs suggests >> this is still a popular solution. I agree. Quite a bit of Metron system >> management is currently duplicated between Ambari and the management UI, >> and we could maintain or expand on the current feature set in our UI as >> desired. For general Hadoop/cluster management, we would no longer provide >> a prepackaged option - BYOD as it were. There are numerous OSS and >> commercial solutions available today, and users and vendors alike would be >> free to choose one that works best for their specific use case(s) and >> customer(s). >> >> Thanks everyone. This is not a vote, but opening the floor for discussion. >> >> Best, >> Mike Miklavcic >> Metron PMC >> >> *Notes:* >> >> (1) From the Cloudera earnings call - >> >> https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx >> >> "Secondly, the distribution of our compiled software, the binaries, will >> be >> > limited. Specifically, access to these binaries, as well as support >> > services and technical expertise will be available only with the current >> > subscription agreement. The binaries contain Cloudera-specific >> intellectual >> > property, the testing, securing, and integration of various Open Source >> > projects into an enterprise grade system that meets the requirements of >> our >> > customers. >> > >> >> >> Consistent with the Red Hat model, our binaries will no longer be freely >> > available to non-paying customers. These changes to licensing and >> > distribution will affect all subsequent versions of our current products >> as >> > well as new products, ensuring their future development by Cloudera and >> the >> > community is better protected." >> >
