Re: [DISCUSS] Metron installation and management - deprecating Ambari support
Otto - Also, in reference to the Slack channel note about Ansible - the installation/commands would not be managed by Ansible. We'd likely only use it as needed to do very basic calls out to the install/setup scripts that we would provide for Metron. There's probably some difference between how this looks for our dev environment (as mentioned, Docker for the Hadoop services) and how a user might choose to deploy and manage their hadoop cluster. Here are Apache instructions for Bigtop, for example - https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+1.2.0 . I absolutely hear what you're saying regarding a single zip file with NiFi. We're actually not *that* different. I think the only material way we deviate from that is when distributing our client jaas for Kerberos - each node needs some basic Metron artifacts. Even so, providing a script for running a distributed scp given a list of hostnames doesn't seem too big a challenge. It's possible we should also consider how Debs and RPMs fit in this mix. I think the only reason we do that currently is for integration with Ambari. We fundamentally could bundle up the entire application in one fell swoop as a zip and explode it out onto a node. On Thu, Sep 12, 2019 at 3:22 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > > There is no equivalent usable ‘Metron’ application. > > That is absolutely part of the problem here. We are a platform on a > platform, whereas many of the other projects in the Hadoop stack have very > loose coupling/integrations with the other projects. HDF, which NiFi is a > part of, would perhaps be more similar. Again, I don't believe the HDF > MPack/Ambari work is in its own Apache project - that is a commercial > bundle afaik. > > On Thu, Sep 12, 2019 at 3:13 PM Otto Fowler > wrote: > >> I need a little while to think about this, but I would just like to say >> that NiFi is a really bad example to compare Metron too. >> >> Nifi can be deployed as a single zip file and run from wherever it is >> unzipped. It is more of an application that can be clustered with other >> instances of that application. Metron is pretty different from that. >> >> There is no equivalent usable ‘Metron’ application. >> >> Can we think of a better example for inspiration? >> >> >> >> >> On September 12, 2019 at 16:15:06, Michael Miklavcic ( >> michael.miklav...@gmail.com) wrote: >> >> I'd like to discuss Metron's installation and management. We have used >> Ambari for some time now, with and without an MPack for Metron (and >> Elasticsearch). While this mechanism has proved useful to the project, it >> is not without cost. This makes us an outlier among Apache projects in >> terms of what we support as part of our installation and management >> mechanism. It greatly expands the scope of responsibility for the Metron >> community in terms of feature coverage that is not core to the >> cybersecurity platform. >> >> We are currently undertaking an upgrade ( >> https://issues.apache.org/jira/browse/METRON-2088) to beat some product >> EOL >> deadlines for Storm and the HDP binaries we depend on for deployment with >> Ambari. This upgrade is critical for the Metron community. Ambari does not >> prepackage any Hadoop binaries (see for example >> https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so >> for convenience we had been leveraging a commercial vendor's (Hortonworks) >> binaries for HDP to fill this gap. There are a couple issues with this. >> >> First, and most dire, is that in the recent Cloudera earnings call >> (Hortonworks and Cloudera have merged, if you're unaware), they have >> publicly stated that *binaries will no longer be freely available to the >> public *(see footnote 1). This means that our development environment is >> tightly coupled to deployment artifacts that may cease to exist, and >> subsequently break over night. We need to act on this immediately. >> >> Second, there is currently a bug in the latest versions of Ambari that >> supports HDP 3.1 and our MPack and we are still working out if there is a >> fix we can implement on our end to get around it - see >> https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a >> blocker for an upgrade if we cannot find a workaround (there has been one >> failed attempt, another is under way so fingers crossed). >> >> Many other projects simply provide binaries and basic instructions for >> deploying and using their products. See NiFi, for example - >> >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi >> . >> >> The product is developed in an a flexible and extensible way, and >> individuals and vendors are free to expand on that basic functionality. >> >> I propose we take a similar approach and move to deprecate and replace >> Ambari in one fell swoop as part of our 1.0 Metron release initiative. In >> addition to the reasons listed above, this
Re: [DISCUSS] Metron installation and management - deprecating Ambari support
> There is no equivalent usable ‘Metron’ application. That is absolutely part of the problem here. We are a platform on a platform, whereas many of the other projects in the Hadoop stack have very loose coupling/integrations with the other projects. HDF, which NiFi is a part of, would perhaps be more similar. Again, I don't believe the HDF MPack/Ambari work is in its own Apache project - that is a commercial bundle afaik. On Thu, Sep 12, 2019 at 3:13 PM Otto Fowler wrote: > I need a little while to think about this, but I would just like to say > that NiFi is a really bad example to compare Metron too. > > Nifi can be deployed as a single zip file and run from wherever it is > unzipped. It is more of an application that can be clustered with other > instances of that application. Metron is pretty different from that. > > There is no equivalent usable ‘Metron’ application. > > Can we think of a better example for inspiration? > > > > > On September 12, 2019 at 16:15:06, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'd like to discuss Metron's installation and management. We have used > Ambari for some time now, with and without an MPack for Metron (and > Elasticsearch). While this mechanism has proved useful to the project, it > is not without cost. This makes us an outlier among Apache projects in > terms of what we support as part of our installation and management > mechanism. It greatly expands the scope of responsibility for the Metron > community in terms of feature coverage that is not core to the > cybersecurity platform. > > We are currently undertaking an upgrade ( > https://issues.apache.org/jira/browse/METRON-2088) to beat some product > EOL > deadlines for Storm and the HDP binaries we depend on for deployment with > Ambari. This upgrade is critical for the Metron community. Ambari does not > prepackage any Hadoop binaries (see for example > https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so > for convenience we had been leveraging a commercial vendor's (Hortonworks) > binaries for HDP to fill this gap. There are a couple issues with this. > > First, and most dire, is that in the recent Cloudera earnings call > (Hortonworks and Cloudera have merged, if you're unaware), they have > publicly stated that *binaries will no longer be freely available to the > public *(see footnote 1). This means that our development environment is > tightly coupled to deployment artifacts that may cease to exist, and > subsequently break over night. We need to act on this immediately. > > Second, there is currently a bug in the latest versions of Ambari that > supports HDP 3.1 and our MPack and we are still working out if there is a > fix we can implement on our end to get around it - see > https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a > blocker for an upgrade if we cannot find a workaround (there has been one > failed attempt, another is under way so fingers crossed). > > Many other projects simply provide binaries and basic instructions for > deploying and using their products. See NiFi, for example - > > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi > . > > The product is developed in an a flexible and extensible way, and > individuals and vendors are free to expand on that basic functionality. > > I propose we take a similar approach and move to deprecate and replace > Ambari in one fell swoop as part of our 1.0 Metron release initiative. In > addition to the reasons listed above, this would also simplify our > development process, ease onboarding of new contributors and committers in > the community, and decrease the overall burden to the community for new > features and releases. The replacement for the development environment > could be a combination of Ansible and shell/Python scripts for managing > laying down the Metron bits. One option that has been discussed in the past > is to leverage open source versions of Docker containers for our service > dependencies. Recent conversations I've had with community devs suggests > this is still a popular solution. I agree. Quite a bit of Metron system > management is currently duplicated between Ambari and the management UI, > and we could maintain or expand on the current feature set in our UI as > desired. For general Hadoop/cluster management, we would no longer provide > a prepackaged option - BYOD as it were. There are numerous OSS and > commercial solutions available today, and users and vendors alike would be > free to choose one that works best for their specific use case(s) and > customer(s). > > Thanks everyone. This is not a vote, but opening the floor for discussion. > > Best, > Mike Miklavcic > Metron PMC > > *Notes:* > > (1) From the Cloudera earnings call - > > https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx > > "Secondly, the distribution of our compiled
Re: [DISCUSS] Metron installation and management - deprecating Ambari support
I need a little while to think about this, but I would just like to say that NiFi is a really bad example to compare Metron too. Nifi can be deployed as a single zip file and run from wherever it is unzipped. It is more of an application that can be clustered with other instances of that application. Metron is pretty different from that. There is no equivalent usable ‘Metron’ application. Can we think of a better example for inspiration? On September 12, 2019 at 16:15:06, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: I'd like to discuss Metron's installation and management. We have used Ambari for some time now, with and without an MPack for Metron (and Elasticsearch). While this mechanism has proved useful to the project, it is not without cost. This makes us an outlier among Apache projects in terms of what we support as part of our installation and management mechanism. It greatly expands the scope of responsibility for the Metron community in terms of feature coverage that is not core to the cybersecurity platform. We are currently undertaking an upgrade ( https://issues.apache.org/jira/browse/METRON-2088) to beat some product EOL deadlines for Storm and the HDP binaries we depend on for deployment with Ambari. This upgrade is critical for the Metron community. Ambari does not prepackage any Hadoop binaries (see for example https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so for convenience we had been leveraging a commercial vendor's (Hortonworks) binaries for HDP to fill this gap. There are a couple issues with this. First, and most dire, is that in the recent Cloudera earnings call (Hortonworks and Cloudera have merged, if you're unaware), they have publicly stated that *binaries will no longer be freely available to the public *(see footnote 1). This means that our development environment is tightly coupled to deployment artifacts that may cease to exist, and subsequently break over night. We need to act on this immediately. Second, there is currently a bug in the latest versions of Ambari that supports HDP 3.1 and our MPack and we are still working out if there is a fix we can implement on our end to get around it - see https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a blocker for an upgrade if we cannot find a workaround (there has been one failed attempt, another is under way so fingers crossed). Many other projects simply provide binaries and basic instructions for deploying and using their products. See NiFi, for example - https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi. The product is developed in an a flexible and extensible way, and individuals and vendors are free to expand on that basic functionality. I propose we take a similar approach and move to deprecate and replace Ambari in one fell swoop as part of our 1.0 Metron release initiative. In addition to the reasons listed above, this would also simplify our development process, ease onboarding of new contributors and committers in the community, and decrease the overall burden to the community for new features and releases. The replacement for the development environment could be a combination of Ansible and shell/Python scripts for managing laying down the Metron bits. One option that has been discussed in the past is to leverage open source versions of Docker containers for our service dependencies. Recent conversations I've had with community devs suggests this is still a popular solution. I agree. Quite a bit of Metron system management is currently duplicated between Ambari and the management UI, and we could maintain or expand on the current feature set in our UI as desired. For general Hadoop/cluster management, we would no longer provide a prepackaged option - BYOD as it were. There are numerous OSS and commercial solutions available today, and users and vendors alike would be free to choose one that works best for their specific use case(s) and customer(s). Thanks everyone. This is not a vote, but opening the floor for discussion. Best, Mike Miklavcic Metron PMC *Notes:* (1) From the Cloudera earnings call - https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx "Secondly, the distribution of our compiled software, the binaries, will be > limited. Specifically, access to these binaries, as well as support > services and technical expertise will be available only with the current > subscription agreement. The binaries contain Cloudera-specific intellectual > property, the testing, securing, and integration of various Open Source > projects into an enterprise grade system that meets the requirements of our > customers. > Consistent with the Red Hat model, our binaries will no longer be freely > available to non-paying customers. These changes to licensing and > distribution will affect all subsequent versions of our current products as > well
[DISCUSS] Metron installation and management - deprecating Ambari support
I'd like to discuss Metron's installation and management. We have used Ambari for some time now, with and without an MPack for Metron (and Elasticsearch). While this mechanism has proved useful to the project, it is not without cost. This makes us an outlier among Apache projects in terms of what we support as part of our installation and management mechanism. It greatly expands the scope of responsibility for the Metron community in terms of feature coverage that is not core to the cybersecurity platform. We are currently undertaking an upgrade ( https://issues.apache.org/jira/browse/METRON-2088) to beat some product EOL deadlines for Storm and the HDP binaries we depend on for deployment with Ambari. This upgrade is critical for the Metron community. Ambari does not prepackage any Hadoop binaries (see for example https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so for convenience we had been leveraging a commercial vendor's (Hortonworks) binaries for HDP to fill this gap. There are a couple issues with this. First, and most dire, is that in the recent Cloudera earnings call (Hortonworks and Cloudera have merged, if you're unaware), they have publicly stated that *binaries will no longer be freely available to the public *(see footnote 1). This means that our development environment is tightly coupled to deployment artifacts that may cease to exist, and subsequently break over night. We need to act on this immediately. Second, there is currently a bug in the latest versions of Ambari that supports HDP 3.1 and our MPack and we are still working out if there is a fix we can implement on our end to get around it - see https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a blocker for an upgrade if we cannot find a workaround (there has been one failed attempt, another is under way so fingers crossed). Many other projects simply provide binaries and basic instructions for deploying and using their products. See NiFi, for example - https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi. The product is developed in an a flexible and extensible way, and individuals and vendors are free to expand on that basic functionality. I propose we take a similar approach and move to deprecate and replace Ambari in one fell swoop as part of our 1.0 Metron release initiative. In addition to the reasons listed above, this would also simplify our development process, ease onboarding of new contributors and committers in the community, and decrease the overall burden to the community for new features and releases. The replacement for the development environment could be a combination of Ansible and shell/Python scripts for managing laying down the Metron bits. One option that has been discussed in the past is to leverage open source versions of Docker containers for our service dependencies. Recent conversations I've had with community devs suggests this is still a popular solution. I agree. Quite a bit of Metron system management is currently duplicated between Ambari and the management UI, and we could maintain or expand on the current feature set in our UI as desired. For general Hadoop/cluster management, we would no longer provide a prepackaged option - BYOD as it were. There are numerous OSS and commercial solutions available today, and users and vendors alike would be free to choose one that works best for their specific use case(s) and customer(s). Thanks everyone. This is not a vote, but opening the floor for discussion. Best, Mike Miklavcic Metron PMC *Notes:* (1) From the Cloudera earnings call - https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx "Secondly, the distribution of our compiled software, the binaries, will be > limited. Specifically, access to these binaries, as well as support > services and technical expertise will be available only with the current > subscription agreement. The binaries contain Cloudera-specific intellectual > property, the testing, securing, and integration of various Open Source > projects into an enterprise grade system that meets the requirements of our > customers. > Consistent with the Red Hat model, our binaries will no longer be freely > available to non-paying customers. These changes to licensing and > distribution will affect all subsequent versions of our current products as > well as new products, ensuring their future development by Cloudera and the > community is better protected."
