FYI - I opened a ticket to serve as an epic for this work and the feature branch.
https://issues.apache.org/jira/browse/METRON-2088 On Mon, Apr 22, 2019 at 3:32 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > +1 to starting a feature branch for this. > +1 to removing our custom implementations if the newest revs are in fact > stable now. > > Regarding the profile option - if it's possible to keep 2.6.5 for a bit and > not require separate branches or code trees, this is probably OK. > Otherwise, I'm inclined to take the approach we've taken in the past with > every other upgrade and only support 1 version. I think we should prepare > users for the likelihood that if/when we cut over, there will be no more > updates to 2.6.x. > > I talked through this a bit with Nick and Ryan Merriman offline. There are > a number of major version revs of components from HDP 2.6 to 3.x that are > likely to have backwards compatibility problems. HBase is a big one that > comes to mind - I noticed the HTable interface was deprecated while working > through the coprocessor implementation, and Ryan found that it was removed > completely in the new version. That affects our integration tests as well > bc we have a rather large mock implementation of HBase in use that is built > around the removed API. We will either need to migrate to the new API or > find alternative approach to integration testing with HBase. > > I'll let Nick add more detail in the Epic/Jira and feature branch plan, but > here is a sampling of some of what we can expect to require some work to > upgrade: > > - Ambari - the current MPack is incompatible with Ambari 2.7.3, however > there isn't a breaking changes document, so we'll have to work through > this > brute force or hopefully find some help from the Ambari community. > - MaaS - YARN major change > - PCAP - HDFS, Kafka > - Indexing - HDFS, Solr > - All topologies - Kafka > - Stellar - HDFS, HBase > - Enrichment - HBase > - Enrichment Coprocessor (the enrichments listing) - HBase > - Integration tests - Kafka and HBase have changed considerably. > - UI, REST - Solr, HDFS, HBase > - Knox > - Kerberos (hopefully this is a kick-the-tires effort, though there is > some possible risk if Ambari and the individual components introduce > changes here) > > Fortunately, Zookeeper appears to have stayed the same across versions. It > might be worthwhile to get a chart of the versions for each platform added > to the epic Jira for reference while performing this work. > > Best, > Mike > > > On Mon, Apr 22, 2019, 12:50 PM Nick Allen <n...@nickallen.org> wrote: > > > We currently support running Metron on an HDP 2.6.5 > > < > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > > > > > cluster. > > I'd like to get Metron updated to run in an HDP 3.1.0 > > < > > > https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html > > > > > cluster. > > This provides a number of significant updates to the core platform > > components that we depend on like Kafka, HBase, Ambari, etc. > > > > ### Feature Branch > > > > I'd like to create a feature branch in which to do this. This will take > a > > good amount of effort and multiple PRs. To avoid any impact to master as > we > > progress through this, a feature branch would make sense. > > > > If you have concerns or interest in this effort, please speak up. Here > are > > some relevant discussion points based on what I know so far. > > > > ### CentOS 7 > > > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7 > RPMs. > > Because of this we will likely need to transition Full Dev over to CentOS > > 7. I don't see a downside to doing this since 6 is rather old and I > assume > > that most users run variants of 7 already anyways. > > > > ### HDP 2.6.5 > > > > I'd like to try and make these changes backwards compatible with HDP > 2.6.5 > > if possible, but only as long as that does not increase our ongoing > > development burden. > > > > For example, if I can simply define a separate build profile for 3.1.0 > and > > things are generally backwards compatible, then I'm all for maintaining > > support for 2.6.5. On the other hand, I would not want to go as far as > > maintaining separate master branches for each. In my mind the ongoing > cost > > there is too high. > > > > ### HDP 2.5.6 > > > > There are some workaround in the code base that were introduced to > support > > HDP > > 2.5.6 > > < > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html > > > > > when > > we moved to HDP 2.6.5. There are some workarounds specifically for older > > versions of Storm like 1.0.x. Rather than maintaining this going forward, > > I'd prefer we remove this technical debt and not support anything older > > than HDP 2.6.5. > > > > > > > > > > Best, > > Nick > > >
