+1 to starting a feature branch for this. +1 to removing our custom implementations if the newest revs are in fact stable now.
Regarding the profile option - if it's possible to keep 2.6.5 for a bit and not require separate branches or code trees, this is probably OK. Otherwise, I'm inclined to take the approach we've taken in the past with every other upgrade and only support 1 version. I think we should prepare users for the likelihood that if/when we cut over, there will be no more updates to 2.6.x. I talked through this a bit with Nick and Ryan Merriman offline. There are a number of major version revs of components from HDP 2.6 to 3.x that are likely to have backwards compatibility problems. HBase is a big one that comes to mind - I noticed the HTable interface was deprecated while working through the coprocessor implementation, and Ryan found that it was removed completely in the new version. That affects our integration tests as well bc we have a rather large mock implementation of HBase in use that is built around the removed API. We will either need to migrate to the new API or find alternative approach to integration testing with HBase. I'll let Nick add more detail in the Epic/Jira and feature branch plan, but here is a sampling of some of what we can expect to require some work to upgrade: - Ambari - the current MPack is incompatible with Ambari 2.7.3, however there isn't a breaking changes document, so we'll have to work through this brute force or hopefully find some help from the Ambari community. - MaaS - YARN major change - PCAP - HDFS, Kafka - Indexing - HDFS, Solr - All topologies - Kafka - Stellar - HDFS, HBase - Enrichment - HBase - Enrichment Coprocessor (the enrichments listing) - HBase - Integration tests - Kafka and HBase have changed considerably. - UI, REST - Solr, HDFS, HBase - Knox - Kerberos (hopefully this is a kick-the-tires effort, though there is some possible risk if Ambari and the individual components introduce changes here) Fortunately, Zookeeper appears to have stayed the same across versions. It might be worthwhile to get a chart of the versions for each platform added to the epic Jira for reference while performing this work. Best, Mike On Mon, Apr 22, 2019, 12:50 PM Nick Allen <n...@nickallen.org> wrote: > We currently support running Metron on an HDP 2.6.5 > < > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > > > cluster. > I'd like to get Metron updated to run in an HDP 3.1.0 > < > https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html > > > cluster. > This provides a number of significant updates to the core platform > components that we depend on like Kafka, HBase, Ambari, etc. > > ### Feature Branch > > I'd like to create a feature branch in which to do this. This will take a > good amount of effort and multiple PRs. To avoid any impact to master as we > progress through this, a feature branch would make sense. > > If you have concerns or interest in this effort, please speak up. Here are > some relevant discussion points based on what I know so far. > > ### CentOS 7 > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7 RPMs. > Because of this we will likely need to transition Full Dev over to CentOS > 7. I don't see a downside to doing this since 6 is rather old and I assume > that most users run variants of 7 already anyways. > > ### HDP 2.6.5 > > I'd like to try and make these changes backwards compatible with HDP 2.6.5 > if possible, but only as long as that does not increase our ongoing > development burden. > > For example, if I can simply define a separate build profile for 3.1.0 and > things are generally backwards compatible, then I'm all for maintaining > support for 2.6.5. On the other hand, I would not want to go as far as > maintaining separate master branches for each. In my mind the ongoing cost > there is too high. > > ### HDP 2.5.6 > > There are some workaround in the code base that were introduced to support > HDP > 2.5.6 > < > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.6/bk_release-notes/content/comp_versions.html > > > when > we moved to HDP 2.6.5. There are some workarounds specifically for older > versions of Storm like 1.0.x. Rather than maintaining this going forward, > I'd prefer we remove this technical debt and not support anything older > than HDP 2.6.5. > > > > > Best, > Nick >
