Hi all, We're getting close to wrapping up the feature branch work for upgrading to HDP 3.1. As previously discussed, this is a backwards incompatible upgrade and is going to require a 1.x Metron release. There has already been some discussion on this here - https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E .
I want to hash out any remaining tasks we think would be gating factors for getting the upgrade feature branch merged into master, and subsequently a 1.x Metron release. Based on feedback in the previous dev list discussion from above, I see a utility for helping automate the upgrade process (i.e. config backup and restore) being the last major task. Including the upgrade script, these are the remaining tasks I've identified: 1. Finish Metron upgrade script 2. Upgrade hadoop - PR to follow from https://github.com/mmiklavc/metron/tree/METRON-2232 (I'm currently running this up in Vagrant) 3. Cleanup Centos 7 dev environment documentation 4. Finish Kerberos testing with all components now upgraded (depends on hadoop upgrade being finished) https://github.com/apache/metron/pull/1519. This should serve as our final acceptance testing for the upgrade - we've heavily tested the FB on each PR, so this would be a final e2e check of Metron with Kerberos enabled. 5. Final DISCUSS to merge the FB into master (previous example of what we did with PCAP here - https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E ) *Note*: there are a couple tasks in the FB epic Jira around deprecating our custom code with Kafka spouts. As it is, changing/upgrading this code is not required for the upgrade. To reduce overall upgrade risk, we want to consider this work as a follow on in a later point release. Here are the implications of this upgrade work from a supported components perspective: - HDP 2.x is no longer supported - must upgrade - Centos 6 is no longer supported - must upgrade to Centos 7 (Ambari relies on binaries shipped by the HDP 3.1 distro, and they do not have any binaries for Centos 6) - Solr has also been upgraded from 6.6.2 to 7.4.0 - https://issues.apache.org/jira/browse/METRON-2225 (No changes to the Elasticsearch version have been made) I think that covers everything. Once we get the FB merged, I would expect we would want to move towards a discussion and vote on releasing Metron 1.0. Does this sound reasonable to everyone? Best, Mike
