I think we also need: -
Draft of upgrade guide, reviewed and tried. What is missing? What is hard? Possible refactoring from there. - Draft of the new supported versions of XXXXXX for Metron - How are we going to tell current users the change in support? I am not sure burying in the readme is good enough. George isn’t doing blogs anymore, but something like that would be cool, we should also be ready with twitter and other things we can think of. Maybe James can do something like that? - Should the docker images we use for building etc be centos 7 too? - Have we removed the centos6 vagrant completely? updated the docs? Is that in the upgrade notes? - We have to make sure the full-dev stuff is all set and solid - What about the Amazon script and stuff, the documentation for that? I see if I can think of more, hopefully you all will too. On September 30, 2019 at 17:14:09, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Hi all, We're getting close to wrapping up the feature branch work for upgrading to HDP 3.1. As previously discussed, this is a backwards incompatible upgrade and is going to require a 1.x Metron release. There has already been some discussion on this here - https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E . I want to hash out any remaining tasks we think would be gating factors for getting the upgrade feature branch merged into master, and subsequently a 1.x Metron release. Based on feedback in the previous dev list discussion from above, I see a utility for helping automate the upgrade process (i.e. config backup and restore) being the last major task. Including the upgrade script, these are the remaining tasks I've identified: 1. Finish Metron upgrade script 2. Upgrade hadoop - PR to follow from https://github.com/mmiklavc/metron/tree/METRON-2232 (I'm currently running this up in Vagrant) 3. Cleanup Centos 7 dev environment documentation 4. Finish Kerberos testing with all components now upgraded (depends on hadoop upgrade being finished) https://github.com/apache/metron/pull/1519. This should serve as our final acceptance testing for the upgrade - we've heavily tested the FB on each PR, so this would be a final e2e check of Metron with Kerberos enabled. 5. Final DISCUSS to merge the FB into master (previous example of what we did with PCAP here - https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E ) *Note*: there are a couple tasks in the FB epic Jira around deprecating our custom code with Kafka spouts. As it is, changing/upgrading this code is not required for the upgrade. To reduce overall upgrade risk, we want to consider this work as a follow on in a later point release. Here are the implications of this upgrade work from a supported components perspective: - HDP 2.x is no longer supported - must upgrade - Centos 6 is no longer supported - must upgrade to Centos 7 (Ambari relies on binaries shipped by the HDP 3.1 distro, and they do not have any binaries for Centos 6) - Solr has also been upgraded from 6.6.2 to 7.4.0 - https://issues.apache.org/jira/browse/METRON-2225 (No changes to the Elasticsearch version have been made) I think that covers everything. Once we get the FB merged, I would expect we would want to move towards a discussion and vote on releasing Metron 1.0. Does this sound reasonable to everyone? Best, Mike
