Hey Otto, Thanks for taking the time and providing feedback. Some answers/questions inline below.
On Mon, Sep 30, 2019 at 3:30 PM Otto Fowler <ottobackwa...@gmail.com> wrote: > I think we also need: > > - > > Draft of upgrade guide, reviewed and tried. What is missing? What is > hard? Possible refactoring from there. > That's part of #1 I mentioned in my list. We're working with Anand and Mohan on this as well - the expectation is that we will have guidance that is a combination of scripted backup/upgrade along with manual steps that should be clear enough to follow. Any platform-specific upgrade tasks (for a Hadoop cluster, specifically) will be linked to as part of this documentation. > - > > Draft of the new supported versions of XXXXXX for Metron > Versions noted in the Jira - https://issues.apache.org/jira/browse/METRON-2088. Are you thinking a specific mention in our READMEs beyond what we currently have documented? Either way, this is easy enough. > - > > How are we going to tell current users the change in support? I am not > sure burying in the readme is good enough. George isn’t doing blogs > anymore, but something like that would be cool, we should also be ready > with twitter and other things we can think of. Maybe James can do > something > like that? > This will be a major milestone for the project. I agree we should have some fanfare not only for a 1.0 release, but around notifying users of the upcoming changes. Wouldn't this just be part of our typical release announcement, or are you suggesting that in advance of the next release we make an announcement? Just to be clear, this is going to initiate a 1.0 for the next release, but there will be a period of time where the feature branch has been merged into master and we've not yet done a release. I don't know how much time will pass between those 2 steps, so it's worth noting. - > > Should the docker images we use for building etc be centos 7 too? > I don't think that's strictly necessary - the artifacts all work being built by centos 6 currently and EOL for centos 6 is nov 2020. https://wiki.centos.org/About/Product > - > > Have we removed the centos6 vagrant completely? updated the docs? Is > that in the upgrade notes? > We will need to add this to the upgrade notes if it is not already present. > - > > We have to make sure the full-dev stuff is all set and solid > Yes - we're doing all dev testing in our Vagrant box. Anand and Mohan are performing multi-node cluster testing (along with the upgrade testing) > - > > What about the Amazon script and stuff, the documentation for that? > I'm honestly not sure how to handle the AWS stuff because it's not free to do. Is anyone willing to donate some resources for testing this? > > I see if I can think of more, hopefully you all will too. > > > > > On September 30, 2019 at 17:14:09, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Hi all, > > We're getting close to wrapping up the feature branch work for upgrading to > HDP 3.1. As previously discussed, this is a backwards incompatible upgrade > and is going to require a 1.x Metron release. There has already been some > discussion on this here - > > https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E > . > > I want to hash out any remaining tasks we think would be gating factors for > getting the upgrade feature branch merged into master, and subsequently a > 1.x Metron release. Based on feedback in the previous dev list discussion > from above, I see a utility for helping automate the upgrade process (i.e. > config backup and restore) being the last major task. Including the upgrade > script, these are the remaining tasks I've identified: > > 1. Finish Metron upgrade script > 2. Upgrade hadoop - PR to follow from > https://github.com/mmiklavc/metron/tree/METRON-2232 (I'm currently > running this up in Vagrant) > 3. Cleanup Centos 7 dev environment documentation > 4. Finish Kerberos testing with all components now upgraded (depends on > hadoop upgrade being finished) https://github.com/apache/metron/pull/1519. > This should serve as our final acceptance testing for the upgrade - we've > heavily tested the FB on each PR, so this would be a final e2e check of > Metron with Kerberos enabled. > 5. Final DISCUSS to merge the FB into master (previous example of what > we did with PCAP here - > > https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E > ) > > *Note*: there are a couple tasks in the FB epic Jira around deprecating our > custom code with Kafka spouts. As it is, changing/upgrading this code is > not required for the upgrade. To reduce overall upgrade risk, we want to > consider this work as a follow on in a later point release. > > Here are the implications of this upgrade work from a supported components > perspective: > > - HDP 2.x is no longer supported - must upgrade > - Centos 6 is no longer supported - must upgrade to Centos 7 (Ambari > relies on binaries shipped by the HDP 3.1 distro, and they do not have any > binaries for Centos 6) > - Solr has also been upgraded from 6.6.2 to 7.4.0 - > https://issues.apache.org/jira/browse/METRON-2225 (No changes to the > Elasticsearch version have been made) > > I think that covers everything. Once we get the FB merged, I would expect > we would want to move towards a discussion and vote on releasing Metron > 1.0. Does this sound reasonable to everyone? > > Best, > Mike >
