FYI - we were able to move forward on getting Kerberos working without requiring a Curator upgrade. We probably want to consider some architecture changes around how we manage this queue in MaaS at some point, however right now we will be continuing with the current version of Curator. The Zookeeper version is the same from HDP 2.5 to 2.6 to 3.1, so there should be no added risk.
On Fri, Sep 20, 2019 at 1:52 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Discussed this a bit further with Nick and Ryan offline. The original > issue with Curator appears to have come up during Kerberos testing. Storm > brings in a whole host of extra libs when Kerberos is enabled (some > preliminary testing was started in parallel to the HBase and Hadoop > upgrades). I'm told the lib in conflict is Guava. There had also been a > minor version bump in the Hadoop upgrade branch of Ryan's that I've been > basing some of my current work off of - I'm unclear of the need for that > change, but it looks like we might be able to avoid it. > > I'm investigating whether we can get away with simply shading and > relocating Guava in this case and leave a Curator upgrade as a follow-on > for a point release. It appears that Nick was able to get maas tests > running with an upgraded Hadoop without having to change the curator > versions. There are probably more classpath issues with a Hadoop upgrade > and Curator whether or not we rev the version, so that should be a wash in > risk and effort. However, the one added big risk with upgrading Curator is > that it's 1. a couple major version changes we're upgrading, and 2. we > already ran into one change in functionality based on the issues seen in > https://github.com/apache/metron/pull/1516. We still have more work to do > on upgrading Hadoop as well as Kerberos once Hadoop is finished being > upgraded. This means we could have a much higher probability of more > functionality changes as we get further into testing. It's possible there > are other reasons to motivate a Curator upgrade now, but this may be > premature without more concrete details and having tried a smaller scope > change with dependency relocation first. > > Mike > > On Tue, Sep 17, 2019 at 4:04 PM Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> +1 >> >> >> >> >> On September 17, 2019 at 14:53:13, Michael Miklavcic ( >> michael.miklav...@gmail.com) wrote: >> >> Also, caught this come through the HBase dev list - >> https://issues.apache.org/jira/browse/HBASE-23032. Might be worth a look >> at >> 4.2.0 while we're at it and possibly also avoid this issue that I was >> running into before when trying to use curator-tests 4.x >> https://issues.apache.org/jira/browse/CURATOR-446. Fixed in 4.0.1, which >> would naturally be included with 4.2.0 assuming we don't run into any >> other >> classpath or API compatibility issues. >> >> On Tue, Sep 17, 2019 at 11:20 AM Nick Allen <n...@nickallen.org> wrote: >> >> > +1 to making the change on the feature branch. >> > >> > We don't really know how this might affect master which is still >> building >> > against HDP 2.6, nor is it strictly needed there. Going to Curator >> 4.0.0 >> > is only needed due to the HDP 3.1 upgrade. This is also likely to get >> > more focused testing cycles in the feature branch before it has a >> chance to >> > break anything in master. >> > >> > On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic < >> > michael.miklav...@gmail.com> wrote: >> > >> > > Hey all, >> > > >> > > While working through the feature branch upgrade for HDP 3.1, we came >> > > across some classpath related issues conflicting with Storm and Guava >> > while >> > > testing out Kerberos. Initially, this seemed reasonable to roll into >> the >> > > Kerberos fix PR, but the scope has expanded a bit because we found >> > > additional impacts while working through the Hadoop upgrade branch as >> > well. >> > > We're now looking at splitting this out into a separate PR in the >> feature >> > > branch. Curator 4.0.0 is backwards compatible and will work with >> > Zookeeper >> > > 3.4.6. Strictly speaking, this *could* be done against master first, >> but >> > > there may be some duplicate testing effort there, and I'm really not >> it's >> > > worth it as there is absolutely no issue currently with Curator >> 2.12.0 in >> > > master - I call this out purely for transparency/information share >> with >> > the >> > > community. I'm leaning towards us making this change against the >> feature >> > > branch directly as that is the only place we currently have >> compatibility >> > > issues. >> > > >> > > Best, >> > > Mike >> > > >> > >> >>
