Thanks to everyone who participated in the discussion and suggested improvements like additional testing.
This has been merged on all branches, and I have also updated the docs to include the new policy. Istvan On Tue, Dec 17, 2024 at 2:51 PM Nick Dimiduk <ndimi...@apache.org> wrote: > I've come around to be in favor of bumping the default version for > branch-2.6 and maintaining it up on the latest backward-compatible minor > releases that we can. I arrive at this conclusion because we do not have > the bandwidth as a community to make more frequent minor or major releases. > That means the only way that we can stay ahead of the aggressive CVE train > coming out of our massive dependency stack is to aggressively upgrade the > deps. > > I've given my +1 on HBASE-28980 for branch-2.6 > > Thanks, > Nick > > On 2024/11/15 06:26:08 Istvan Toth wrote: > > With Duo's fix for HBASE-28965 , there are no more known blockers. > > > > Should I go ahead with making 3.4.1 the default for branch-2.5 and > > branch-2.6 ? > > > > On Tue, Nov 5, 2024 at 7:16 AM Istvan Toth <st...@cloudera.com> wrote: > > > > > I've just committed the - hopefully - last test change, so the > nightlies > > > will not always fail on the packaging tests. > > > > > > All non-released branches (i.e. 2.7+ now default to building with > 3.4.1). > > > > > > I wanted to revisit updating the default version on the released (2.5, > > > 2.6) branches, because Nick has expressed concerns about it. > > > > > > The dev tests are good (the test failures seem to be "normal" > flakies). As > > > we've not updated the default version, we're not yet running the > packaging > > > tests > > > on branch-2.5 and 2.6 against the older Hadoop versions, but we do on > the > > > rest of branches, and if we update it, we will also run them on 2.5 > and 2.6. > > > > > > Without repeating the arguments I have already made for it, I want to > add > > > a new one: > > > > > > Security and CVEs are getting more and more emphasis, which is great, > but > > > has some drawbacks. > > > While the proliferation of static analyzers leads to a lot of > frustrating > > > CVE witch hunts and false positives, the majority of users cannot > evaluate > > > the actual security impact, > > > or even if they can, they are tied by inflexible policies. > > > > > > We at Phoenix had recent security discussions with Trino, and ended up > > > having to dependencyManage the transitive Hadoop dependencies in our > shaded > > > uberjar to address their concerns. > > > (Which is an antipattern) > > > > > > While updating the Hadoop version in a patch release undoubtedly > increases > > > the risk of regressions, IMO we are protected by the Hadoop backwards > > > compatibility promises, and we have added a reasonable number of tests > to > > > catch any issues. I am confident in our test coverage when building > HBase > > > with any of the supported HBase versions. We also have the packaging > > > (smoke) tests for running the official binaries against earlier Hadoop > > > clusters, which would catch (some of) the Hadoop wire api > incompatibilities > > > . > > > > > > However, IMO not updating Hadoop is net negative to the project's > health, > > > as the binary (maven or assembly) releases are used primarily either by > > > other libraries interfacing with HBase, or by new users, POC clusters, > etc. > > > and these are the use cases where the transitive CVEs can prevent > projects > > > from adding (or maintaining as in the case of Trino) HBase support, or > > > discourage new users from adopting HBase. > > > > > > Obviously, I have a very skewed view of HBase users, but I think that > most > > > production HBase users either use a vendor or cloud provider version, > or > > > have enough in-house expertise to rebuild HBase with their Hadoop > version > > > if something goes wrong (despite the Hadoop backwards compatibility > > > promises) > > > > > > > > > > > > > > > > > > On Wed, Oct 30, 2024 at 12:41 PM Istvan Toth <st...@cloudera.com> > wrote: > > > > > >> Thanks! > > >> > > >> I will backport the test changes, but keep the default Hadoop version. > > >> > > >> We will have more information then. > > >> > > >> Istvan > > >> > > >> On Wed, Oct 30, 2024 at 10:22 AM Nick Dimiduk <ndimi...@apache.org> > > >> wrote: > > >> > > >>> On Mon, Oct 28, 2024 at 11:00 AM Istvan Toth > <st...@cloudera.com.invalid> > > >>> wrote: > > >>> > > > >>> > I have looked at branch-2.5, but the nightly looks off there, as it > > >>> runs > > >>> > the packaging tests with Hadoop 3.1.1, which it doesn't even > > >>> officially > > >>> > support anymore. > > >>> > > > >>> > What should we do with branch-2.5 ? > > >>> > > > >>> > I think that it would not be a lot of extra work to backport > > >>> everything, > > >>> > both the backwards compatibility tests and defaulting Hadoop to > 3.4.1. > > >>> > We just have to update the version in the pom, and add 3.2.4 to the > > >>> list of > > >>> > versions to test for backwards compatibility and integration (and > > >>> remove > > >>> > 3.1.1). > > >>> > > > >>> > I would prefer to have uniform tests and default to Hadoop 3.4.1 > on all > > >>> > active branches. > > >>> > Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x > support > > >>> may > > >>> > be useful for users for migration and CVE mitigation purposes. > > >>> > > > >>> > WDYT ? > > >>> > > >>> branch-2.5's default hadoop3 version is 3.2.4. That's a big > dependency > > >>> to change for a patch release. I don't think that we can get away > with > > >>> that change and maintain our compatibility obligations. I'm not up to > > >>> speed on the current state of CVEs for this older (EOL?) version, so > > >>> we have that dimension to consider. If the newer version is "drop-in" > > >>> compatible (and only if), then I have no issue with moving that > > >>> release line forward. Ultimately it's the release manager for 2.5 to > > >>> make a determination, so I defer to Andrew's assessment. > > >>> > > >>> I am in favor of backing-porting the improved testing coverage you've > > >>> added to branch-2.5. It would be great to understand if branch-2.5 > (1) > > >>> compiled against 3.2.4 will run on Hadoop 3.4.1 and (2) builds and > > >>> tests out on 3.4.1. That will give the more security-minded users > > >>> additional confidence in bumping their hadoop dependency on their > own. > > >>> > > >>> Thanks, > > >>> Nick > > >>> > > >>> > On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth <st...@cloudera.com> > > >>> wrote: > > >>> > > > >>> > > We could also move the default to 3.4.1 directly. > > >>> > > We already test for 3.4.0 in the nightly job. > > >>> > > > > >>> > > On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) < > palomino...@gmail.com > > >>> > > > >>> > > wrote: > > >>> > > > > >>> > >> And seems hadoop 3.4.1 is out. we could see whether to bump to > this > > >>> > >> version later? > > >>> > >> > > >>> > >> Istvan Toth <st...@cloudera.com.invalid> 于2024年10月21日周一 > 20:56写道： > > >>> > >> > > > >>> > >> > I have merged the new tests to the nightly Jenkins runs on > master. > > >>> > >> > > > >>> > >> > They have identified another 3.4.0 incompatibility: > > >>> > >> > HBASE-28929 < > https://issues.apache.org/jira/browse/HBASE-28929> > > >>> > >> > > > >>> > >> > I will hold off backporting the test changes until > HBASE-28929 is > > >>> > >> resolved. > > >>> > >> > > >>> > > > > >>> > > > > >>> > > -- > > >>> > > *István Tóth* | Sr. Staff Software Engineer > > >>> > > *Email*: st...@cloudera.com > > >>> > > cloudera.com <https://www.cloudera.com> > > >>> > > [image: Cloudera] <https://www.cloudera.com/> > > >>> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > [image: > > >>> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > [image: > > >>> > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera > > > > >>> > > ------------------------------ > > >>> > > ------------------------------ > > >>> > > > > >>> > > > >>> > > > >>> > -- > > >>> > *István Tóth* | Sr. Staff Software Engineer > > >>> > *Email*: st...@cloudera.com > > >>> > cloudera.com <https://www.cloudera.com> > > >>> > [image: Cloudera] <https://www.cloudera.com/> > > >>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > [image: > > >>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > >>> Cloudera > > >>> > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >>> > ------------------------------ > > >>> > ------------------------------ > > >>> > > >> > > >> > > >> -- > > >> *István Tóth* | Sr. Staff Software Engineer > > >> *Email*: st...@cloudera.com > > >> cloudera.com <https://www.cloudera.com> > > >> [image: Cloudera] <https://www.cloudera.com/> > > >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > >> ------------------------------ > > >> ------------------------------ > > >> > > > > > > > > > -- > > > *István Tóth* | Sr. Staff Software Engineer > > > *Email*: st...@cloudera.com > > > cloudera.com <https://www.cloudera.com> > > > [image: Cloudera] <https://www.cloudera.com/> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > ------------------------------ > > > ------------------------------ > > > > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: st...@cloudera.com > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------ > > > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
