With Duo's fix for HBASE-28965 , there are no more known blockers. Should I go ahead with making 3.4.1 the default for branch-2.5 and branch-2.6 ?
On Tue, Nov 5, 2024 at 7:16 AM Istvan Toth <st...@cloudera.com> wrote: > I've just committed the - hopefully - last test change, so the nightlies > will not always fail on the packaging tests. > > All non-released branches (i.e. 2.7+ now default to building with 3.4.1). > > I wanted to revisit updating the default version on the released (2.5, > 2.6) branches, because Nick has expressed concerns about it. > > The dev tests are good (the test failures seem to be "normal" flakies). As > we've not updated the default version, we're not yet running the packaging > tests > on branch-2.5 and 2.6 against the older Hadoop versions, but we do on the > rest of branches, and if we update it, we will also run them on 2.5 and 2.6. > > Without repeating the arguments I have already made for it, I want to add > a new one: > > Security and CVEs are getting more and more emphasis, which is great, but > has some drawbacks. > While the proliferation of static analyzers leads to a lot of frustrating > CVE witch hunts and false positives, the majority of users cannot evaluate > the actual security impact, > or even if they can, they are tied by inflexible policies. > > We at Phoenix had recent security discussions with Trino, and ended up > having to dependencyManage the transitive Hadoop dependencies in our shaded > uberjar to address their concerns. > (Which is an antipattern) > > While updating the Hadoop version in a patch release undoubtedly increases > the risk of regressions, IMO we are protected by the Hadoop backwards > compatibility promises, and we have added a reasonable number of tests to > catch any issues. I am confident in our test coverage when building HBase > with any of the supported HBase versions. We also have the packaging > (smoke) tests for running the official binaries against earlier Hadoop > clusters, which would catch (some of) the Hadoop wire api incompatibilities > . > > However, IMO not updating Hadoop is net negative to the project's health, > as the binary (maven or assembly) releases are used primarily either by > other libraries interfacing with HBase, or by new users, POC clusters, etc. > and these are the use cases where the transitive CVEs can prevent projects > from adding (or maintaining as in the case of Trino) HBase support, or > discourage new users from adopting HBase. > > Obviously, I have a very skewed view of HBase users, but I think that most > production HBase users either use a vendor or cloud provider version, or > have enough in-house expertise to rebuild HBase with their Hadoop version > if something goes wrong (despite the Hadoop backwards compatibility > promises) > > > > > > On Wed, Oct 30, 2024 at 12:41 PM Istvan Toth <st...@cloudera.com> wrote: > >> Thanks! >> >> I will backport the test changes, but keep the default Hadoop version. >> >> We will have more information then. >> >> Istvan >> >> On Wed, Oct 30, 2024 at 10:22 AM Nick Dimiduk <ndimi...@apache.org> >> wrote: >> >>> On Mon, Oct 28, 2024 at 11:00 AM Istvan Toth <st...@cloudera.com.invalid> >>> wrote: >>> > >>> > I have looked at branch-2.5, but the nightly looks off there, as it >>> runs >>> > the packaging tests with Hadoop 3.1.1, which it doesn't even >>> officially >>> > support anymore. >>> > >>> > What should we do with branch-2.5 ? >>> > >>> > I think that it would not be a lot of extra work to backport >>> everything, >>> > both the backwards compatibility tests and defaulting Hadoop to 3.4.1. >>> > We just have to update the version in the pom, and add 3.2.4 to the >>> list of >>> > versions to test for backwards compatibility and integration (and >>> remove >>> > 3.1.1). >>> > >>> > I would prefer to have uniform tests and default to Hadoop 3.4.1 on all >>> > active branches. >>> > Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x support >>> may >>> > be useful for users for migration and CVE mitigation purposes. >>> > >>> > WDYT ? >>> >>> branch-2.5's default hadoop3 version is 3.2.4. That's a big dependency >>> to change for a patch release. I don't think that we can get away with >>> that change and maintain our compatibility obligations. I'm not up to >>> speed on the current state of CVEs for this older (EOL?) version, so >>> we have that dimension to consider. If the newer version is "drop-in" >>> compatible (and only if), then I have no issue with moving that >>> release line forward. Ultimately it's the release manager for 2.5 to >>> make a determination, so I defer to Andrew's assessment. >>> >>> I am in favor of backing-porting the improved testing coverage you've >>> added to branch-2.5. It would be great to understand if branch-2.5 (1) >>> compiled against 3.2.4 will run on Hadoop 3.4.1 and (2) builds and >>> tests out on 3.4.1. That will give the more security-minded users >>> additional confidence in bumping their hadoop dependency on their own. >>> >>> Thanks, >>> Nick >>> >>> > On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth <st...@cloudera.com> >>> wrote: >>> > >>> > > We could also move the default to 3.4.1 directly. >>> > > We already test for 3.4.0 in the nightly job. >>> > > >>> > > On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) <palomino...@gmail.com >>> > >>> > > wrote: >>> > > >>> > >> And seems hadoop 3.4.1 is out. we could see whether to bump to this >>> > >> version later? >>> > >> >>> > >> Istvan Toth <st...@cloudera.com.invalid> 于2024年10月21日周一 20:56写道： >>> > >> > >>> > >> > I have merged the new tests to the nightly Jenkins runs on master. >>> > >> > >>> > >> > They have identified another 3.4.0 incompatibility: >>> > >> > HBASE-28929 <https://issues.apache.org/jira/browse/HBASE-28929> >>> > >> > >>> > >> > I will hold off backporting the test changes until HBASE-28929 is >>> > >> resolved. >>> > >> >>> > > >>> > > >>> > > -- >>> > > *István Tóth* | Sr. Staff Software Engineer >>> > > *Email*: st...@cloudera.com >>> > > cloudera.com <https://www.cloudera.com> >>> > > [image: Cloudera] <https://www.cloudera.com/> >>> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >>> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >>> > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> >>> > > ------------------------------ >>> > > ------------------------------ >>> > > >>> > >>> > >>> > -- >>> > *István Tóth* | Sr. Staff Software Engineer >>> > *Email*: st...@cloudera.com >>> > cloudera.com <https://www.cloudera.com> >>> > [image: Cloudera] <https://www.cloudera.com/> >>> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >>> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >>> Cloudera >>> > on LinkedIn] <https://www.linkedin.com/company/cloudera> >>> > ------------------------------ >>> > ------------------------------ >>> >> >> >> -- >> *István Tóth* | Sr. Staff Software Engineer >> *Email*: st...@cloudera.com >> cloudera.com <https://www.cloudera.com> >> [image: Cloudera] <https://www.cloudera.com/> >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> >> ------------------------------ >> ------------------------------ >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
