Looking at the nightly tests, the test stability has decreased. AFAICT there is not much of a pattern to the test failures, and most of the failures seem to be minicluster startup failures. Some of the failures are for Hadoop2, and Hadoop 3.3.5 and 3.3.6 seems as prone to the failures as 3.4.0 and 3.4.1, so I don't think these are directly caused by the upgrade. Some of this may be explained by us running much more tests now, and thus having a higher chance of triggering random/transient errors in them. Unfortunately, debugging the minicluster failures would require the full debug log from the failing tests, which none of the jobs save.
On Mon, Oct 28, 2024 at 10:58 AM Istvan Toth <st...@cloudera.com> wrote: > I have backported the compatibility checks into the nightlies down to > branch-2.6. > As soon as the tests are stable, I will also backport the change for > bumping the default version to 3.4.1 in branch-2.6 and later. > > I have looked at branch-2.5, but the nightly looks off there, as it runs > the packaging tests with Hadoop 3.1.1, which it doesn't even officially > support anymore. > > What should we do with branch-2.5 ? > > I think that it would not be a lot of extra work to backport everything, > both the backwards compatibility tests and defaulting Hadoop to 3.4.1. > We just have to update the version in the pom, and add 3.2.4 to the list > of versions to test for backwards compatibility and integration (and remove > 3.1.1). > > I would prefer to have uniform tests and default to Hadoop 3.4.1 on all > active branches. > Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x support may > be useful for users for migration and CVE mitigation purposes. > > WDYT ? > > On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth <st...@cloudera.com> wrote: > >> We could also move the default to 3.4.1 directly. >> We already test for 3.4.0 in the nightly job. >> >> On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) <palomino...@gmail.com> >> wrote: >> >>> And seems hadoop 3.4.1 is out. we could see whether to bump to this >>> version later? >>> >>> Istvan Toth <st...@cloudera.com.invalid> 于2024年10月21日周一 20:56写道： >>> > >>> > I have merged the new tests to the nightly Jenkins runs on master. >>> > >>> > They have identified another 3.4.0 incompatibility: >>> > HBASE-28929 <https://issues.apache.org/jira/browse/HBASE-28929> >>> > >>> > I will hold off backporting the test changes until HBASE-28929 is >>> resolved. >>> >> >> >> -- >> *István Tóth* | Sr. Staff Software Engineer >> *Email*: st...@cloudera.com >> cloudera.com <https://www.cloudera.com> >> [image: Cloudera] <https://www.cloudera.com/> >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> >> ------------------------------ >> ------------------------------ >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
