I was too hasty to reply. You did not suggest dropping support for older releases in the released branches.
Still, I would keep Hadoop 3.3.x support at least on branch-2. We effectively need to support Hadoop 3.3.x for 2.5 and 2.6 anyway, so supporting 3.3 on more branches only costs us some machine time for running the nightly tests. On Fri, Sep 27, 2024 at 7:26 PM Istvan Toth <st...@cloudera.com> wrote: > I am not sure that we should drop support for all releases with CVEs. > That could easily lead to not having any CVE-less Hadoop releases to > support for some periods of time. > > Once I manage the proper backwards test matrix working, we could support > more releases in parallel. > > I would suggest deciding on a set of Hadoop releases when we release a new > minor version, > and trying to keep support for those for the life of that HBase minor > version, regardless of any CVEs. (We can of course add support for newer > Hadoop versions) > > We do something similar in Phoenix WRT HBase versions. > > If the default is the newest Hadoop release, then our binaries will be as > CVE-free as possible, while not forcing existing users to upgrade Hadoop > before they upgrade HBase. > > Istvan > > > > > > > On Fri, Sep 27, 2024 at 2:07 PM 张铎(Duo Zhang) <palomino...@gmail.com> > wrote: > >> There is a CVE for hadoop version less than 3.4.0... >> >> https://nvd.nist.gov/vuln/detail/CVE-2024-23454 >> >> Maybe we can only support hadoop 3.4.0 on master, branch-3 and >> branch-2(for hadoop 3) now? >> >> Thanks. >> >> Istvan Toth <st...@cloudera.com.invalid> 于2024年9月27日周五 16:16写道： >> > >> > I'm still working on this, but I struggle a lot with the Jenkinsfile, >> and >> > some Yetus bugs also make testing hard. >> > >> > On Thu, Sep 19, 2024 at 9:25 AM Istvan Toth <st...@cloudera.com> wrote: >> > >> > > Master is still going to support 3.3.5, so master still needs to use >> > > reflection as well. >> > > >> > > The point of these changes is allowing Hbase to default to a newer >> Hadoop >> > > version, without dropping support for older releases. >> > > >> > > Dropping support for 3.3.5 would be a different discussion, and I >> > > personally feel that it would be too early. >> > > >> > > Istvan >> > > >> > > On Wed, Sep 18, 2024 at 7:46 PM Wei-Chiu Chuang >> > > <weic...@cloudera.com.invalid> wrote: >> > > >> > >> So....I was wondering now that we'll be using Hadoop 3.4.0, if it's >> okay >> > >> to >> > >> port HBASE-27769 <https://issues.apache.org/jira/browse/HBASE-27769> >> to >> > >> the >> > >> master branch. >> > >> This will allow Ozone to be used by HBase. We are preparing Apache >> Ozone >> > >> 2.0 release and having a usable Apache HBase to work with is >> important. It >> > >> is working now with Cloudera's HBase work but I'd like to open up >> this >> > >> opportunity to the community as well. >> > >> >> > >> We can start with master, and then I can find a solution (something >> that >> > >> involves the use of reflection) and backport to lower branches. >> > >> Ultimately, >> > >> release a version of HBase with this feature. >> > >> >> > >> cc: Stephen. >> > >> >> > >> On Wed, Sep 18, 2024 at 12:08 AM Istvan Toth >> <st...@cloudera.com.invalid> >> > >> wrote: >> > >> >> > >> > Created a new ticket, as the old one was for 3.3.6 but we've >> agreed on >> > >> > 3.4.0, and expanded the scope. >> > >> > >> > >> > https://issues.apache.org/jira/browse/HBASE-28846 >> > >> > >> > >> > On Tue, Sep 17, 2024 at 3:47 PM 张铎(Duo Zhang) < >> palomino...@gmail.com> >> > >> > wrote: >> > >> > >> > >> > > I think we could start the work from branch-2.6? Branch-2.5 >> should >> > >> > > reach its EOL soon, after 2.6.x is stable enough. >> > >> > > >> > >> > > In this way we only need to deal with 3.3.x and 3.4.x. >> > >> > > >> > >> > > Istvan Toth <st...@cloudera.com.invalid> 于2024年9月17日周二 16:56写道： >> > >> > > > >> > >> > > > Thanks for the assessment, Wei-Chiu. >> > >> > > > >> > >> > > > Transitive dependency updates in Hadoop are normal (desired >> even), >> > >> > that's >> > >> > > > something that HBase needs to manage. >> > >> > > > >> > >> > > > As for the test: >> > >> > > > >> > >> > > > - Duo's suggestion is to extend the Hadoop compatibility >> tests, and >> > >> run >> > >> > > > them with multiple Hadoop 3 releases. >> > >> > > > Looking at the nightly results, those tests are fast, it takes >> 14 >> > >> > minutes >> > >> > > > for Hadoop2 and Hadoop3. >> > >> > > > I've peeked into hbase_nightly_pseudo-distributed-test.sh , the >> > >> tests >> > >> > > there >> > >> > > > are indeed quite minimal, >> > >> > > > more of a smoke test, and seem to be targeted to check the >> shaded >> > >> > > artifacts. >> > >> > > > >> > >> > > > - Nick's suggestion is to run DevTests and set the Hadoop >> version. >> > >> > > > The runAllTests step in the current nightly takes 8+ hours. >> > >> > > > On my 6+8 core laptop, my last attempt failed after 90 >> minutes, so >> > >> > let's >> > >> > > > say the full run takes 120 minutes. >> > >> > > > >> > >> > > > I don't know how many free resources HBase has, but if we can >> > >> utilize >> > >> > > > another VM per branch we could run the dev tests with four >> HBase >> > >> > > versions, >> > >> > > > and still finish about the same time as the full test job does. >> > >> > > > >> > >> > > > We don't need to test with the default version, as we already >> run >> > >> the >> > >> > > full >> > >> > > > suite for that one. >> > >> > > > >> > >> > > > Assuming that we officially support 3.4.0 on all active >> branches, >> > >> and >> > >> > > also >> > >> > > > default to 3.4.0 on all branches, and trusting Hadoop's >> > >> compatibility >> > >> > so >> > >> > > > that we don't need to test interim >> > >> > > > patch releases within a minor version, we could go with these >> > >> versions: >> > >> > > > >> > >> > > > branch-2.5 : 3.2.3, 3.2.4, 3.3.2, 3.3.6 >> > >> > > > branch-2.6 : 3.3.5, 3.3.6 >> > >> > > > branch-3: 3.3.5, 3.3.6 >> > >> > > > branch-4: 3.3.5, 3.3.6 >> > >> > > > >> > >> > > > If we trust Hadoop not to break compatibility in patch >> releases, we >> > >> > > could >> > >> > > > reduce this to only the oldest patch releases: >> > >> > > > >> > >> > > > branch-2.5 : 3.2.3, 3.3.2 >> > >> > > > branch-2.6 : 3.3.5 >> > >> > > > branch-3: 3.3.5 >> > >> > > > branch-4: 3.3.5 >> > >> > > > >> > >> > > > or if we trust it not break compatibility in specific minor >> > >> versions, >> > >> > we >> > >> > > > could further reduce it to just the oldest supported release: >> > >> > > > >> > >> > > > branch-2.5 : 3.2.3 >> > >> > > > branch-2.6 : 3.3.5 >> > >> > > > branch-3: 3.3.5 >> > >> > > > branch-4: 3.3.5 >> > >> > > > >> > >> > > > Of course running every devTest is an overkill, as the vast >> > >> majority of >> > >> > > the >> > >> > > > tests use the same set of Hadoop APIs and features, and we'd >> only >> > >> > really >> > >> > > > need to run the tests that cover that feature set. >> > >> > > > Figuring out a subset of tests that exercise the full Hadoop >> API >> > >> (that >> > >> > we >> > >> > > > use) is a hard and error prone task, so if we have the >> resources, we >> > >> > can >> > >> > > > just brute force it with devTests. >> > >> > > > >> > >> > > > As a base for further discussion: >> > >> > > > >> > >> > > > Let's take the first (first and last supported patch level for >> each >> > >> > minor >> > >> > > > release) set of versions, >> > >> > > > and run both the pseudistributed tests and the devTests on >> them. >> > >> > > > >> > >> > > > Does that sound good ? Do we have the resources for that ? Do >> we >> > >> have a >> > >> > > > better idea ? >> > >> > > > >> > >> > > > Istvan >> > >> > > > >> > >> > > > On Mon, Sep 16, 2024 at 7:20 PM Wei-Chiu Chuang < >> weic...@apache.org >> > >> > >> > >> > > wrote: >> > >> > > > >> > >> > > > > I strive to meet that stated compatibility goal when I >> release >> > >> > Hadoop. >> > >> > > > > But we don't have a rigorous compatibility/upgrade test in >> Hadoop >> > >> so >> > >> > > YMMV >> > >> > > > > (we now have in Ozone!) >> > >> > > > > >> > >> > > > > There are so many gotchas that it really depends on the RM >> to do >> > >> the >> > >> > > > > hardwork, checking protobuf definitions, running API compat >> > >> report, >> > >> > > > > compiling against downstream applications. >> > >> > > > > The other thing is thirdparty dependency update. Whenever I >> bump >> > >> > Netty >> > >> > > or >> > >> > > > > Jetty version, new transitive dependencies slip in as part >> of the >> > >> > > update, >> > >> > > > > which sometimes break HBase because of the dependency check >> in >> > >> > shading. >> > >> > > > > >> > >> > > > > On Mon, Sep 16, 2024 at 4:48 AM Istvan Toth >> > >> > <st...@cloudera.com.invalid >> > >> > > > >> > >> > > > > wrote: >> > >> > > > > >> > >> > > > > > On Wed, Sep 11, 2024 at 4:30 PM 张铎(Duo Zhang) < >> > >> > palomino...@gmail.com >> > >> > > > >> > >> > > > > > wrote: >> > >> > > > > > >> > >> > > > > > > There is a problem that, usually, you can use an old >> hadoop >> > >> > client >> > >> > > to >> > >> > > > > > > communicate with a new hadoop server, but not vice versa. >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > Do we have examples of that ? >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > >> > >> > >> > >> >> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html >> > >> > > > > > specifically states otherwise: >> > >> > > > > > >> > >> > > > > > In addition to the limitations imposed by being Stable >> > >> > > > > > < >> > >> > > > > > >> > >> > > > > >> > >> > > >> > >> > >> > >> >> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/InterfaceClassification.html#Stable >> > >> > > > > > >, >> > >> > > > > > Hadoop’s wire protocols MUST also be forward compatible >> across >> > >> > minor >> > >> > > > > > releases within a major version according to the following: >> > >> > > > > > >> > >> > > > > > - Client-Server compatibility MUST be maintained so as >> to >> > >> allow >> > >> > > users >> > >> > > > > to >> > >> > > > > > continue using older clients even after upgrading the >> server >> > >> > > (cluster) >> > >> > > > > > to a >> > >> > > > > > later version (or vice versa). For example, a Hadoop >> 2.1.0 >> > >> > client >> > >> > > > > > talking >> > >> > > > > > to a Hadoop 2.3.0 cluster. >> > >> > > > > > - Client-Server compatibility MUST be maintained so as >> to >> > >> allow >> > >> > > users >> > >> > > > > to >> > >> > > > > > upgrade the client before upgrading the server >> (cluster). For >> > >> > > > > example, a >> > >> > > > > > Hadoop 2.4.0 client talking to a Hadoop 2.3.0 cluster. >> This >> > >> > allows >> > >> > > > > > deployment of client-side bug fixes ahead of full >> cluster >> > >> > > upgrades. >> > >> > > > > Note >> > >> > > > > > that new cluster features invoked by new client APIs or >> shell >> > >> > > commands >> > >> > > > > > will >> > >> > > > > > not be usable. YARN applications that attempt to use >> new APIs >> > >> > > > > (including >> > >> > > > > > new fields in data structures) that have not yet been >> > >> deployed >> > >> > to >> > >> > > the >> > >> > > > > > cluster can expect link exceptions. >> > >> > > > > > - Client-Server compatibility MUST be maintained so as >> to >> > >> allow >> > >> > > > > > upgrading individual components without upgrading >> others. For >> > >> > > example, >> > >> > > > > > upgrade HDFS from version 2.1.0 to 2.2.0 without >> upgrading >> > >> > > MapReduce. >> > >> > > > > > - Server-Server compatibility MUST be maintained so as >> to >> > >> allow >> > >> > > mixed >> > >> > > > > > versions within an active cluster so the cluster may be >> > >> upgraded >> > >> > > > > without >> > >> > > > > > downtime in a rolling fashion. >> > >> > > > > > >> > >> > > > > > Admittedly, I don't have a lot of experience with >> mismatched >> > >> Hadoop >> > >> > > > > > versions, but my proposal should be covered by the second >> > >> clause. >> > >> > > > > > >> > >> > > > > > Usage of newer APIs should be caught when compiling with >> older >> > >> > Hadoop >> > >> > > > > > versions. >> > >> > > > > > The only risk I can see is when we use a new feature which >> was >> > >> > added >> > >> > > > > > without changing the API signature (such as adding a new >> > >> constant >> > >> > > value >> > >> > > > > for >> > >> > > > > > some new behaviour) >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > When deploying HBase, HBase itself acts as a client of >> hadoop, >> > >> > > that's >> > >> > > > > > > why we always stay on the oldest support hadoop version. >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > Not true for 2.6 , which according to the docs supports >> Hadoop >> > >> 3.2, >> > >> > > but >> > >> > > > > > defaults to Hadoop 3.3 >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > For me, technically I think bumping to the newest patch >> > >> release >> > >> > of >> > >> > > a >> > >> > > > > > > minor release should be fine, which is the proposal 1. >> > >> > > > > > > >> > >> > > > > > > But the current hadoopcheck is not enough, since it can >> only >> > >> > ensure >> > >> > > > > > > that there is no complation error. >> > >> > > > > > > Maybe we should also run some simple dev tests in the >> > >> hadoopcheck >> > >> > > > > > > stage, and in integration tests, we should try to build >> with >> > >> all >> > >> > > the >> > >> > > > > > > support hadoop version and run the basic read write >> tests. >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > Do we need to test all versions ? >> > >> > > > > > If We test with say, 3.3.0 and 3.3.6 , do we need to test >> with >> > >> > > 3.3.[1-5] >> > >> > > > > ? >> > >> > > > > > Or if we test with 3.2.5 and 3.3.6, do we need to test >> with >> > >> any of >> > >> > > the >> > >> > > > > > interim versions ? >> > >> > > > > > >> > >> > > > > > Basically, how much do we trust Hadoop to keep to its >> > >> compatibility >> > >> > > > > rules ? >> > >> > > > > > >> > >> > > > > > Running a limited number of tests should not be a problem. >> > >> > > > > > Should we add a new test category, so that they can be >> easily >> > >> > started >> > >> > > > > from >> > >> > > > > > Maven ? >> > >> > > > > > >> > >> > > > > > Can you suggest some tests that we should run for the >> > >> compatibility >> > >> > > > > check ? >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > Thanks. >> > >> > > > > > > >> > >> > > > > > > Istvan Toth <st...@cloudera.com.invalid> 于2024年9月11日周三 >> > >> 21:05写道： >> > >> > > > > > > > >> > >> > > > > > > > Let me summarize my take of the discussion so far: >> > >> > > > > > > > >> > >> > > > > > > > There are two aspects to the HBase version we build >> with: >> > >> > > > > > > > 1. Source code quality/compatibility >> > >> > > > > > > > 2. Security and usability of the public binary >> assemblies >> > >> and >> > >> > > > > (shaded) >> > >> > > > > > > > hbase maven artifacts. >> > >> > > > > > > > >> > >> > > > > > > > 1. Source code quality/compatibility >> > >> > > > > > > > >> > >> > > > > > > > AFAICT we have the following hard goals: >> > >> > > > > > > > 1.a : Ensure that HBase compiles and runs well with the >> > >> earlier >> > >> > > > > > supported >> > >> > > > > > > > Hadoop version on the given branch >> > >> > > > > > > > 1.b: Ensure that HBase compiles and runs well with the >> > >> latest >> > >> > > > > supported >> > >> > > > > > > > Hadoop version on the given branch >> > >> > > > > > > > >> > >> > > > > > > > In my opinion we should also strive for these goals: >> > >> > > > > > > > 1.c: Aim to officially support the newest possible >> Hadoop >> > >> > > releases >> > >> > > > > > > > 1.d: Take advantage of new features in newer Hadoop >> > >> versions >> > >> > > > > > > > >> > >> > > > > > > > 2. Public binary usability wish list: >> > >> > > > > > > > >> > >> > > > > > > > 2.a: We want them to work OOB for as many use cases as >> > >> possible >> > >> > > > > > > > 2.b: We want to work them as well as possible >> > >> > > > > > > > 2.c: We want to have as few CVEs in them as possible >> > >> > > > > > > > 2.d: We want to make upgrades as painless as possible, >> > >> > > especially for >> > >> > > > > > > patch >> > >> > > > > > > > releases >> > >> > > > > > > > >> > >> > > > > > > > The factor that Hadoop does not have an explicit >> end-of-life >> > >> > > policy >> > >> > > > > of >> > >> > > > > > > > course complicates things. >> > >> > > > > > > > >> > >> > > > > > > > Our current policy seems to be that we pick a Hadoop >> > >> version to >> > >> > > build >> > >> > > > > > > with >> > >> > > > > > > > when releasing a minor version, >> > >> > > > > > > > and stay on that version until there is a newer patch >> > >> released >> > >> > of >> > >> > > > > that >> > >> > > > > > > > minor version with direct CVE fixes. >> > >> > > > > > > > This does not seem to be an absolute, for example the >> > >> recently >> > >> > > > > released >> > >> > > > > > > > HBase 2.4.18 still defaults to Hadoop 3.1.2, >> > >> > > > > > > > which has several old CVEs, many of which are >> reportedly >> > >> fixed >> > >> > in >> > >> > > > > 3.1.3 >> > >> > > > > > > and >> > >> > > > > > > > 3.1.4. >> > >> > > > > > > > >> > >> > > > > > > > my proposals are : >> > >> > > > > > > > >> > >> > > > > > > > Proposal 1: >> > >> > > > > > > > >> > >> > > > > > > > Whenever a new Hadoop patch release is released for a >> minor >> > >> > > version, >> > >> > > > > > then >> > >> > > > > > > > unless it breaks source compatibility, we should >> > >> automatically >> > >> > > update >> > >> > > > > > the >> > >> > > > > > > > default Hadoop version for >> > >> > > > > > > > all branches that use the same minor version. >> > >> > > > > > > > The existing hadoopcheck mechanism should be good >> enough to >> > >> > > guarantee >> > >> > > > > > > that >> > >> > > > > > > > we do not break compatibility with the earlier patch >> > >> releases. >> > >> > > > > > > > >> > >> > > > > > > > This would ensure that the binaries use the latest and >> > >> greatest >> > >> > > > > Hadoop >> > >> > > > > > > (of >> > >> > > > > > > > that minor branch) and that users of the binaries get >> the >> > >> > latest >> > >> > > > > fixes, >> > >> > > > > > > > both CVE and functionality wise, and >> > >> > > > > > > > the binaries also get the transitive CVE fixes in that >> > >> release. >> > >> > > > > > > > For example,if we did this we could use the new >> feature in >> > >> > > 3.3.6 in >> > >> > > > > > > > HBASE-27769 (via reflection) and also test it, thereby >> > >> > improving >> > >> > > > > Ozone >> > >> > > > > > > > support. >> > >> > > > > > > > >> > >> > > > > > > > On the other hand we minimize changes and maximize >> > >> > compatibility >> > >> > > by >> > >> > > > > > > > sticking to the same Hadoop minor release. >> > >> > > > > > > > >> > >> > > > > > > > Proposal 2: >> > >> > > > > > > > >> > >> > > > > > > > We should default to the latest hadoop version >> (currently >> > >> > 3.4.0) >> > >> > > on >> > >> > > > > > > > unreleased branches. >> > >> > > > > > > > This should ensure that when we do release we default >> to the >> > >> > > latest >> > >> > > > > > > > version, and we've tested it as thoroughly as possible. >> > >> > > > > > > > >> > >> > > > > > > > Again. the existing Hadoopcheck mechanism should ensure >> > >> that we >> > >> > > do >> > >> > > > > not >> > >> > > > > > > > break compatibility with earlier supported versions. >> > >> > > > > > > > >> > >> > > > > > > > Istvan >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > On Mon, Sep 9, 2024 at 9:41 PM Nick Dimiduk < >> > >> > ndimi...@apache.org >> > >> > > > >> > >> > > > > > wrote: >> > >> > > > > > > > >> > >> > > > > > > > > Yes, we’ll use reflection to make use of APIs >> introduced >> > >> in >> > >> > > newer >> > >> > > > > > HDFS >> > >> > > > > > > > > versions than the stated dependency until the stated >> > >> > dependency >> > >> > > > > > finally >> > >> > > > > > > > > catches up. >> > >> > > > > > > > > >> > >> > > > > > > > > On Mon, 9 Sep 2024 at 19:55, Wei-Chiu Chuang < >> > >> > > weic...@apache.org> >> > >> > > > > > > wrote: >> > >> > > > > > > > > >> > >> > > > > > > > > > Reflection is probably the way to go to ensure >> maximum >> > >> > > > > > compatibility >> > >> > > > > > > TBH >> > >> > > > > > > > > > >> > >> > > > > > > > > > On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth >> > >> > > > > > > <st...@cloudera.com.invalid> >> > >> > > > > > > > > > wrote: >> > >> > > > > > > > > > >> > >> > > > > > > > > > > Stephen Wu has kindly sent me the link for the >> > >> previous >> > >> > > email >> > >> > > > > > > thread: >> > >> > > > > > > > > > > >> > >> > > > > >> https://lists.apache.org/thread/2k4tvz3wpg06sgkynkhgvxrodmj86vsj >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > Reading it, I cannot see anything there that >> would >> > >> > > > > contraindicate >> > >> > > > > > > > > > upgrading >> > >> > > > > > > > > > > to 3.3.6 from 3.3.5, at least on the branches >> that >> > >> > already >> > >> > > > > > default >> > >> > > > > > > to >> > >> > > > > > > > > > > 3.3.5, i.e. 2.6+. >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > At first glance, the new logic in HBASE-27769 >> could >> > >> also >> > >> > be >> > >> > > > > > > implemented >> > >> > > > > > > > > > > with the usual reflection hacks, while >> preserving the >> > >> old >> > >> > > logic >> > >> > > > > > for >> > >> > > > > > > > > > Hadoop >> > >> > > > > > > > > > > 3.3.5 and earlier. >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > Thanks, >> > >> > > > > > > > > > > Istvan >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > On Mon, Sep 9, 2024 at 1:42 PM Istvan Toth < >> > >> > > st...@cloudera.com >> > >> > > > > > >> > >> > > > > > > wrote: >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > > Thanks for your reply, Nick. >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > There are no listed direct CVEs in either >> Hadoop >> > >> 3.2.4 >> > >> > or >> > >> > > > > > 3.3.5, >> > >> > > > > > > but >> > >> > > > > > > > > > > there >> > >> > > > > > > > > > > > are CVEs in their transitive dependencies. >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > My impression is that rather than shipping the >> > >> oldest >> > >> > > 'safe' >> > >> > > > > > > version, >> > >> > > > > > > > > > > > HBase does seem to update the default Hadoop >> > >> version to >> > >> > > the >> > >> > > > > > > > > latest-ish >> > >> > > > > > > > > > at >> > >> > > > > > > > > > > > the time of the start >> > >> > > > > > > > > > > > of the release process, otherwise 2.6 would >> still >> > >> > > default to >> > >> > > > > > > 3.2.4. >> > >> > > > > > > > > > > (HBase >> > >> > > > > > > > > > > > 2.6 release was already underway when Hadoop >> 3.4.0 >> > >> was >> > >> > > > > > released) >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > For now, we (Phoenix) have resorted to >> dependency >> > >> > > managing >> > >> > > > > > > transitive >> > >> > > > > > > > > > > > dependencies coming in (only) via Hadoop in >> Phoenix, >> > >> > > > > > > > > > > > but that is a slippery slope, and adds a layer >> of >> > >> > > > > uncertainty, >> > >> > > > > > > as it >> > >> > > > > > > > > > may >> > >> > > > > > > > > > > > introduce incompatibilities in Hadoop that we >> don't >> > >> > have >> > >> > > > > tests >> > >> > > > > > > for. >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > Our situation is similar to that of the HBase >> shaded >> > >> > > > > artifacts, >> > >> > > > > > > where >> > >> > > > > > > > > > we >> > >> > > > > > > > > > > > ship a huge uberjar that includes much of both >> HBase >> > >> > and >> > >> > > > > Hadoop >> > >> > > > > > > on >> > >> > > > > > > > > top >> > >> > > > > > > > > > of >> > >> > > > > > > > > > > > (or rather below) Phoenix, >> > >> > > > > > > > > > > > similar to the hbase-client-shaded jar. >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > I will look into to hadoop check CI tests that >> > >> you've >> > >> > > > > > mentioned, >> > >> > > > > > > > > then I >> > >> > > > > > > > > > > > will try to resurrect HBASE-27931, and if I >> don't >> > >> find >> > >> > > any >> > >> > > > > > > issues, >> > >> > > > > > > > > and >> > >> > > > > > > > > > > > there are no objections, then >> > >> > > > > > > > > > > > I will put a PR to update the unreleased >> version to >> > >> > > default >> > >> > > > > to >> > >> > > > > > > 3.4.0. >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > Istvan >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > On Mon, Sep 9, 2024 at 11:06 AM Nick Dimiduk < >> > >> > > > > > > ndimi...@apache.org> >> > >> > > > > > > > > > > wrote: >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > >> My understanding of our hadoop dependency >> policy is >> > >> > > that we >> > >> > > > > > ship >> > >> > > > > > > > > poms >> > >> > > > > > > > > > > with >> > >> > > > > > > > > > > >> hadoop versions pinned to the oldest >> compatible, >> > >> > "safe" >> > >> > > > > > version >> > >> > > > > > > that >> > >> > > > > > > > > > is >> > >> > > > > > > > > > > >> supported. Our test infrastructure has a >> "hadoop >> > >> > check" >> > >> > > > > > > procedure >> > >> > > > > > > > > that >> > >> > > > > > > > > > > >> does >> > >> > > > > > > > > > > >> some validation against other patch release >> > >> versions. >> > >> > > > > > > > > > > >> >> > >> > > > > > > > > > > >> I don't know if anyone has done a CVE sweep >> > >> recently. >> > >> > If >> > >> > > > > there >> > >> > > > > > > are >> > >> > > > > > > > > new >> > >> > > > > > > > > > > >> CVEs, we do bump the minimum supported version >> > >> > > specified in >> > >> > > > > > the >> > >> > > > > > > pom >> > >> > > > > > > > > as >> > >> > > > > > > > > > > >> part >> > >> > > > > > > > > > > >> of patch releases. These changes need to >> include a >> > >> > > pretty >> > >> > > > > > > thorough >> > >> > > > > > > > > > > >> compatibility check so that we can include >> release >> > >> > notes >> > >> > > > > about >> > >> > > > > > > any >> > >> > > > > > > > > > > >> introduced incompatibilities. >> > >> > > > > > > > > > > >> >> > >> > > > > > > > > > > >> I am in favor of a dependency bump so as to >> address >> > >> > > known >> > >> > > > > CVEs >> > >> > > > > > > as >> > >> > > > > > > > > best >> > >> > > > > > > > > > > as >> > >> > > > > > > > > > > >> we reasonably can. >> > >> > > > > > > > > > > >> >> > >> > > > > > > > > > > >> Thanks, >> > >> > > > > > > > > > > >> Nick >> > >> > > > > > > > > > > >> >> > >> > > > > > > > > > > >> On Mon, Sep 9, 2024 at 10:59 AM Istvan Toth < >> > >> > > > > st...@apache.org >> > >> > > > > > > >> > >> > > > > > > > > wrote: >> > >> > > > > > > > > > > >> >> > >> > > > > > > > > > > >> > Hi! >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> > I'm working on building the Phoenix >> uberjars with >> > >> > > newer >> > >> > > > > > Hadoop >> > >> > > > > > > > > > > versions >> > >> > > > > > > > > > > >> by >> > >> > > > > > > > > > > >> > default to improve its CVE stance, and I >> realized >> > >> > that >> > >> > > > > HBase >> > >> > > > > > > > > itself >> > >> > > > > > > > > > > does >> > >> > > > > > > > > > > >> > not use the latest releases. >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> > branch-2.5 defaults to 3.2.4 >> > >> > > > > > > > > > > >> > branch-2.6 and later defaults to 3.3.5 >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> > I can kind of understand that we don't want >> to >> > >> bump >> > >> > > the >> > >> > > > > > minor >> > >> > > > > > > > > > version >> > >> > > > > > > > > > > >> for >> > >> > > > > > > > > > > >> > branch-2.5 from the one it was released >> with. >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> > However, I don't see the rationale for not >> > >> upgrading >> > >> > > > > > > branch-2.6 to >> > >> > > > > > > > > > at >> > >> > > > > > > > > > > >> least >> > >> > > > > > > > > > > >> > 3.3.6, and the unreleased branches >> (branch-2, >> > >> > > branch-3, >> > >> > > > > > > master) to >> > >> > > > > > > > > > > >> 3.4.0. >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> > I found a mention of wanting to stay off the >> > >> latest >> > >> > > patch >> > >> > > > > > > release >> > >> > > > > > > > > > > >> > HBASE-27931, but I could not figure if it >> has a >> > >> > > technical >> > >> > > > > > > reason, >> > >> > > > > > > > > or >> > >> > > > > > > > > > > if >> > >> > > > > > > > > > > >> > this is a written (or unwritten) policy. >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> > best regards >> > >> > > > > > > > > > > >> > Istvan >> > >> > > > > > > > > > > >> > >> > >> > > > > > > > > > > >> >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > -- >> > >> > > > > > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > >> > > > > > > > > > > > *Email*: st...@cloudera.com >> > >> > > > > > > > > > > > cloudera.com <https://www.cloudera.com> >> > >> > > > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > >> > > > > > > > > > > > [image: Cloudera on Twitter] < >> > >> > > https://twitter.com/cloudera> >> > >> > > > > > > [image: >> > >> > > > > > > > > > > > Cloudera on Facebook] < >> > >> > https://www.facebook.com/cloudera >> > >> > > > >> > >> > > > > > > [image: >> > >> > > > > > > > > > > > Cloudera on LinkedIn] < >> > >> > > > > > https://www.linkedin.com/company/cloudera >> > >> > > > > > > > >> > >> > > > > > > > > > > > ------------------------------ >> > >> > > > > > > > > > > > ------------------------------ >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > -- >> > >> > > > > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > >> > > > > > > > > > > *Email*: st...@cloudera.com >> > >> > > > > > > > > > > cloudera.com <https://www.cloudera.com> >> > >> > > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > >> > > > > > > > > > > [image: Cloudera on Twitter] < >> > >> > https://twitter.com/cloudera >> > >> > > > >> > >> > > > > > > [image: >> > >> > > > > > > > > > > Cloudera on Facebook] < >> > >> https://www.facebook.com/cloudera >> > >> > > >> > >> > > > > > [image: >> > >> > > > > > > > > > Cloudera >> > >> > > > > > > > > > > on LinkedIn] < >> > >> https://www.linkedin.com/company/cloudera> >> > >> > > > > > > > > > > ------------------------------ >> > >> > > > > > > > > > > ------------------------------ >> > >> > > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > -- >> > >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > >> > > > > > > > *Email*: st...@cloudera.com >> > >> > > > > > > > cloudera.com <https://www.cloudera.com> >> > >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > >> > > > > > > > [image: Cloudera on Twitter] < >> https://twitter.com/cloudera> >> > >> > > [image: >> > >> > > > > > > > Cloudera on Facebook] < >> https://www.facebook.com/cloudera> >> > >> > > [image: >> > >> > > > > > > Cloudera >> > >> > > > > > > > on LinkedIn] < >> https://www.linkedin.com/company/cloudera> >> > >> > > > > > > > ------------------------------ >> > >> > > > > > > > ------------------------------ >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > -- >> > >> > > > > > *István Tóth* | Sr. Staff Software Engineer >> > >> > > > > > *Email*: st...@cloudera.com >> > >> > > > > > cloudera.com <https://www.cloudera.com> >> > >> > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > >> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera >> > >> > >> > [image: >> > >> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> >> > >> [image: >> > >> > > > > Cloudera >> > >> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > >> > > > > > ------------------------------ >> > >> > > > > > ------------------------------ >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > > -- >> > >> > > > *István Tóth* | Sr. Staff Software Engineer >> > >> > > > *Email*: st...@cloudera.com >> > >> > > > cloudera.com <https://www.cloudera.com> >> > >> > > > [image: Cloudera] <https://www.cloudera.com/> >> > >> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> >> [image: >> > >> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> >> [image: >> > >> > > Cloudera >> > >> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > >> > > > ------------------------------ >> > >> > > > ------------------------------ >> > >> > > >> > >> > >> > >> > >> > >> > -- >> > >> > *István Tóth* | Sr. Staff Software Engineer >> > >> > *Email*: st...@cloudera.com >> > >> > cloudera.com <https://www.cloudera.com> >> > >> > [image: Cloudera] <https://www.cloudera.com/> >> > >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> >> [image: >> > >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> > >> Cloudera >> > >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > >> > ------------------------------ >> > >> > ------------------------------ >> > >> > >> > >> >> > > >> > > >> > > -- >> > > *István Tóth* | Sr. Staff Software Engineer >> > > *Email*: st...@cloudera.com >> > > cloudera.com <https://www.cloudera.com> >> > > [image: Cloudera] <https://www.cloudera.com/> >> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > > ------------------------------ >> > > ------------------------------ >> > > >> > >> > >> > -- >> > *István Tóth* | Sr. Staff Software Engineer >> > *Email*: st...@cloudera.com >> > cloudera.com <https://www.cloudera.com> >> > [image: Cloudera] <https://www.cloudera.com/> >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > ------------------------------ >> > ------------------------------ >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
