OK, I'm gonna look into downloading multiple Hadoop 3 versions, and running those tests with each one.
On Mon, Sep 16, 2024 at 3:08 PM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > And if we can make sure the compatibility, I agree that we could > depend on the newest possible hadoop version by default. As you said, > it can reduce most transitive security issues. > > There are still 3 security issues on master branch because of netty 3, > which should be fixed in 3.4.0. > > 张铎(Duo Zhang) <palomino...@gmail.com> 于2024年9月16日周一 21:03写道： > > > > There is a devTests profile in our pom, we can make use of it first. > > > > And on integration tests, I mean this one > > > > > https://github.com/apache/hbase/blob/4446d297112899dab59c0952489457c4419366d3/dev-support/Jenkinsfile#L755 > > > > We could extend this test to test different combinations. > > > > Istvan Toth <st...@cloudera.com.invalid> 于2024年9月16日周一 19:48写道： > > > > > > On Wed, Sep 11, 2024 at 4:30 PM 张铎(Duo Zhang) <palomino...@gmail.com> > wrote: > > > > > > > There is a problem that, usually, you can use an old hadoop client to > > > > communicate with a new hadoop server, but not vice versa. > > > > > > > > > > Do we have examples of that ? > > > > > > > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html > > > specifically states otherwise: > > > > > > In addition to the limitations imposed by being Stable > > > < > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/InterfaceClassification.html#Stable > >, > > > Hadoop’s wire protocols MUST also be forward compatible across minor > > > releases within a major version according to the following: > > > > > > - Client-Server compatibility MUST be maintained so as to allow > users to > > > continue using older clients even after upgrading the server > (cluster) to a > > > later version (or vice versa). For example, a Hadoop 2.1.0 client > talking > > > to a Hadoop 2.3.0 cluster. > > > - Client-Server compatibility MUST be maintained so as to allow > users to > > > upgrade the client before upgrading the server (cluster). For > example, a > > > Hadoop 2.4.0 client talking to a Hadoop 2.3.0 cluster. This allows > > > deployment of client-side bug fixes ahead of full cluster upgrades. > Note > > > that new cluster features invoked by new client APIs or shell > commands will > > > not be usable. YARN applications that attempt to use new APIs > (including > > > new fields in data structures) that have not yet been deployed to > the > > > cluster can expect link exceptions. > > > - Client-Server compatibility MUST be maintained so as to allow > > > upgrading individual components without upgrading others. For > example, > > > upgrade HDFS from version 2.1.0 to 2.2.0 without upgrading > MapReduce. > > > - Server-Server compatibility MUST be maintained so as to allow > mixed > > > versions within an active cluster so the cluster may be upgraded > without > > > downtime in a rolling fashion. > > > > > > Admittedly, I don't have a lot of experience with mismatched Hadoop > > > versions, but my proposal should be covered by the second clause. > > > > > > Usage of newer APIs should be caught when compiling with older Hadoop > > > versions. > > > The only risk I can see is when we use a new feature which was added > > > without changing the API signature (such as adding a new constant > value for > > > some new behaviour) > > > > > > > > > > When deploying HBase, HBase itself acts as a client of hadoop, that's > > > > why we always stay on the oldest support hadoop version. > > > > > > > > > > > Not true for 2.6 , which according to the docs supports Hadoop 3.2, but > > > defaults to Hadoop 3.3 > > > > > > > > > > For me, technically I think bumping to the newest patch release of a > > > > minor release should be fine, which is the proposal 1. > > > > > > > > But the current hadoopcheck is not enough, since it can only ensure > > > > that there is no complation error. > > > > Maybe we should also run some simple dev tests in the hadoopcheck > > > > stage, and in integration tests, we should try to build with all the > > > > support hadoop version and run the basic read write tests. > > > > > > > > > Do we need to test all versions ? > > > If We test with say, 3.3.0 and 3.3.6 , do we need to test with > 3.3.[1-5] ? > > > Or if we test with 3.2.5 and 3.3.6, do we need to test with any of the > > > interim versions ? > > > > > > Basically, how much do we trust Hadoop to keep to its compatibility > rules ? > > > > > > Running a limited number of tests should not be a problem. > > > Should we add a new test category, so that they can be easily started > from > > > Maven ? > > > > > > Can you suggest some tests that we should run for the compatibility > check ? > > > > > > > > > > Thanks. > > > > > > > > Istvan Toth <st...@cloudera.com.invalid> 于2024年9月11日周三 21:05写道： > > > > > > > > > > Let me summarize my take of the discussion so far: > > > > > > > > > > There are two aspects to the HBase version we build with: > > > > > 1. Source code quality/compatibility > > > > > 2. Security and usability of the public binary assemblies and > (shaded) > > > > > hbase maven artifacts. > > > > > > > > > > 1. Source code quality/compatibility > > > > > > > > > > AFAICT we have the following hard goals: > > > > > 1.a : Ensure that HBase compiles and runs well with the earlier > supported > > > > > Hadoop version on the given branch > > > > > 1.b: Ensure that HBase compiles and runs well with the latest > supported > > > > > Hadoop version on the given branch > > > > > > > > > > In my opinion we should also strive for these goals: > > > > > 1.c: Aim to officially support the newest possible Hadoop releases > > > > > 1.d: Take advantage of new features in newer Hadoop versions > > > > > > > > > > 2. Public binary usability wish list: > > > > > > > > > > 2.a: We want them to work OOB for as many use cases as possible > > > > > 2.b: We want to work them as well as possible > > > > > 2.c: We want to have as few CVEs in them as possible > > > > > 2.d: We want to make upgrades as painless as possible, especially > for > > > > patch > > > > > releases > > > > > > > > > > The factor that Hadoop does not have an explicit end-of-life > policy of > > > > > course complicates things. > > > > > > > > > > Our current policy seems to be that we pick a Hadoop version to > build > > > > with > > > > > when releasing a minor version, > > > > > and stay on that version until there is a newer patch released of > that > > > > > minor version with direct CVE fixes. > > > > > This does not seem to be an absolute, for example the recently > released > > > > > HBase 2.4.18 still defaults to Hadoop 3.1.2, > > > > > which has several old CVEs, many of which are reportedly fixed in > 3.1.3 > > > > and > > > > > 3.1.4. > > > > > > > > > > my proposals are : > > > > > > > > > > Proposal 1: > > > > > > > > > > Whenever a new Hadoop patch release is released for a minor > version, then > > > > > unless it breaks source compatibility, we should automatically > update the > > > > > default Hadoop version for > > > > > all branches that use the same minor version. > > > > > The existing hadoopcheck mechanism should be good enough to > guarantee > > > > that > > > > > we do not break compatibility with the earlier patch releases. > > > > > > > > > > This would ensure that the binaries use the latest and greatest > Hadoop > > > > (of > > > > > that minor branch) and that users of the binaries get the latest > fixes, > > > > > both CVE and functionality wise, and > > > > > the binaries also get the transitive CVE fixes in that release. > > > > > For example,if we did this we could use the new feature in 3.3.6 > in > > > > > HBASE-27769 (via reflection) and also test it, thereby improving > Ozone > > > > > support. > > > > > > > > > > On the other hand we minimize changes and maximize compatibility by > > > > > sticking to the same Hadoop minor release. > > > > > > > > > > Proposal 2: > > > > > > > > > > We should default to the latest hadoop version (currently 3.4.0) on > > > > > unreleased branches. > > > > > This should ensure that when we do release we default to the latest > > > > > version, and we've tested it as thoroughly as possible. > > > > > > > > > > Again. the existing Hadoopcheck mechanism should ensure that we do > not > > > > > break compatibility with earlier supported versions. > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2024 at 9:41 PM Nick Dimiduk <ndimi...@apache.org> > wrote: > > > > > > > > > > > Yes, we’ll use reflection to make use of APIs introduced in > newer HDFS > > > > > > versions than the stated dependency until the stated dependency > finally > > > > > > catches up. > > > > > > > > > > > > On Mon, 9 Sep 2024 at 19:55, Wei-Chiu Chuang <weic...@apache.org > > > > > > wrote: > > > > > > > > > > > > > Reflection is probably the way to go to ensure maximum > compatibility > > > > TBH > > > > > > > > > > > > > > On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth > > > > <st...@cloudera.com.invalid> > > > > > > > wrote: > > > > > > > > > > > > > > > Stephen Wu has kindly sent me the link for the previous email > > > > thread: > > > > > > > > > https://lists.apache.org/thread/2k4tvz3wpg06sgkynkhgvxrodmj86vsj > > > > > > > > > > > > > > > > Reading it, I cannot see anything there that would > contraindicate > > > > > > > upgrading > > > > > > > > to 3.3.6 from 3.3.5, at least on the branches that already > default > > > > to > > > > > > > > 3.3.5, i.e. 2.6+. > > > > > > > > > > > > > > > > At first glance, the new logic in HBASE-27769 could also be > > > > implemented > > > > > > > > with the usual reflection hacks, while preserving the old > logic for > > > > > > > Hadoop > > > > > > > > 3.3.5 and earlier. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2024 at 1:42 PM Istvan Toth < > st...@cloudera.com> > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for your reply, Nick. > > > > > > > > > > > > > > > > > > There are no listed direct CVEs in either Hadoop 3.2.4 or > 3.3.5, > > > > but > > > > > > > > there > > > > > > > > > are CVEs in their transitive dependencies. > > > > > > > > > > > > > > > > > > My impression is that rather than shipping the oldest > 'safe' > > > > version, > > > > > > > > > HBase does seem to update the default Hadoop version to the > > > > > > latest-ish > > > > > > > at > > > > > > > > > the time of the start > > > > > > > > > of the release process, otherwise 2.6 would still default > to > > > > 3.2.4. > > > > > > > > (HBase > > > > > > > > > 2.6 release was already underway when Hadoop 3.4.0 was > released) > > > > > > > > > > > > > > > > > > For now, we (Phoenix) have resorted to dependency managing > > > > transitive > > > > > > > > > dependencies coming in (only) via Hadoop in Phoenix, > > > > > > > > > but that is a slippery slope, and adds a layer of > uncertainty, > > > > as it > > > > > > > may > > > > > > > > > introduce incompatibilities in Hadoop that we don't have > tests > > > > for. > > > > > > > > > > > > > > > > > > Our situation is similar to that of the HBase shaded > artifacts, > > > > where > > > > > > > we > > > > > > > > > ship a huge uberjar that includes much of both HBase and > Hadoop > > > > on > > > > > > top > > > > > > > of > > > > > > > > > (or rather below) Phoenix, > > > > > > > > > similar to the hbase-client-shaded jar. > > > > > > > > > > > > > > > > > > I will look into to hadoop check CI tests that you've > mentioned, > > > > > > then I > > > > > > > > > will try to resurrect HBASE-27931, and if I don't find any > > > > issues, > > > > > > and > > > > > > > > > there are no objections, then > > > > > > > > > I will put a PR to update the unreleased version to > default to > > > > 3.4.0. > > > > > > > > > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2024 at 11:06 AM Nick Dimiduk < > > > > ndimi...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> My understanding of our hadoop dependency policy is that > we ship > > > > > > poms > > > > > > > > with > > > > > > > > >> hadoop versions pinned to the oldest compatible, "safe" > version > > > > that > > > > > > > is > > > > > > > > >> supported. Our test infrastructure has a "hadoop check" > > > > procedure > > > > > > that > > > > > > > > >> does > > > > > > > > >> some validation against other patch release versions. > > > > > > > > >> > > > > > > > > >> I don't know if anyone has done a CVE sweep recently. If > there > > > > are > > > > > > new > > > > > > > > >> CVEs, we do bump the minimum supported version specified > in the > > > > pom > > > > > > as > > > > > > > > >> part > > > > > > > > >> of patch releases. These changes need to include a pretty > > > > thorough > > > > > > > > >> compatibility check so that we can include release notes > about > > > > any > > > > > > > > >> introduced incompatibilities. > > > > > > > > >> > > > > > > > > >> I am in favor of a dependency bump so as to address known > CVEs > > > > as > > > > > > best > > > > > > > > as > > > > > > > > >> we reasonably can. > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Nick > > > > > > > > >> > > > > > > > > >> On Mon, Sep 9, 2024 at 10:59 AM Istvan Toth < > st...@apache.org> > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> > Hi! > > > > > > > > >> > > > > > > > > > >> > I'm working on building the Phoenix uberjars with newer > Hadoop > > > > > > > > versions > > > > > > > > >> by > > > > > > > > >> > default to improve its CVE stance, and I realized that > HBase > > > > > > itself > > > > > > > > does > > > > > > > > >> > not use the latest releases. > > > > > > > > >> > > > > > > > > > >> > branch-2.5 defaults to 3.2.4 > > > > > > > > >> > branch-2.6 and later defaults to 3.3.5 > > > > > > > > >> > > > > > > > > > >> > I can kind of understand that we don't want to bump the > minor > > > > > > > version > > > > > > > > >> for > > > > > > > > >> > branch-2.5 from the one it was released with. > > > > > > > > >> > > > > > > > > > >> > However, I don't see the rationale for not upgrading > > > > branch-2.6 to > > > > > > > at > > > > > > > > >> least > > > > > > > > >> > 3.3.6, and the unreleased branches (branch-2, branch-3, > > > > master) to > > > > > > > > >> 3.4.0. > > > > > > > > >> > > > > > > > > > >> > I found a mention of wanting to stay off the latest > patch > > > > release > > > > > > > > >> > HBASE-27931, but I could not figure if it has a > technical > > > > reason, > > > > > > or > > > > > > > > if > > > > > > > > >> > this is a written (or unwritten) policy. > > > > > > > > >> > > > > > > > > > >> > best regards > > > > > > > > >> > Istvan > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *István Tóth* | Sr. Staff Software Engineer > > > > > > > > > *Email*: st...@cloudera.com > > > > > > > > > cloudera.com <https://www.cloudera.com> > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera > > > > > > [image: > > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > > > > [image: > > > > > > > > > Cloudera on LinkedIn] < > https://www.linkedin.com/company/cloudera > > > > > > > > > > > > > > ------------------------------ > > > > > > > > > ------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *István Tóth* | Sr. Staff Software Engineer > > > > > > > > *Email*: st...@cloudera.com > > > > > > > > cloudera.com <https://www.cloudera.com> > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > > > > [image: > > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> > [image: > > > > > > > Cloudera > > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > > > > ------------------------------ > > > > > > > > ------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *István Tóth* | Sr. Staff Software Engineer > > > > > *Email*: st...@cloudera.com > > > > > cloudera.com <https://www.cloudera.com> > > > > > [image: Cloudera] <https://www.cloudera.com/> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> > [image: > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > > Cloudera > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > > > ------------------------------ > > > > > ------------------------------ > > > > > > > > > > > > > -- > > > *István Tóth* | Sr. Staff Software Engineer > > > *Email*: st...@cloudera.com > > > cloudera.com <https://www.cloudera.com> > > > [image: Cloudera] <https://www.cloudera.com/> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > ------------------------------ > > > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
