Yes, we’ll use reflection to make use of APIs introduced in newer HDFS versions than the stated dependency until the stated dependency finally catches up.
On Mon, 9 Sep 2024 at 19:55, Wei-Chiu Chuang <weic...@apache.org> wrote: > Reflection is probably the way to go to ensure maximum compatibility TBH > > On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth <st...@cloudera.com.invalid> > wrote: > > > Stephen Wu has kindly sent me the link for the previous email thread: > > https://lists.apache.org/thread/2k4tvz3wpg06sgkynkhgvxrodmj86vsj > > > > Reading it, I cannot see anything there that would contraindicate > upgrading > > to 3.3.6 from 3.3.5, at least on the branches that already default to > > 3.3.5, i.e. 2.6+. > > > > At first glance, the new logic in HBASE-27769 could also be implemented > > with the usual reflection hacks, while preserving the old logic for > Hadoop > > 3.3.5 and earlier. > > > > Thanks, > > Istvan > > > > > > > > On Mon, Sep 9, 2024 at 1:42 PM Istvan Toth <st...@cloudera.com> wrote: > > > > > Thanks for your reply, Nick. > > > > > > There are no listed direct CVEs in either Hadoop 3.2.4 or 3.3.5, but > > there > > > are CVEs in their transitive dependencies. > > > > > > My impression is that rather than shipping the oldest 'safe' version, > > > HBase does seem to update the default Hadoop version to the latest-ish > at > > > the time of the start > > > of the release process, otherwise 2.6 would still default to 3.2.4. > > (HBase > > > 2.6 release was already underway when Hadoop 3.4.0 was released) > > > > > > For now, we (Phoenix) have resorted to dependency managing transitive > > > dependencies coming in (only) via Hadoop in Phoenix, > > > but that is a slippery slope, and adds a layer of uncertainty, as it > may > > > introduce incompatibilities in Hadoop that we don't have tests for. > > > > > > Our situation is similar to that of the HBase shaded artifacts, where > we > > > ship a huge uberjar that includes much of both HBase and Hadoop on top > of > > > (or rather below) Phoenix, > > > similar to the hbase-client-shaded jar. > > > > > > I will look into to hadoop check CI tests that you've mentioned, then I > > > will try to resurrect HBASE-27931, and if I don't find any issues, and > > > there are no objections, then > > > I will put a PR to update the unreleased version to default to 3.4.0. > > > > > > Istvan > > > > > > On Mon, Sep 9, 2024 at 11:06 AM Nick Dimiduk <ndimi...@apache.org> > > wrote: > > > > > >> My understanding of our hadoop dependency policy is that we ship poms > > with > > >> hadoop versions pinned to the oldest compatible, "safe" version that > is > > >> supported. Our test infrastructure has a "hadoop check" procedure that > > >> does > > >> some validation against other patch release versions. > > >> > > >> I don't know if anyone has done a CVE sweep recently. If there are new > > >> CVEs, we do bump the minimum supported version specified in the pom as > > >> part > > >> of patch releases. These changes need to include a pretty thorough > > >> compatibility check so that we can include release notes about any > > >> introduced incompatibilities. > > >> > > >> I am in favor of a dependency bump so as to address known CVEs as best > > as > > >> we reasonably can. > > >> > > >> Thanks, > > >> Nick > > >> > > >> On Mon, Sep 9, 2024 at 10:59 AM Istvan Toth <st...@apache.org> wrote: > > >> > > >> > Hi! > > >> > > > >> > I'm working on building the Phoenix uberjars with newer Hadoop > > versions > > >> by > > >> > default to improve its CVE stance, and I realized that HBase itself > > does > > >> > not use the latest releases. > > >> > > > >> > branch-2.5 defaults to 3.2.4 > > >> > branch-2.6 and later defaults to 3.3.5 > > >> > > > >> > I can kind of understand that we don't want to bump the minor > version > > >> for > > >> > branch-2.5 from the one it was released with. > > >> > > > >> > However, I don't see the rationale for not upgrading branch-2.6 to > at > > >> least > > >> > 3.3.6, and the unreleased branches (branch-2, branch-3, master) to > > >> 3.4.0. > > >> > > > >> > I found a mention of wanting to stay off the latest patch release > > >> > HBASE-27931, but I could not figure if it has a technical reason, or > > if > > >> > this is a written (or unwritten) policy. > > >> > > > >> > best regards > > >> > Istvan > > >> > > > >> > > > > > > > > > -- > > > *István Tóth* | Sr. Staff Software Engineer > > > *Email*: st...@cloudera.com > > > cloudera.com <https://www.cloudera.com> > > > [image: Cloudera] <https://www.cloudera.com/> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > > > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > > > ------------------------------ > > > ------------------------------ > > > > > > > > > -- > > *István Tóth* | Sr. Staff Software Engineer > > *Email*: st...@cloudera.com > > cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > ------------------------------ > > ------------------------------ > > >
