BTW does anyone have an idea why HBase is still on 3.3.5 instead of 3.3.6 ? I found some mention that HBase does not want to use the latest Hadoop, but I could not find the rationale for that.
On Mon, Sep 9, 2024 at 10:30 AM Istvan Toth <st...@cloudera.com> wrote: > Sounds good. > > Let me put up a PR for the Hadoop upgrade, we can re-run OWASP, or > whatever CVE checker after that. > > On Sat, Sep 7, 2024 at 1:28 AM Viraj Jasani <vjas...@apache.org> wrote: > >> Sounds good, I agree with this solution. We can keep them in profile and >> disable profile once Hadoop resolves CVEs. >> I feel we should resolve Avro and Jettison CVEs and rollout 5.2.1 unless >> newer released version of Hadoop already has them resolved. WDYT? >> >> >> On Thu, Sep 5, 2024 at 9:46 PM Istvan Toth <st...@cloudera.com.invalid> >> wrote: >> >> > Thanks for your response, Viraj. >> > >> > Sure, managing transitive dependencies and using a newer Hadoop are not >> > mutually exclusive. >> > >> > For old HBase versions like 2.1 even the newest officially supported >> Hadoop >> > is going to be old. >> > >> > What do you think about my proposals on how to handle the transitive >> > dependency management ? >> > I quite like the idea of putting those into a profile (which is enabled >> by >> > default) so that they can be disabled if a newer hadoop is used for >> > building. >> > >> > Istvan >> > >> > On Thu, Sep 5, 2024 at 5:37 PM Viraj Jasani <vjas...@apache.org> wrote: >> > >> > > I agree with upgrading Hadoop version but it is helpful only if the >> CVEs >> > > from indirect dependencies have been resolved by Hadoop. >> > > The release frequency is too slow for Hadoop (similar to us) so some >> CVE >> > > resolution might be WIP and take considerable time to get released. In >> > such >> > > cases, maybe we can also try to manage dependency within Phoenix pom. >> > WDYT? >> > > >> > > >> > > On Wed, Sep 4, 2024 at 1:02 AM Istvan Toth <st...@apache.org> wrote: >> > > >> > > > Hi! >> > > > >> > > > In light of the recent discussions with the Trino team, we should >> think >> > > > about improving our CVE stance. >> > > > >> > > > We are doing a good job of updating our direct dependencies, but we >> > don't >> > > > have a good solution for our indirect dependencies, which are >> included >> > in >> > > > our official shaded binary artifacts, and show up in CVE scans. >> > > > >> > > > These are basically all coming from Hadoop, as HBase is also good at >> > > > keeping its direct dependencies in order. >> > > > >> > > > The main problem is that in order to maximize compatibility, we >> build >> > our >> > > > binaries with the oldest supported Hadoop major.minor version, >> which is >> > > > often long EOL. >> > > > >> > > > I don't have experience with mixing Hadoop server and client >> versions, >> > > but >> > > > I recall that Hadoop major versions are supposed to be backwards >> wire >> > > > compatible. >> > > > >> > > > The only binaries that include the Hadoop (and HBase) code are the >> > > > phoenix-client-embedded and phoenix-client-lite uberjars, which are >> > > > unfortunately the most widely used artifacts. >> > > > >> > > > My first proposal is to use the latest supported Hadoop version >> instead >> > > of >> > > > the oldest one for each HBase profile. This should remove many of >> the >> > > > existing CVEs from the binaries. >> > > > >> > > > Do you think this would cause backwards compatibility issues ? >> > > > >> > > > If not, do you think that we should do this for 5.2.1 ? >> > > > >> > > > My second discussion point is dependencyManage-ing transitive >> > dependency >> > > > versions for the CVEs that present in whatever Hadoop version we >> build >> > > > with. >> > > > This is a slippery slope, and I personally would like to minimize >> and >> > > > compartmentalize it as much as possible. >> > > > >> > > > What is your opinion on this ? >> > > > Is this worth doing ? >> > > > Should we do it in the main code, or only for the uberjars (which >> would >> > > be >> > > > big a testing problem) >> > > > Should we put those dependencyManage entries in a profile, so that >> they >> > > can >> > > > be turned off when building with newer Hadoop versions ? >> > > > >> > > > looking forward to hearing your thoughts on this >> > > > >> > > > Istvan >> > > > >> > > >> > >> > >> > -- >> > *István Tóth* | Sr. Staff Software Engineer >> > *Email*: st...@cloudera.com >> > cloudera.com <https://www.cloudera.com> >> > [image: Cloudera] <https://www.cloudera.com/> >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > ------------------------------ >> > ------------------------------ >> > >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
