I agree with Geoffrey that OMID should align with Phoenix on default dependency versions but based on this you don’t have an immediate integration problem. I don’t believe the Configuration API has changed in a long time.
The security posture of Hadoop 2 in general is a problem, though. I know Hadoop released 2.10.2 to address some CVE worthy problems but it is unclear if 2.10.2 addresses all known issues, unlike 3.3.4. Also as you know Hadoop 2 has unpatchable dependencies on org.codehaus versions of Jackson and Jetty, which themselves have high scoring CVEs that will never be fixed because they are EOL, and other similar issues. Hadoop 3 doesn’t completely solve such problems but is the only realistic place we can hope they can be addressed as required. For organizations that implement or require a top to bottom security audit of their software bill of materials, it seems possible to avoid some user pain here by aligning build defaults. I also see Geoffrey started a discussion on dev@hbase about HBase producing Hadoop 3 variant builds that could be consumed by downstreamers. I have responded on that thread today to express my intention to sponsor that effort, for what it’s worth. > On Aug 25, 2022, at 10:41 PM, Istvan Toth <[email protected]> wrote: > > The only non-hbase Hadoop APIs referred in Omid are > org.apache.hadoop.conf.Configuration, and a few classes from of > org.apache.hadoop.security > (the latter is not referenced directly from the coprocessors) . > > Also, the coprocessor JARs don't have any Hbase or Hadoop code shaded in. > > > >> On Fri, Aug 26, 2022 at 5:45 AM Andrew Purtell <[email protected]> >> wrote: >> >> Is any OMID coprocessor code using Hadoop APIs? >> >>> On Aug 25, 2022, at 9:41 AM, Istvan Toth <[email protected]> >> wrote: >>> >>> We dependencyManage every Hadoop component in Phoenix, so going to >> Hadoop3 >>> in Omid is not strictly necessary. >>> Hadoop is also added to the TSO libraries in the assembly, but AFAIK >>> Hadoop2 and Hadoop3 are wire compatible >>> (and I'm not sure if Omid even calls Hadoop endpoints directly), so that >> is >>> not a show stopper either. >>> >>> If we go with the assumption that there is no significant usage of Omid >>> outside HBase, then updating to Hadoop3 is a no-brainer, if only to avoid >>> juggling >>> the Hadoop2 and Hadoop3 compiled HBases when developing. >>> Unfortunately it's quite a bit of work with all the dependency >> management, >>> build docs, and updating the CI pipelines to rebuild HBase >>> like we do in Phoenix. >>> >>> Istvan >>> >>> >>>> On Wed, Aug 24, 2022 at 9:24 PM Geoffrey Jacoby <[email protected]> >> wrote: >>>> >>>> Thanks for volunteering to be RM for the next Omid release. >>>> >>>> I agree we should have a release soon (I think we'll want Phoenix 5.2 to >>>> use the new Omid). In addition to OMID-226, I suggest we also upgrade >>>> Omid's internal Hadoop version to align with whatever default Hadoop we >>>> choose for Phoenix 5.2, to keep Phoenix's transitive dependencies (a >>>> little) simpler. Right now Omid appears to depend on Hadoop 2.10, which >>>> since Phoenix 5 is Hadoop 3-based, seems incorrect. >>>> >>>> Geoffrey >>>> >>>>> On Wed, Aug 24, 2022 at 12:46 AM Istvan Toth <[email protected]> wrote: >>>>> >>>>> Hi! >>>>> >>>>> Most of the planned OMID changes for Phoenix 5.2 have landed. >>>>> The only outstanding ticket that I'm aware of is OMID-226 which I also >>>>> expect to land soon. >>>>> >>>>> Unless someone has more changes targeted for the next release, I >> propose >>>>> that we release the next Omid version soon after OMID-226. >>>>> >>>>> I also propose bumping the version to 1.1.0, though because of the >> HBase >>>>> 1.x compatibility removal, and maven artifact changes we could also >> argue >>>>> for 2.0.0 >>>>> >>>>> If there are no other volunteers, I also volunteer to be the RM for the >>>>> release. >>>>> >>>>> regards >>>>> Istvan >>>>> >>>> >>> >>> >>> -- >>> *István Tóth* | Staff Software Engineer >>> [email protected] <https://www.cloudera.com> >>> [image: Cloudera] <https://www.cloudera.com/> >>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera >>> on LinkedIn] <https://www.linkedin.com/company/cloudera> >>> <https://www.cloudera.com/> >>> ------------------------------ >> > > > -- > *István Tóth* | Staff Software Engineer > [email protected] <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > <https://www.cloudera.com/> > ------------------------------
