Suggestions: - For HBase 2.x releases, we should continue to publish default builds, those without any -hadoop3- or -widgetfoo- modifiers, against Hadoop 2.
- For HBase 3, it makes sense to move the default to Hadoop 3, no other build variants needed there. This is the kind of thing a major version increment allows us to do per our dependency compatibility guidelines. - While eventually it may be necessary to differentiate between minor release lines of Hadoop it would be simpler to pick one Hadoop 3 version, like 3.3.4, and build and publish a -hadoop3- artifact for each current releasing 2.x code line: 2.4.15-hadoop3, 2.5.2-hadoop3, 2.6.0-hadoop3. - The process of building releases is automated by create-release, which all RMs use now. create-release automates the process of building and signing tarballs and publishing to Nexus. There should be no significant new burden on the RM, beyond an increase in time for create-release execution, to parameterize it and iterate over one or more variant builds. That is a long way of suggesting we do publish variant tarballs too, they are almost "for free" if we've gone to the trouble to build for publishing to Nexus. On Wed, Oct 19, 2022 at 12:52 AM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > After some investigating, I think using the $revision placeholder can > solve the problem here, i.e, using different command line to publish > different artifacts for hadoop2 and hadoop3, with the same souce code. > You can see the comment on HBASE-27359 for more details. > > Next I will open an issue to land the $revision change. And here, I > think first we need to discuss how many new artifacts we want to > publish. For example, for 2.6.0, we only want to publish a > 2.6.0-hadoop3, with the default hadoop3 version? Or we publish > 2.6.0-hadoop3.2, 2.6.0-hadoop3.3 for different hadoop minor release > lines? And do we want to publish different tarballs for hadoop2 and > hadoop3? > > Thanks. > > Andrew Purtell <apurt...@apache.org> 于2022年8月31日周三 00:19写道： > > > > I also don't think we should change the defaults in branch-2 until > Hadoop 2 > > is EOLed. > > > > On Mon, Aug 29, 2022 at 10:22 AM Sean Busbey <bus...@apache.org> wrote: > > > > > I think changing the default hadoop profile for builds in branch-2 > would > > > unnecessarily complicate our compatibility messaging so long as Hadoop > 2 > > > hasn't gone EOL. > > > > > > On Mon, Aug 29, 2022 at 5:30 AM Nick Dimiduk <ndimi...@apache.org> > wrote: > > > > > > > Should we also make hadoop3 the default active profile for branch-2 > going > > > > forward? > > > > > > > > On Fri, Aug 26, 2022 at 5:25 PM Andrew Purtell < > andrew.purt...@gmail.com > > > > > > > > wrote: > > > > > > > > > The security posture of Hadoop 2 in general is a problem, because > > > > > maintenance on that branch is spotty, that is just how it goes. We > had > > > > the > > > > > same situation with our now EOL branch-1. I know Hadoop released > 2.10.2 > > > > to > > > > > address some CVE worthy problems but it is unclear if 2.10.2 > addresses > > > > all > > > > > known issues, unlike 3.3.4. Also as you know Hadoop 2 has > unpatchable > > > > > dependencies on org.codehaus versions of Jackson and Jetty, which > > > > > themselves have high scoring CVEs that will never be fixed because > they > > > > are > > > > > EOL, and other similar issues. Hadoop 3 doesn’t completely solve > such > > > > > problems but is the only realistic place we can hope they can be > > > > addressed > > > > > as required. For organizations that implement or require a top to > > > bottom > > > > > security audit of their software bill of materials, it seems > possible > > > to > > > > > avoid user pain by providing supported convenience artifacts *and* > > > > > libraries built against Hadoop 3 APIs in the Apache repository > > > > addressable > > > > > with a Maven classifier. > > > > > > > > > > My employer has some interests in this area that align so I would > like > > > to > > > > > sponsor (implement, review, commit, RM backfill releases, etc.) > this > > > > work. > > > > > Would there be any objections? Read through the thread for some > > > thoughts > > > > on > > > > > approach. Summarized: > > > > > > > > > > - Amend create-release to build, stage, and deploy a -hadoop3 > variant > > > > > build by activating the Hadoop 3 build profile. > > > > > > > > > > - Amend the Hadoop 3 build profile to flatten POMs before > deployment to > > > > > resolve potential downstream issues due to Hadoop 3 being a > non-default > > > > > build profile. (This could also be applied to all builds.) > > > > > > > > > > - Amend hbase-vote to be aware of and evaluate if present -hadoop3 > > > > variant > > > > > artifacts. > > > > > > > > > > > > > > > > On Aug 25, 2022, at 10:40 AM, Andrew Purtell < > > > andrew.purt...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > Thanks, that would work. > > > > > > > > > > > >> On Aug 25, 2022, at 11:35 AM, Sean Busbey <bus...@apache.org> > > > wrote: > > > > > >> > > > > > >> yes, the flatten plugin. We use it in hbase-connectors already. > > > > > >> > > > > > >> https://www.mojohaus.org/flatten-maven-plugin/ > > > > > >> > > > > > >> this sounds like it could also be a use case for BOMs, which > would > > > > also > > > > > >> benefit users of our client artifacts that use build tools that > > > don't > > > > > >> respect maven profiles generally, like gradle. > > > > > >> > > > > > >>> On Thu, Aug 25, 2022 at 10:30 AM Andrew Purtell < > > > > > andrew.purt...@gmail.com> > > > > > >>> wrote: > > > > > >>> > > > > > >>> Thinking about this a bit more, we will have an issue in that > the > > > > POMs > > > > > >>> published from our -hadoop3 build will not have a default > > > activation > > > > > of our > > > > > >>> Hadoop 3 build profile. The convenience binaries will function > as > > > > > expected > > > > > >>> but Maven will read and process eg Phoenix POMs, then download > and > > > > > perform > > > > > >>> substitutions on HBase POMs, and then etc, so downstreamers > like > > > > > Phoenix > > > > > >>> will have to set up the hadoop.profile variable for us in their > > > > default > > > > > >>> build profile or else the transitive paths through us may be > > > wrong. I > > > > > >>> wonder if there is a Maven plugin available for deploying POMs > with > > > > all > > > > > >>> variable substitutions performed before deployment, that would > > > solve > > > > > that > > > > > >>> problem and all conceivable related issues. > > > > > >>> > > > > > >>>> On Aug 25, 2022, at 11:03 AM, Andrew Purtell < > > > > > andrew.purt...@gmail.com> > > > > > >>> wrote: > > > > > >>>> > > > > > >>>> I think 2.x is going to have a few years of life remaining > so it > > > > > would > > > > > >>> be best, if we are going to address this, to have a 2.x > solution > > > was > > > > > well > > > > > >>> as a 3.x one. > > > > > >>>> > > > > > >>>> In my opinion we can continue to publish 2.4 and 2.5 (and 2.6) > > > > > unchanged > > > > > >>> and then also introduce a Hadoop 3 release using “hadoop3” or > > > similar > > > > > as > > > > > >>> Maven classifier. Phoenix could specify this classifier in > their > > > > POMs. > > > > > >>> Everyone should be happy. Users who already are comfortable > with > > > the > > > > > Hadoop > > > > > >>> 2 default don’t have to change anything. A one time POM change > on > > > the > > > > > >>> Phoenix side is required but that’s it. > > > > > >>>> > > > > > >>>> The additional build time complexity for generating two > releases > > > can > > > > > be > > > > > >>> incorporated into create-release. Nobody does manual releases > any > > > > more > > > > > as > > > > > >>> far as I know. Likewise, download and verification of -hadoop3 > > > > > convenience > > > > > >>> binaries can be added to hbase-vote. I believe we are all using > > > that > > > > > tool > > > > > >>> for verification of releases now. After these one time changes > are > > > > > landed > > > > > >>> the cost for RMs and PMC will be only in a roughly doubled > amount > > > of > > > > > time > > > > > >>> needed to build and verify releases. > > > > > >>>> > > > > > >>>>>> On Aug 17, 2022, at 9:06 AM, Nick Dimiduk < > ndimi...@apache.org> > > > > > wrote: > > > > > >>>>>> > > > > > >>>>>> Hi Geoffrey, > > > > > >>>>>> > > > > > >>>>>> I have no complaints with shipping convenience binaries > built > > > > > against > > > > > >>> both > > > > > >>>>> Hadoop2 and Hadoop3. The primary challenge is implementing > the > > > > > >>>>> necessary build changes, the secondary challenge is > > > > > verifying/testing it > > > > > >>>>> works reliably. > > > > > >>>>> > > > > > >>>>> But for Phoenix, are you asking for convenience binaries, or > are > > > > you > > > > > >>> asking > > > > > >>>>> for artifacts published into maven that have the Hadoop3 > profile > > > > > >>> activated > > > > > >>>>> and specify the associated dependencies? > > > > > >>>>> > > > > > >>>>> I'm afraid that the 2.5.0 release ship has already sailed. > I've > > > > heard > > > > > >>> talk > > > > > >>>>> of a 2.6 "fast-follow", so maybe someone can have the build > > > changes > > > > > >>> ready > > > > > >>>>> for that? Also, isn't this a too little, too late situation? > > > > > Shouldn't > > > > > >>> we > > > > > >>>>> shift our focus to releasing 3.0, which has dropped support > for > > > > > Hadoop2? > > > > > >>>>> > > > > > >>>>> Thanks, > > > > > >>>>> Nick > > > > > >>>>> > > > > > >>>>>>> On Tue, Aug 16, 2022 at 9:30 PM Geoffrey Jacoby < > > > > > gjac...@apache.org> > > > > > >>> wrote: > > > > > >>>>>> > > > > > >>>>>> I see that the next HBase 2.5 RC is imminent, and before > that's > > > > set > > > > > in > > > > > >>>>>> stone, I wanted to bring up the question of whether there > will > > > be > > > > > >>> official > > > > > >>>>>> HBase 2.5 binaries built with the Hadoop 3 profile and > available > > > > in > > > > > the > > > > > >>>>>> usual Maven repositories. (In addition to the usual Hadoop 2 > > > > profile > > > > > >>>>>> binaries) > > > > > >>>>>> > > > > > >>>>>> The HBase 2.x line has a commitment to maintain support for > > > Hadoop > > > > > >>> 2.x, but > > > > > >>>>>> Hadoop 3.3 is the current stable Hadoop line and the most > recent > > > > > >>> release > > > > > >>>>>> notes [1] encourage all users of Hadoop 2.x to upgrade to > > > Hadoop > > > > 3. > > > > > >>>>>> > > > > > >>>>>> Without convenience artifacts built against Hadoop 3, no > > > end-users > > > > > with > > > > > >>>>>> Hadoop 3 clusters will be able to use the Apache-distributed > > > > > binaries > > > > > >>> and > > > > > >>>>>> will instead have to recompile HBase from source > themselves, or > > > > use > > > > > a > > > > > >>> 3rd > > > > > >>>>>> party distribution that does so for them. > > > > > >>>>>> > > > > > >>>>>> This is especially inconvenient for downstream projects > such as > > > > > Apache > > > > > >>>>>> Phoenix, which has never officially supported the HBase > 2.x / > > > > > Hadoop > > > > > >>> 2.10 > > > > > >>>>>> combination. (It currently supports only HBase 2.3 or 2.4 > with > > > > > Hadoop > > > > > >>> 3. > > > > > >>>>>> HBase 2.5 support will be added very shortly after its > release > > > as > > > > > part > > > > > >>> of > > > > > >>>>>> Phoenix 5.2.) > > > > > >>>>>> > > > > > >>>>>> To even run the Phoenix IT tests locally requires > contributors > > > to > > > > > >>> download > > > > > >>>>>> the HBase source release and manually mvn install to their > local > > > > > maven > > > > > >>> repo > > > > > >>>>>> using the Hadoop 3 profile, to avoid crashes in the HBase > > > > > >>> minicluster.[2] > > > > > >>>>>> This is a barrier to new contributors and confuses even > veteran > > > > > ones, > > > > > >>> and > > > > > >>>>>> has to be done again for every new HBase release. > > > > > >>>>>> > > > > > >>>>>> In general, I expect the Hadoop 3 user base to grow and the > > > Hadoop > > > > > 2.10 > > > > > >>>>>> user base to shrink with every future HBase 2 release, so I > > > think > > > > > this > > > > > >>> is a > > > > > >>>>>> worthwhile improvement. > > > > > >>>>>> > > > > > >>>>>> Thanks, > > > > > >>>>>> > > > > > >>>>>> Geoffrey > > > > > >>>>>> > > > > > >>>>>> [1] https://hadoop.apache.org/release/3.3.4.html > > > > > >>>>>> [2] > https://github.com/apache/phoenix/blob/master/BUILDING.md > > > > > >>>>>> > > > > > >>> > > > > > > > > > > > > > > > > > > -- > > Best regards, > > Andrew > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > It's what we’ve earned > > Welcome, apocalypse, what’s taken you so long? > > Bring us the fitting end that we’ve been counting on > > - A23, Welcome, Apocalypse > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
