After some investigating, I think using the $revision placeholder can solve the problem here, i.e, using different command line to publish different artifacts for hadoop2 and hadoop3, with the same souce code. You can see the comment on HBASE-27359 for more details.
Next I will open an issue to land the $revision change. And here, I think first we need to discuss how many new artifacts we want to publish. For example, for 2.6.0, we only want to publish a 2.6.0-hadoop3, with the default hadoop3 version? Or we publish 2.6.0-hadoop3.2, 2.6.0-hadoop3.3 for different hadoop minor release lines? And do we want to publish different tarballs for hadoop2 and hadoop3? Thanks. Andrew Purtell <apurt...@apache.org> 于2022年8月31日周三 00:19写道： > > I also don't think we should change the defaults in branch-2 until Hadoop 2 > is EOLed. > > On Mon, Aug 29, 2022 at 10:22 AM Sean Busbey <bus...@apache.org> wrote: > > > I think changing the default hadoop profile for builds in branch-2 would > > unnecessarily complicate our compatibility messaging so long as Hadoop 2 > > hasn't gone EOL. > > > > On Mon, Aug 29, 2022 at 5:30 AM Nick Dimiduk <ndimi...@apache.org> wrote: > > > > > Should we also make hadoop3 the default active profile for branch-2 going > > > forward? > > > > > > On Fri, Aug 26, 2022 at 5:25 PM Andrew Purtell <andrew.purt...@gmail.com > > > > > > wrote: > > > > > > > The security posture of Hadoop 2 in general is a problem, because > > > > maintenance on that branch is spotty, that is just how it goes. We had > > > the > > > > same situation with our now EOL branch-1. I know Hadoop released 2.10.2 > > > to > > > > address some CVE worthy problems but it is unclear if 2.10.2 addresses > > > all > > > > known issues, unlike 3.3.4. Also as you know Hadoop 2 has unpatchable > > > > dependencies on org.codehaus versions of Jackson and Jetty, which > > > > themselves have high scoring CVEs that will never be fixed because they > > > are > > > > EOL, and other similar issues. Hadoop 3 doesn’t completely solve such > > > > problems but is the only realistic place we can hope they can be > > > addressed > > > > as required. For organizations that implement or require a top to > > bottom > > > > security audit of their software bill of materials, it seems possible > > to > > > > avoid user pain by providing supported convenience artifacts *and* > > > > libraries built against Hadoop 3 APIs in the Apache repository > > > addressable > > > > with a Maven classifier. > > > > > > > > My employer has some interests in this area that align so I would like > > to > > > > sponsor (implement, review, commit, RM backfill releases, etc.) this > > > work. > > > > Would there be any objections? Read through the thread for some > > thoughts > > > on > > > > approach. Summarized: > > > > > > > > - Amend create-release to build, stage, and deploy a -hadoop3 variant > > > > build by activating the Hadoop 3 build profile. > > > > > > > > - Amend the Hadoop 3 build profile to flatten POMs before deployment to > > > > resolve potential downstream issues due to Hadoop 3 being a non-default > > > > build profile. (This could also be applied to all builds.) > > > > > > > > - Amend hbase-vote to be aware of and evaluate if present -hadoop3 > > > variant > > > > artifacts. > > > > > > > > > > > > > On Aug 25, 2022, at 10:40 AM, Andrew Purtell < > > andrew.purt...@gmail.com > > > > > > > > wrote: > > > > > > > > > > Thanks, that would work. > > > > > > > > > >> On Aug 25, 2022, at 11:35 AM, Sean Busbey <bus...@apache.org> > > wrote: > > > > >> > > > > >> yes, the flatten plugin. We use it in hbase-connectors already. > > > > >> > > > > >> https://www.mojohaus.org/flatten-maven-plugin/ > > > > >> > > > > >> this sounds like it could also be a use case for BOMs, which would > > > also > > > > >> benefit users of our client artifacts that use build tools that > > don't > > > > >> respect maven profiles generally, like gradle. > > > > >> > > > > >>> On Thu, Aug 25, 2022 at 10:30 AM Andrew Purtell < > > > > andrew.purt...@gmail.com> > > > > >>> wrote: > > > > >>> > > > > >>> Thinking about this a bit more, we will have an issue in that the > > > POMs > > > > >>> published from our -hadoop3 build will not have a default > > activation > > > > of our > > > > >>> Hadoop 3 build profile. The convenience binaries will function as > > > > expected > > > > >>> but Maven will read and process eg Phoenix POMs, then download and > > > > perform > > > > >>> substitutions on HBase POMs, and then etc, so downstreamers like > > > > Phoenix > > > > >>> will have to set up the hadoop.profile variable for us in their > > > default > > > > >>> build profile or else the transitive paths through us may be > > wrong. I > > > > >>> wonder if there is a Maven plugin available for deploying POMs with > > > all > > > > >>> variable substitutions performed before deployment, that would > > solve > > > > that > > > > >>> problem and all conceivable related issues. > > > > >>> > > > > >>>> On Aug 25, 2022, at 11:03 AM, Andrew Purtell < > > > > andrew.purt...@gmail.com> > > > > >>> wrote: > > > > >>>> > > > > >>>> I think 2.x is going to have a few years of life remaining so it > > > > would > > > > >>> be best, if we are going to address this, to have a 2.x solution > > was > > > > well > > > > >>> as a 3.x one. > > > > >>>> > > > > >>>> In my opinion we can continue to publish 2.4 and 2.5 (and 2.6) > > > > unchanged > > > > >>> and then also introduce a Hadoop 3 release using “hadoop3” or > > similar > > > > as > > > > >>> Maven classifier. Phoenix could specify this classifier in their > > > POMs. > > > > >>> Everyone should be happy. Users who already are comfortable with > > the > > > > Hadoop > > > > >>> 2 default don’t have to change anything. A one time POM change on > > the > > > > >>> Phoenix side is required but that’s it. > > > > >>>> > > > > >>>> The additional build time complexity for generating two releases > > can > > > > be > > > > >>> incorporated into create-release. Nobody does manual releases any > > > more > > > > as > > > > >>> far as I know. Likewise, download and verification of -hadoop3 > > > > convenience > > > > >>> binaries can be added to hbase-vote. I believe we are all using > > that > > > > tool > > > > >>> for verification of releases now. After these one time changes are > > > > landed > > > > >>> the cost for RMs and PMC will be only in a roughly doubled amount > > of > > > > time > > > > >>> needed to build and verify releases. > > > > >>>> > > > > >>>>>> On Aug 17, 2022, at 9:06 AM, Nick Dimiduk <ndimi...@apache.org> > > > > wrote: > > > > >>>>>> > > > > >>>>>> Hi Geoffrey, > > > > >>>>>> > > > > >>>>>> I have no complaints with shipping convenience binaries built > > > > against > > > > >>> both > > > > >>>>> Hadoop2 and Hadoop3. The primary challenge is implementing the > > > > >>>>> necessary build changes, the secondary challenge is > > > > verifying/testing it > > > > >>>>> works reliably. > > > > >>>>> > > > > >>>>> But for Phoenix, are you asking for convenience binaries, or are > > > you > > > > >>> asking > > > > >>>>> for artifacts published into maven that have the Hadoop3 profile > > > > >>> activated > > > > >>>>> and specify the associated dependencies? > > > > >>>>> > > > > >>>>> I'm afraid that the 2.5.0 release ship has already sailed. I've > > > heard > > > > >>> talk > > > > >>>>> of a 2.6 "fast-follow", so maybe someone can have the build > > changes > > > > >>> ready > > > > >>>>> for that? Also, isn't this a too little, too late situation? > > > > Shouldn't > > > > >>> we > > > > >>>>> shift our focus to releasing 3.0, which has dropped support for > > > > Hadoop2? > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> Nick > > > > >>>>> > > > > >>>>>>> On Tue, Aug 16, 2022 at 9:30 PM Geoffrey Jacoby < > > > > gjac...@apache.org> > > > > >>> wrote: > > > > >>>>>> > > > > >>>>>> I see that the next HBase 2.5 RC is imminent, and before that's > > > set > > > > in > > > > >>>>>> stone, I wanted to bring up the question of whether there will > > be > > > > >>> official > > > > >>>>>> HBase 2.5 binaries built with the Hadoop 3 profile and available > > > in > > > > the > > > > >>>>>> usual Maven repositories. (In addition to the usual Hadoop 2 > > > profile > > > > >>>>>> binaries) > > > > >>>>>> > > > > >>>>>> The HBase 2.x line has a commitment to maintain support for > > Hadoop > > > > >>> 2.x, but > > > > >>>>>> Hadoop 3.3 is the current stable Hadoop line and the most recent > > > > >>> release > > > > >>>>>> notes [1] encourage all users of Hadoop 2.x to upgrade to > > Hadoop > > > 3. > > > > >>>>>> > > > > >>>>>> Without convenience artifacts built against Hadoop 3, no > > end-users > > > > with > > > > >>>>>> Hadoop 3 clusters will be able to use the Apache-distributed > > > > binaries > > > > >>> and > > > > >>>>>> will instead have to recompile HBase from source themselves, or > > > use > > > > a > > > > >>> 3rd > > > > >>>>>> party distribution that does so for them. > > > > >>>>>> > > > > >>>>>> This is especially inconvenient for downstream projects such as > > > > Apache > > > > >>>>>> Phoenix, which has never officially supported the HBase 2.x / > > > > Hadoop > > > > >>> 2.10 > > > > >>>>>> combination. (It currently supports only HBase 2.3 or 2.4 with > > > > Hadoop > > > > >>> 3. > > > > >>>>>> HBase 2.5 support will be added very shortly after its release > > as > > > > part > > > > >>> of > > > > >>>>>> Phoenix 5.2.) > > > > >>>>>> > > > > >>>>>> To even run the Phoenix IT tests locally requires contributors > > to > > > > >>> download > > > > >>>>>> the HBase source release and manually mvn install to their local > > > > maven > > > > >>> repo > > > > >>>>>> using the Hadoop 3 profile, to avoid crashes in the HBase > > > > >>> minicluster.[2] > > > > >>>>>> This is a barrier to new contributors and confuses even veteran > > > > ones, > > > > >>> and > > > > >>>>>> has to be done again for every new HBase release. > > > > >>>>>> > > > > >>>>>> In general, I expect the Hadoop 3 user base to grow and the > > Hadoop > > > > 2.10 > > > > >>>>>> user base to shrink with every future HBase 2 release, so I > > think > > > > this > > > > >>> is a > > > > >>>>>> worthwhile improvement. > > > > >>>>>> > > > > >>>>>> Thanks, > > > > >>>>>> > > > > >>>>>> Geoffrey > > > > >>>>>> > > > > >>>>>> [1] https://hadoop.apache.org/release/3.3.4.html > > > > >>>>>> [2] https://github.com/apache/phoenix/blob/master/BUILDING.md > > > > >>>>>> > > > > >>> > > > > > > > > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse
