Query : Hadoop Cluster OS upgrade

[Brahma Reddy Battula]

Hi All,


Does anybody tried out/share learnings ,using maintenance state or  upgrade
domains for big data cluster OS upgrades?



Regards,
Brahma

Re: [INFO] Hadoop-3.4 Release Update

[Brahma Reddy Battula]

Thanks for driving this.

One query:
Does all the jiras from 3.3 are part of the 3.4?




On Mon, 22 Jan 2024 at 11:50 AM, slfan1989  wrote:

> Thank you very much for specifying the frozen time for the
> Branch-3.4.0.  @Xiaoqiao
> He 
>
> If there is a need to backport any PRs to branch-3.4/branch-3.4.0, please
> feel free to contact me.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He  wrote:
>
> > Hi All,
> >
> > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
> > If there is any blocker/critical PR please backport to branch-3.4 and
> > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
> >
> > If any reported issues are not ready in time please wait for the next
> > release.
> >
> > Thanks all.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >
> >
> > On Mon, Jan 22, 2024 at 10:43 AM slfan1989  wrote:
> >
> >> Thank you for the feedback!
> >>
> >> I apologize for updating partial information in the Hadoop module.
> >>
> >> Today, I will complete the rollback of the modified JIRA and add back
> the
> >> 3.4.0 version.
> >>
> >> Best
> >>
> >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
> >> wrote:
> >>
> >>> Thanks all for your input. Connected to Shilun offline yesterday, and
> he
> >>> will
> >>> update or recover the fix version tag before 3.4.0-RC1.
> >>>
> >>> Best Regards,
> >>> - He Xiaoqiao
> >>>
> >>>
> >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki  >
> >>> wrote:
> >>>
> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> >>> still
> >>> > be
> >>> > > included in the Fix versions,
> >>> > > even if it was already released in 3.3.x. If I understand
> correctly,
> >>> this
> >>> > > is usually our practice.
> >>> >
> >>> > +1.
> >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
> >>> > maintaining both branch-3.4 and branch-3.3.
> >>> >
> >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma <
> tasan...@apache.org>
> >>> > wrote:
> >>> > >
> >>> > > Hi Shilun,
> >>> > >
> >>> > > Thank you for leading the 3.4.0 release.
> >>> > >
> >>> > > > 2. In JIRA issues where other release versions have been
> released,
> >>> > remove
> >>> > > > the 3.4.0 entry from the fix version.
> >>> > >
> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> >>> still
> >>> > be
> >>> > > included in the Fix versions,
> >>> > > even if it was already released in 3.3.x. If I understand
> correctly,
> >>> this
> >>> > > is usually our practice.
> >>> > >
> >>> > > Thanks,
> >>> > > - Takanobu
> >>> > >
> >>> > > 2024年1月21日(日) 13:56 slfan1989 :
> >>> > >
> >>> > > > Hi All,
> >>> > > >
> >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
> >>> > perform
> >>> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN,
> >>> and
> >>> > > > MAPREDUCE. Specifically, the tasks include:
> >>> > > >
> >>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as
> >>> > > > non-blocker, update the target version to 3.5.0.
> >>> > > >
> >>> > > > 2. In JIRA issues where other release versions have been
> released,
> >>> > remove
> >>> > > > the 3.4.0 entry from the fix version.
> >>> > > >
> >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version
> >>> set to
> >>> > > > 3.4.0, and status set to
> >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are
> >>> not
> >>> > set,
> >>> > > > complete the information.
> >>> > > >
> >>> > > > Best Regards,
> >>> > > > - Shilun Fan.
> >>> > > >
> >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He  >
> >>> > wrote:
> >>> > > >
> >>> > > > > Hi All,
> >>> > > > >
> >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
> >>> > created[1].
> >>> > > > >
> >>> > > > > Please set the proper fix version while committing jira, 3.5.0
> >>> should
> >>> > > > > be priority. If there is any blocker/critical please also
> >>> backport to
> >>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
> >>> > > > >
> >>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
> >>> > > > >
> >>> > > > > Thanks.
> >>> > > > >
> >>> > > > > Best Regards,
> >>> > > > > - He Xiaoqiao
> >>> > > > >
> >>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
> >>> > > > >
> >>> > > >
> >>> >
> >>> > -
> >>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
> >>> > For additional commands, e-mail: private-h...@hadoop.apache.org
> >>> >
> >>> >
> >>>
> >>
>

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Sorry, I was looking for some CVE’s fixes.will make it progress on this…


On Tue, 11 Jan 2022 at 5:58 PM, Wei-Chiu Chuang  wrote:

> Is this still making progress?
>
> On Tue, Oct 5, 2021 at 8:45 PM Brahma Reddy Battula 
> wrote:
>
> > Hi Akira,
> >
> > Thanks for your email!!
> >
> > I am evaluating the CVE’s which needs to go for this release..
> >
> > Will update soon!!
> >
> >
> > On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka 
> wrote:
> >
> > > Hi Brahma,
> > >
> > > What is the release process going on? Is there any blocker for the RC?
> > >
> > > -Akira
> > >
> > > On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He 
> wrote:
> > >
> > > > Hi Brahma,
> > > >
> > > > The feature 'BPServiceActor processes commands from NameNode
> > > > asynchronously' has been ready for both branch-3.2 and branch-3.2.3.
> > > While
> > > > cherry-picking there is only minor conflict, So I checked in
> directly.
> > > BTW,
> > > > run some unit tests and build pseudo cluster to verify, it seems to
> > work
> > > > fine.
> > > > FYI.
> > > >
> > > > Regards,
> > > > - He Xiaoqiao
> > > >
> > > > On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Please go ahead. Let me know any help required on review.
> > > >>
> > > >> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He 
> > > wrote:
> > > >>
> > > >>> Hi Brahma,
> > > >>>
> > > >>> I plan to involve HDFS-14997 and related JIRAs if possible. I have
> > > >>> resolved the conflict and verified them locally.
> > > >>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
> > > >>> I would like to hear some more response that if we have enough time
> > to
> > > >>> wait for it to be ready.
> > > >>> Thanks.
> > > >>>
> > > >>> Best Regards,
> > > >>> - He Xiaoqiao
> > > >>>
> > > >>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He 
> > > wrote:
> > > >>>
> > > >>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3.
> FYI.
> > > >>>>
> > > >>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula <
> > > bra...@apache.org>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi All,
> > > >>>>>
> > > >>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly
> > > this
> > > >>>>> can
> > > >>>>> be done by this week,then I will try to create the RC next if
> there
> > > is
> > > >>>>> no
> > > >>>>> objection.
> > > >>>>>
> > > >>>>> https://issues.apache.org/jira/browse/HDFS-15160
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula <
> > > >>>>> bra...@apache.org>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>> > @Akira Ajisaka   and @Masatake Iwasaki
> > > >>>>> > 
> > > >>>>> > Looks all are build related issues when you try with bigtop. We
> > can
> > > >>>>> > discuss and prioritize this.. Will connect with you guys.
> > > >>>>> >
> > > >>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
> > > >>>>> > iwasak...@oss.nttdata.co.jp> wrote:
> > > >>>>> >
> > > >>>>> >> >> -
> > > >>>>> >>
> > > >>>>>
> > >
> >
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> > > >>>>> >> >
> > > >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> > > >>>>> >> > we do not see the

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Hi Akira,

Thanks for your email!!

I am evaluating the CVE’s which needs to go for this release..

Will update soon!!


On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka  wrote:

> Hi Brahma,
>
> What is the release process going on? Is there any blocker for the RC?
>
> -Akira
>
> On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He  wrote:
>
> > Hi Brahma,
> >
> > The feature 'BPServiceActor processes commands from NameNode
> > asynchronously' has been ready for both branch-3.2 and branch-3.2.3.
> While
> > cherry-picking there is only minor conflict, So I checked in directly.
> BTW,
> > run some unit tests and build pseudo cluster to verify, it seems to work
> > fine.
> > FYI.
> >
> > Regards,
> > - He Xiaoqiao
> >
> > On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula  >
> > wrote:
> >
> >> Please go ahead. Let me know any help required on review.
> >>
> >> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He 
> wrote:
> >>
> >>> Hi Brahma,
> >>>
> >>> I plan to involve HDFS-14997 and related JIRAs if possible. I have
> >>> resolved the conflict and verified them locally.
> >>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
> >>> I would like to hear some more response that if we have enough time to
> >>> wait for it to be ready.
> >>> Thanks.
> >>>
> >>> Best Regards,
> >>> - He Xiaoqiao
> >>>
> >>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He 
> wrote:
> >>>
> >>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI.
> >>>>
> >>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula <
> bra...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly
> this
> >>>>> can
> >>>>> be done by this week,then I will try to create the RC next if there
> is
> >>>>> no
> >>>>> objection.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/HDFS-15160
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula <
> >>>>> bra...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>> > @Akira Ajisaka   and @Masatake Iwasaki
> >>>>> > 
> >>>>> > Looks all are build related issues when you try with bigtop. We can
> >>>>> > discuss and prioritize this.. Will connect with you guys.
> >>>>> >
> >>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
> >>>>> > iwasak...@oss.nttdata.co.jp> wrote:
> >>>>> >
> >>>>> >> >> -
> >>>>> >>
> >>>>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >>>>> >> >
> >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> >>>>> >> > we do not see the issue usually since branch-3.2 uses
> >>>>> zooekeper-3.4.13,
> >>>>> >> > while it would be harmless to add the exclusion even for
> >>>>> >> zooekeeper-3.4.13.
> >>>>> >>
> >>>>> >> I filed HADOOP-17849 for this.
> >>>>> >>
> >>>>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote:
> >>>>> >> > Thanks for bringing this up, Akira. Let me explain some
> >>>>> background.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >> -
> >>>>> >>
> >>>>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >>>>> >> >
> >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> >>>>> >> > we do not see the issue usually since branch-3.2 uses
> >>>>> zooekeper-3.4.13,
> >>>>> >> > while it would be harmless to add the exclusion even for
> >>>>> >> zooekeeper-3.4.13

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Please go ahead. Let me know any help required on review.

On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He  wrote:

> Hi Brahma,
>
> I plan to involve HDFS-14997 and related JIRAs if possible. I have
> resolved the conflict and verified them locally.
> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
> I would like to hear some more response that if we have enough time to
> wait for it to be ready.
> Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He  wrote:
>
>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI.
>>
>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula 
>> wrote:
>>
>>> Hi All,
>>>
>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly this
>>> can
>>> be done by this week,then I will try to create the RC next if there is no
>>> objection.
>>>
>>> https://issues.apache.org/jira/browse/HDFS-15160
>>>
>>>
>>>
>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula 
>>> wrote:
>>>
>>> > @Akira Ajisaka   and @Masatake Iwasaki
>>> > 
>>> > Looks all are build related issues when you try with bigtop. We can
>>> > discuss and prioritize this.. Will connect with you guys.
>>> >
>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
>>> > iwasak...@oss.nttdata.co.jp> wrote:
>>> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>>> >> >
>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>>> >> > we do not see the issue usually since branch-3.2 uses
>>> zooekeper-3.4.13,
>>> >> > while it would be harmless to add the exclusion even for
>>> >> zooekeeper-3.4.13.
>>> >>
>>> >> I filed HADOOP-17849 for this.
>>> >>
>>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote:
>>> >> > Thanks for bringing this up, Akira. Let me explain some background.
>>> >> >
>>> >> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>>> >> >
>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>>> >> > we do not see the issue usually since branch-3.2 uses
>>> zooekeper-3.4.13,
>>> >> > while it would be harmless to add the exclusion even for
>>> >> zooekeeper-3.4.13.
>>> >> >
>>> >> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>>> >> >
>>> >> > These are relevant to directory structure used by Bigtop package.
>>> >> > If the fix does not break the tarball dist,
>>> >> > it would be nice to have these on Hadoop too.
>>> >> >
>>> >> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>>> >> >
>>> >> > This is for aarch64 and ppe64le lacking required phantomjs.
>>> >> > It is only acceptable for Bigtop not running tests of YARN-UI2 on
>>> >> packaging.
>>> >> > Hadoop needs the phantomjs for testing YARN-UI2.
>>> >> >
>>> >> >
>>> >> > On 2021/08/16 2:59, Brahma Reddy Battula wrote:
>>> >> >> Thanks, I'll check them out.
>>> >> >>
>>> >> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka <
>>> aajis...@apache.org>
>>> >> wrote:
>>> >> >>
>>> >> >>> Thanks Brahma for cutting branch-3.2.3.
>>> >> >>>
>>> >> >>> In Apache Bigtop, there are some patches applied

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Thanks all for attending the session, Hope we can have some more Good
sessions.. Please find the recordings.

https://drive.google.com/drive/u/1/folders/1mtIEUfnEUtil65U0nGGeJNAUYVsPpktt

On Wed, Sep 8, 2021 at 7:37 PM Brahma Reddy Battula 
wrote:

> Hi Sean,
>
> Pacific Time 9th, Sept 9:30 AM…..
>
> Please check the following link relevant time zone..
>
> Time zone :: https://t.co/JmTN6zK8Ga?amp=1
>
>
>
> On Wed, 8 Sep 2021 at 7:12 PM, Sean Busbey  wrote:
>
>> Hi Brahma!
>>
>> Thanks for organizing this. What’s the timezone for the 10p - midnight?
>> Pacific Time?
>>
>> > On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula 
>> wrote:
>> >
>> > Hi All,
>> >
>> > Updated the meeting to record the session.. Please use the following
>> link
>> > to attend the conference tomorrow.
>> >
>> >
>> > Uber's story on running Apache Hadoop deployment in Docker
>> > September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
>> > Google Meet joining info
>> > *Video call link: https://meet.google.com/few-wppc-xoa
>> > <https://meet.google.com/few-wppc-xoa>*
>> > Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
>> > More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915
>> >
>> > On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula <
>> bra...@apache.org>
>> > wrote:
>> >
>> >> Hi All,
>> >>
>> >> Happy to announce the following conference. Block your calendar and
>> make
>> >> yourself available.
>> >> Thanks Mithun and team for accepting.
>> >>
>> >>
>> >> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
>> >>
>> >> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
>> >>
>> >> *Meeting Link :   *Google Meet joining info
>> >>   Video call link:
>> >> https://meet.google.com/akk-qmzy-qsu
>> >>
>> >> *  Note: *Created Teams meeting also as a
>> backup
>> >> [2].
>> >>
>> >> *High level Agenda: *
>> >>
>> >>   - Importance of Hadoop Cluster Management (3-5 min)
>> >>   - Introduction about problem space
>> >>   - Challenges in this space and why it is important to solve them
>> >>   - Evolution of Uber Hadoop Cluster Management (10-15 min)
>> >>   -  How our strategy evolved over time since Hadoop was set up at Uber
>> >>   -  Key learnings from the evolution over the years
>> >>   - Our current approach today (15-20min)
>> >>   -  Key learnings from this major overhaul
>> >>   -  Current challenges that we are working on
>> >>   - Q (20 min)
>> >>
>> >>
>> >> Aiming for 20 min Q assuming that there will be more questions based
>> on
>> >> the blog that was published(1) + presentation at the session.
>> >>
>> >>
>> >>
>> >>
>> >>
>> ---
>> >> 1) https://eng.uber.com/hadoop-container-blog/
>> >> <
>> https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22
>> >
>> >> 2)
>> >>
>> >> *Join on your computer or mobile app*
>> >>
>> >> Click here to join the meeting
>> >> <
>> https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d
>> >
>> >>
>> >> *Join with a video conferencing device*
>> >>
>> >> 967904...@t.plcm.vc
>> >>
>> >> Video Conference ID: 117 464 626 6
>> >>
>> >> Alternate VTC instructions
>> >> <https://dialin.plcm.vc/teams/?key=967904974=1174646266>
>> >>
>> >> *Or call in (audio only)*
>> >>
>> >> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
>> >> Los Angeles
>> >>
>> >> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
>> >> (Toll-free)
>> >>
>> >> Phone Conference ID: 273 569 658#
>> >>
>> >> Find a local number
>> >> <
>> https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658
>> >
>> >> | Reset PIN <https://mysettings.lync.com/pstnconferencing>
>> >>
>> >>
>> >>
>> >> -- Brahma Reddy Battula
>> >>
>> >
>> >
>> > --
>> >
>> >
>> >
>> > --Brahma Reddy Battula
>>
>>
>> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Hi All,

Waiting for the following jira to commit to hadoop-3.2.3 , mostly this can
be done by this week,then I will try to create the RC next if there is no
objection.

https://issues.apache.org/jira/browse/HDFS-15160



On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula 
wrote:

> @Akira Ajisaka   and @Masatake Iwasaki
> 
> Looks all are build related issues when you try with bigtop. We can
> discuss and prioritize this.. Will connect with you guys.
>
> On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>> >
>> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
>> > while it would be harmless to add the exclusion even for
>> zooekeeper-3.4.13.
>>
>> I filed HADOOP-17849 for this.
>>
>> On 2021/08/16 12:02, Masatake Iwasaki wrote:
>> > Thanks for bringing this up, Akira. Let me explain some background.
>> >
>> >
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>> >
>> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
>> > while it would be harmless to add the exclusion even for
>> zooekeeper-3.4.13.
>> >
>> >
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>> >
>> > These are relevant to directory structure used by Bigtop package.
>> > If the fix does not break the tarball dist,
>> > it would be nice to have these on Hadoop too.
>> >
>> >
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>> >
>> > This is for aarch64 and ppe64le lacking required phantomjs.
>> > It is only acceptable for Bigtop not running tests of YARN-UI2 on
>> packaging.
>> > Hadoop needs the phantomjs for testing YARN-UI2.
>> >
>> >
>> > On 2021/08/16 2:59, Brahma Reddy Battula wrote:
>> >> Thanks, I'll check them out.
>> >>
>> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka 
>> wrote:
>> >>
>> >>> Thanks Brahma for cutting branch-3.2.3.
>> >>>
>> >>> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
>> >>>
>> >>>
>> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
>> >>>
>> >>> In these patches, how about backporting the following issues to
>> >>> branch-3.2 and branch-3.2.3?
>> >>> - HADOOP-14922
>> >>> - HADOOP-15939
>> >>> - HADOOP-17569
>> >>>
>> >>> In addition, there are some patches that don't have JIRA issue ID.
>> >>> Maybe we need to create JIRAs and fix those.
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>> >>>
>> >>> Thanks and regards,
>> >>> Akira
>> >>>
>> >>> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He 
>> >>> wrote:
>> >>>>
>> >>>> Thanks Brahma for initiating this and making hadoop-3.2.3 release
>> happen.
>> >>>>
>> >>>> I would li

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Hi Sean,

Pacific Time 9th, Sept 9:30 AM…..

Please check the following link relevant time zone..

Time zone :: https://t.co/JmTN6zK8Ga?amp=1



On Wed, 8 Sep 2021 at 7:12 PM, Sean Busbey  wrote:

> Hi Brahma!
>
> Thanks for organizing this. What’s the timezone for the 10p - midnight?
> Pacific Time?
>
> > On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula 
> wrote:
> >
> > Hi All,
> >
> > Updated the meeting to record the session.. Please use the following link
> > to attend the conference tomorrow.
> >
> >
> > Uber's story on running Apache Hadoop deployment in Docker
> > September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
> > Google Meet joining info
> > *Video call link: https://meet.google.com/few-wppc-xoa
> > <https://meet.google.com/few-wppc-xoa>*
> > Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
> > More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915
> >
> > On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula  >
> > wrote:
> >
> >> Hi All,
> >>
> >> Happy to announce the following conference. Block your calendar and make
> >> yourself available.
> >> Thanks Mithun and team for accepting.
> >>
> >>
> >> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
> >>
> >> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
> >>
> >> *Meeting Link :   *Google Meet joining info
> >>   Video call link:
> >> https://meet.google.com/akk-qmzy-qsu
> >>
> >> *  Note: *Created Teams meeting also as a backup
> >> [2].
> >>
> >> *High level Agenda: *
> >>
> >>   - Importance of Hadoop Cluster Management (3-5 min)
> >>   - Introduction about problem space
> >>   - Challenges in this space and why it is important to solve them
> >>   - Evolution of Uber Hadoop Cluster Management (10-15 min)
> >>   -  How our strategy evolved over time since Hadoop was set up at Uber
> >>   -  Key learnings from the evolution over the years
> >>   - Our current approach today (15-20min)
> >>   -  Key learnings from this major overhaul
> >>   -  Current challenges that we are working on
> >>   - Q (20 min)
> >>
> >>
> >> Aiming for 20 min Q assuming that there will be more questions based
> on
> >> the blog that was published(1) + presentation at the session.
> >>
> >>
> >>
> >>
> >>
> ---
> >> 1) https://eng.uber.com/hadoop-container-blog/
> >> <
> https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22
> >
> >> 2)
> >>
> >> *Join on your computer or mobile app*
> >>
> >> Click here to join the meeting
> >> <
> https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d
> >
> >>
> >> *Join with a video conferencing device*
> >>
> >> 967904...@t.plcm.vc
> >>
> >> Video Conference ID: 117 464 626 6
> >>
> >> Alternate VTC instructions
> >> <https://dialin.plcm.vc/teams/?key=967904974=1174646266>
> >>
> >> *Or call in (audio only)*
> >>
> >> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
> >> Los Angeles
> >>
> >> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
> >> (Toll-free)
> >>
> >> Phone Conference ID: 273 569 658#
> >>
> >> Find a local number
> >> <
> https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658
> >
> >> | Reset PIN <https://mysettings.lync.com/pstnconferencing>
> >>
> >>
> >>
> >> -- Brahma Reddy Battula
> >>
> >
> >
> > --
> >
> >
> >
> > --Brahma Reddy Battula
>
>
> --



--Brahma Reddy Battula

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Hi All,

 Updated the meeting to record the session.. Please use the following link
to attend the conference tomorrow.


Uber's story on running Apache Hadoop deployment in Docker
September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
Google Meet joining info
*Video call link: https://meet.google.com/few-wppc-xoa
<https://meet.google.com/few-wppc-xoa>*
Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915

On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula 
wrote:

> Hi All,
>
> Happy to announce the following conference. Block your calendar and make
> yourself available.
> Thanks Mithun and team for accepting.
>
>
> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
>
> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
>
> *Meeting Link :   *Google Meet joining info
>Video call link:
> https://meet.google.com/akk-qmzy-qsu
>
> *  Note: *Created Teams meeting also as a backup
> [2].
>
> *High level Agenda: *
>
>- Importance of Hadoop Cluster Management (3-5 min)
>- Introduction about problem space
>- Challenges in this space and why it is important to solve them
>- Evolution of Uber Hadoop Cluster Management (10-15 min)
>-  How our strategy evolved over time since Hadoop was set up at Uber
>-  Key learnings from the evolution over the years
>- Our current approach today (15-20min)
>-  Key learnings from this major overhaul
>-  Current challenges that we are working on
>- Q (20 min)
>
>
> Aiming for 20 min Q assuming that there will be more questions based on
> the blog that was published(1) + presentation at the session.
>
>
>
>
> ---
> 1) https://eng.uber.com/hadoop-container-blog/
> <https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22>
> 2)
>
> *Join on your computer or mobile app*
>
> Click here to join the meeting
> <https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d>
>
> *Join with a video conferencing device*
>
> 967904...@t.plcm.vc
>
> Video Conference ID: 117 464 626 6
>
> Alternate VTC instructions
> <https://dialin.plcm.vc/teams/?key=967904974=1174646266>
>
> *Or call in (audio only)*
>
> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
> Los Angeles
>
> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
> (Toll-free)
>
> Phone Conference ID: 273 569 658#
>
> Find a local number
> <https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658>
>  | Reset PIN <https://mysettings.lync.com/pstnconferencing>
>
>
>
> -- Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

[Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Hi All,

Happy to announce the following conference. Block your calendar and make
yourself available.
Thanks Mithun and team for accepting.


*Topic:*Uber's story on running Apache Hadoop deployment in Docker

*Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*

*Meeting Link :   *Google Meet joining info
   Video call link:
https://meet.google.com/akk-qmzy-qsu

*  Note: *Created Teams meeting also as a backup
[2].

*High level Agenda: *

   - Importance of Hadoop Cluster Management (3-5 min)
   - Introduction about problem space
   - Challenges in this space and why it is important to solve them
   - Evolution of Uber Hadoop Cluster Management (10-15 min)
   -  How our strategy evolved over time since Hadoop was set up at Uber
   -  Key learnings from the evolution over the years
   - Our current approach today (15-20min)
   -  Key learnings from this major overhaul
   -  Current challenges that we are working on
   - Q (20 min)


Aiming for 20 min Q assuming that there will be more questions based on
the blog that was published(1) + presentation at the session.



---
1) https://eng.uber.com/hadoop-container-blog/
<https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22>
2)

*Join on your computer or mobile app*

Click here to join the meeting
<https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d>

*Join with a video conferencing device*

967904...@t.plcm.vc

Video Conference ID: 117 464 626 6

Alternate VTC instructions
<https://dialin.plcm.vc/teams/?key=967904974=1174646266>

*Or call in (audio only)*

+1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States, Los
Angeles

(833) 827-4491,,273569658# <8338274491,,273569658#>   United States
(Toll-free)

Phone Conference ID: 273 569 658#

Find a local number
<https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658>
 | Reset PIN <https://mysettings.lync.com/pstnconferencing>



-- Brahma Reddy Battula

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

@Akira Ajisaka   and @Masatake Iwasaki

Looks all are build related issues when you try with bigtop. We can discuss
and prioritize this.. Will connect with you guys.

On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >
> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
> > while it would be harmless to add the exclusion even for
> zooekeeper-3.4.13.
>
> I filed HADOOP-17849 for this.
>
> On 2021/08/16 12:02, Masatake Iwasaki wrote:
> > Thanks for bringing this up, Akira. Let me explain some background.
> >
> >
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >
> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
> > while it would be harmless to add the exclusion even for
> zooekeeper-3.4.13.
> >
> >
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
> >
> > These are relevant to directory structure used by Bigtop package.
> > If the fix does not break the tarball dist,
> > it would be nice to have these on Hadoop too.
> >
> >
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
> >
> > This is for aarch64 and ppe64le lacking required phantomjs.
> > It is only acceptable for Bigtop not running tests of YARN-UI2 on
> packaging.
> > Hadoop needs the phantomjs for testing YARN-UI2.
> >
> >
> > On 2021/08/16 2:59, Brahma Reddy Battula wrote:
> >> Thanks, I'll check them out.
> >>
> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka 
> wrote:
> >>
> >>> Thanks Brahma for cutting branch-3.2.3.
> >>>
> >>> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
> >>>
> >>>
> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
> >>>
> >>> In these patches, how about backporting the following issues to
> >>> branch-3.2 and branch-3.2.3?
> >>> - HADOOP-14922
> >>> - HADOOP-15939
> >>> - HADOOP-17569
> >>>
> >>> In addition, there are some patches that don't have JIRA issue ID.
> >>> Maybe we need to create JIRAs and fix those.
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
> >>>
> >>> Thanks and regards,
> >>> Akira
> >>>
> >>> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He 
> >>> wrote:
> >>>>
> >>>> Thanks Brahma for initiating this and making hadoop-3.2.3 release
> happen.
> >>>>
> >>>> I would like to validate the HBase project (both the latest release
> and
> >>>> trunk branch).
> >>>> Chao Sun will validate the Spark Project (Got in touch with Chao
> >>> already).
> >>>> once RC is out.
> >>>>
> >>>> Thanks and Regards,
> >>>> - He Xiaoqiao
> >>>>
> >>>>
> >>>> On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula <
> bra...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Thanks, I'll check them out.

On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka  wrote:

> Thanks Brahma for cutting branch-3.2.3.
>
> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
>
> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
>
> In these patches, how about backporting the following issues to
> branch-3.2 and branch-3.2.3?
> - HADOOP-14922
> - HADOOP-15939
> - HADOOP-17569
>
> In addition, there are some patches that don't have JIRA issue ID.
> Maybe we need to create JIRAs and fix those.
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>
> Thanks and regards,
> Akira
>
> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He 
> wrote:
> >
> > Thanks Brahma for initiating this and making hadoop-3.2.3 release happen.
> >
> > I would like to validate the HBase project (both the latest release and
> > trunk branch).
> > Chao Sun will validate the Spark Project (Got in touch with Chao
> already).
> > once RC is out.
> >
> > Thanks and Regards,
> > - He Xiaoqiao
> >
> >
> > On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula 
> > wrote:
> >
> > > Hi All,
> > >
> > > I cut branch-3.2.3 and it is ready for release. Please commit to
> > > branch-3.2.3 if any critical/blocker issues need to go.
> > >
> > > *This time I am thinking of having downstream projects and companies'
> > > voices,let's know how this can go. *
> > >
> > >- Planning to check with downstream projects like Spark,HBase and
> Hive
> > >if they can help on validation(Or running their UT on this branch)
> > >- Collecting info from Companies who are already deployed and using
> the
> > >branch-3.2
> > >
> > >
> > > so that we can make a more stable release on 3.2 (so that features
> released
> > > on this branch impact can be known) ,please let me know anybody from
> these
> > > communities who can help on this.
> > >
> > >
> > > Planning to create RC by this month end. Any suggestions are welcome.
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
>
-- 



--Brahma Reddy Battula

Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Hi All,

I cut branch-3.2.3 and it is ready for release. Please commit to
branch-3.2.3 if any critical/blocker issues need to go.

*This time I am thinking of having downstream projects and companies'
voices,let's know how this can go. *

   - Planning to check with downstream projects like Spark,HBase and Hive
   if they can help on validation(Or running their UT on this branch)
   - Collecting info from Companies who are already deployed and using the
   branch-3.2


so that we can make a more stable release on 3.2 (so that features released
on this branch impact can be known) ,please let me know anybody from these
communities who can help on this.


Planning to create RC by this month end. Any suggestions are welcome.



--Brahma Reddy Battula

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Brahma Reddy Battula]

@Wei-Chiu Chuang  looks this is not concluded yet...
Can we move forward..?

On Thu, Jul 15, 2021 at 11:09 PM Brahma Reddy Battula 
wrote:

>
> I agree with Ahmed Hussein…Jira should not be used for number generation..
>
> We can always revisit the jira to see useful discussion at one place…
>
> @wei-chu, +1 on proposal for cleaning the PR’s..
>
>
> On Thu, 15 Jul 2021 at 9:15 PM, epa...@apache.org 
> wrote:
>
>>  > I usually use PR comments to discuss about the patch submitted.
>> My concern is that still leaves multiple places to look in order to get a
>> full picture of an issue.
>> -Eric
>>
>> On Wednesday, July 14, 2021, 7:07:30 PM CDT, Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp> wrote:
>>
>>  > - recently, JIRA became some sort of a "number generator" with
>> insufficient
>> > description/details as the
>> >developers and the reviewers spending more time discussing in the PR.
>>
>> JIRA issues contain useful information in the fields.
>> We are leveraging them in development and release process.
>>
>> * https://yetus.apache.org/documentation/0.13.0/releasedocmaker/
>> *
>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336122
>>
>> I usually use PR comments to discuss about the patch submitted.
>> JIRA comments are used for background or design discussion before and
>> after submitting PR.
>> There would be no problem having no comment in minor/trivial JIRA issues.
>>
>>
>> On 2021/07/14 23:50, Ahmed Hussein wrote:
>> > Do you consider migrating Jira issues to Github issues?
>> >
>> > I am a little bit concerned that there are some committers who still
>> prefer
>> > Jira-precommits over GitHub PR
>> > (P.S. I am not a committer).
>> >
>> > Their point is that Github-PR confuses them with discussions/comments
>> being
>> > in two places rather than one.
>> >
>> > Personally, I found several Github-PRs comments discussing the validity
>> of
>> > the feature/bug.
>> > As a result:
>> > - recently, JIRA became some sort of a "number generator" with
>> insufficient
>> > description/details as the
>> >developers and the reviewers spending more time discussing in the PR.
>> > - the relation between a single Jira and Github-PR is 1-to-M. In order
>> to
>> > find related discussions, the user may
>> >need to visit every PR (that may include closed ones)
>> >
>> >
>> >
>> > On Wed, Jul 14, 2021 at 8:46 AM Steve Loughran
>> 
>> > wrote:
>> >
>> >> not sure about stale PR closing; when you've a patch which is still
>> pending
>> >> review it's not that fun to have it closed.
>> >>
>> >> maybe better to have review sessions. I recall many, many years ago
>> >> attempts to try and catch up with all outstanding patch reviews.
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, 14 Jul 2021 at 03:00, Akira Ajisaka 
>> wrote:
>> >>
>> >>> Thank you Wei-Chiu for starting the discussion,
>> >>>
>> >>>> 3. JIRA security
>> >>> I'm +1 to use private JIRA issues to handle vulnerabilities.
>> >>>
>> >>>> 5. Doc update
>> >>> +1, I build the document daily and it helps me fixing documents:
>> >>> https://aajisaka.github.io/hadoop-document/ It's great if the latest
>> >>> document is built and published by the Apache Hadoop community.
>> >>>
>> >>> My idea related to GitHub PR:
>> >>> 1. Disable the precommit jobs for JIRA, always use GitHub PR. It saves
>> >>> costs to configure and debug the precommit jobs.
>> >>> https://issues.apache.org/jira/browse/HADOOP-17798
>> >>> 2. Improve the pull request template for the contributors
>> >>> https://issues.apache.org/jira/browse/HADOOP-17799
>> >>>
>> >>> Regards,
>> >>> Akira
>> >>>
>> >>> On Tue, Jul 13, 2021 at 12:35 PM Wei-Chiu Chuang 
>> >>> wrote:
>> >>>>
>> >>>> I work on multiple projects and learned a bunch from those
>> >> projects.There
>> >>>> are nice add-ons that help with productivity. There are things we can
>> >> do
>> >>> to
>> >>>> help us manage the project better.
>>

Re: [External Sender] Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

**Important update**  Hadoop-3.2.3 Release progress.


I cut branch-3.2.3 and it is ready for release. Please commit to
branch-3.2.3 if any critical/blocker issues need to go.
As previously communicated,Planning to create RC by this month end. Any
suggestions are welcome.






On Thu, Aug 5, 2021 at 10:15 PM Jason Wen  wrote:

> Hi Steve,
>
> Which aws-java-sdk-bundle version did you test and encounter the following
> test failures?
>
> Thanks,
> Jason
>
> On 8/4/21, 7:51 AM, "Steve Loughran"  wrote:
>
> it didn't take, so don't worry about it. Test failures
>
> java.lang.IllegalArgumentException: Input is expected to be encoded in
> multiple of 4 bytes but found: 75
> at com.amazonaws.util.Base64Codec.decode(Base64Codec.java:198)
> at com.amazonaws.util.Base64.decode(Base64.java:116)
> at
>
> com.amazonaws.services.s3.AmazonS3Client.populateSSE_C(AmazonS3Client.java:4620)
> at
>
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1806)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1603)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$13(S3AFileSystem.java:2828)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2826)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2801)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2146)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2079)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2304)
> at
>
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
> at
>
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)
>
> On Tue, 3 Aug 2021 at 19:41, Brahma Reddy Battula 
> wrote:
>
> > Hi Steve,
> >
> > Is there any jira for this ??
> >
> >
> >
> > On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran
>  > >
> > wrote:
> >
> > > I'm testing how well a backport of the latest AWS SDK goes; if all
> is
> > > docile there then I'll merge that. Due diligence more than anything
> > > elseone of the shaded netty JARs (never used by our code) has
> a CVE
> > in
> > > older versions
> > >
> > > On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka 
> wrote:
> > >
> > > > Hi Steven,
> > > >
> > > > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
> > > > opened a PR to backport YARN-8990:
> > > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_pull_3254=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o=dXWDhhZW6SZQt-9oQONYtGPMFTrUL4225rMKK-494YA=tLml0Gisl-xPsd4B67BNBZtPsz1T0qBPppHYRzLLl_0=
> > > >
> > > > Thanks,
> > > > Akira
> > > >
> > > > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand <
> stevenjr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > I think it would be helpful if we could include YARN-8990 and
> > YARN-8992
> > > > in the 3.2.3 release. Both are important fixes which were
> included in
> > > > 3.2.0, but never made their way to branch-3.2, so were omitted
> from
> > both
> > > > 3.2.1 and 3.2.2.
> > > > >
> > > > > Best,
> > > > > Steve
> > > > >
> > > > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He <
> xq.he2...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> cc @dev mail-list.
> > > > >>
> > > > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He <
> xq.he2...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > Hi Brahma,
> > > > >> >
>     > > > >> > I just created

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Hi Steve,

Is there any jira for this ??



On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran 
wrote:

> I'm testing how well a backport of the latest AWS SDK goes; if all is
> docile there then I'll merge that. Due diligence more than anything
> elseone of the shaded netty JARs (never used by our code) has a CVE in
> older versions
>
> On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka  wrote:
>
> > Hi Steven,
> >
> > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
> > opened a PR to backport YARN-8990:
> > https://github.com/apache/hadoop/pull/3254
> >
> > Thanks,
> > Akira
> >
> > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand 
> > wrote:
> > >
> > > I think it would be helpful if we could include YARN-8990 and YARN-8992
> > in the 3.2.3 release. Both are important fixes which were included in
> > 3.2.0, but never made their way to branch-3.2, so were omitted from both
> > 3.2.1 and 3.2.2.
> > >
> > > Best,
> > > Steve
> > >
> > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He 
> wrote:
> > >>
> > >> cc @dev mail-list.
> > >>
> > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He 
> > wrote:
> > >>
> > >> > Hi Brahma,
> > >> >
> > >> > I just created version 3.2.4, and changed all unresolved issues
> > (target
> > >> > version/s: 3.2.3) to 3.2.4 after checking both of them are not
> blocker
> > >> > issues. Dashboard[1] is clean now.
> > >> >
> > >> > Regards,
> > >> > - He Xiaoqiao
> > >> >
> > >> > [1]
> > >> >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> > >> >
> > >> > On Sun, Jul 25, 2021 at 7:45 PM Brahma Reddy Battula <
> > bra...@apache.org>
> > >> > wrote:
> > >> >
> > >> >> Hi Xiaoqiao,
> > >> >>
> > >> >> Thanks for creating the Dashboard, we need to change the filters
> and
> > >> >> target versions in the jira.
> > >> >>
> > >> >> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He 
> > wrote:
> > >> >>
> > >> >>> Thanks Brahma for volunteering and driving this release plan. I
> just
> > >> >>> created a dashboard for 3.2.3 release[1].
> > >> >>> I would like to support for this release line if need. (cc Brahma)
> > >> >>>
> > >> >>> Thanks. Regards,
> > >> >>> - He Xiaoqiao
> > >> >>>
> > >> >>> [1]
> > >> >>>
> > >> >>>
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> > >> >>>
> > >> >>>
> > >> >>> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka <
> aajis...@apache.org>
> > >> >>> wrote:
> > >> >>>
> > >> >>> > Hi Brahma,
> > >> >>> >
> > >> >>> > Thank you for volunteering!
> > >> >>> >
> > >> >>> > -Akira
> > >> >>> >
> > >> >>> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula <
> > >> >>> bra...@apache.org>
> > >> >>> > wrote:
> > >> >>> > >
> > >> >>> > > Hi Akira,
> > >> >>> > >
> > >> >>> > > Thanks for bringing this..
> > >> >>> > >
> > >> >>> > > I want to drive this if nobody already plan to do this..
> > >> >>> > >
> > >> >>> > >
> > >> >>> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka <
> > aajis...@apache.org>
> > >> >>> > wrote:
> > >> >>> > >
> > >> >>> > > > Hi all,
> > >> >>> > > >
> > >> >>> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
> > >> >>> > > > accumulated more than 230 commits [1]. Therefore I want to
> > start
> > >> >>> the
> > >> >>> > > > release work for 3.2.3.
> > >> &

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Hi Xiaoqiao,

Thanks for creating the Dashboard, we need to change the filters and target
versions in the jira.

On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He  wrote:

> Thanks Brahma for volunteering and driving this release plan. I just
> created a dashboard for 3.2.3 release[1].
> I would like to support for this release line if need. (cc Brahma)
>
> Thanks. Regards,
> - He Xiaoqiao
>
> [1]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
>
>
> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka  wrote:
>
> > Hi Brahma,
> >
> > Thank you for volunteering!
> >
> > -Akira
> >
> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula 
> > wrote:
> > >
> > > Hi Akira,
> > >
> > > Thanks for bringing this..
> > >
> > > I want to drive this if nobody already plan to do this..
> > >
> > >
> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
> > > > accumulated more than 230 commits [1]. Therefore I want to start the
> > > > release work for 3.2.3.
> > > >
> > > > There is one blocker for 3.2.3 [2].
> > > > - https://issues.apache.org/jira/browse/HDFS-12920
> > > >
> > > > Is there anyone who would volunteer to be the 3.2.3 release manager?
> > > > Are there any other blockers? If any, please file an issue, raise the
> > > > blocker, and add the target version.
> > > >
> > > > [1]
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
> > > > [2]
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
> > > >
> > > > Regards,
> > > > Akira
> > > >
> > > > -----
> > > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail:
> mapreduce-dev-h...@hadoop.apache.org
> > > >
> > > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Hi Akira,

Thanks for bringing this..

I want to drive this if nobody already plan to do this..


On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka  wrote:

> Hi all,
>
> Hadoop 3.2.2 was released half a year ago, and now, we have
> accumulated more than 230 commits [1]. Therefore I want to start the
> release work for 3.2.3.
>
> There is one blocker for 3.2.3 [2].
> - https://issues.apache.org/jira/browse/HDFS-12920
>
> Is there anyone who would volunteer to be the 3.2.3 release manager?
> Are there any other blockers? If any, please file an issue, raise the
> blocker, and add the target version.
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
> [2]
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
>
> Regards,
> Akira
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
> --



--Brahma Reddy Battula

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Brahma Reddy Battula]

;> can perform static analysis, catching potential issues. For example,
> >>> Ozone
> >>>> adds SonarQube to post-commit, and exports the report to SonarCloud.
> >>> Other
> >>>> add-ons are available to scan for docker images, vulnerabilities
> scans.
> >>>>
> >>>> 3. JIRA security
> >>>> It is possible to set up security level (public/private) in JIRA. This
> >>> can
> >>>> be used to track vulnerability issues and be made only visible to
> >>>> committers. Example: INFRA-15258
> >>>> <https://issues.apache.org/jira/browse/INFRA-15258>
> >>>>
> >>>> 4. New JIRA fields
> >>>> It's possible to add new fields. For example, we can add a "Reviewer"
> >>>> field, which could help improve the attention to issues.
> >>>>
> >>>> 5. Doc update
> >>>> It is possible to set up automation such that the doc on the Hadoop
> >>> website
> >>>> is refreshed for every commit, providing the latest doc to the public.
> >>>>
> >>>> 6. Webhook
> >>>> It's possible to set up webhook such that every commit in GitHub sends
> >> a
> >>>> notification to the ASF slack. It can be used for other kinds of
> >>>> automation. Sky's the limit.
> >>>>
> >>>> Thoughts? What else can do we?
> >>>
> >>> -
> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>
> >
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

-- 



--Brahma Reddy Battula

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Wei-Chiu,

Thanks for driving this..


Looks index.html is not updated and Changelog is broken.


On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.3.1.
>
> This is the first stable release of Apache Hadoop 3.3.x line. It contains
> 697 bug fixes, improvements and enhancements since 3.3.0.
>
> Users are encouraged to read the overview of major changes
> <https://hadoop.apache.org/docs/r3.3.1/index.html> since 3.3.0. For
> details
> of 697 bug fixes, improvements, and other enhancements since the previous
> 3.3.0 release, please check release notes
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> >
>  and changelog
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> >
> detail
> the changes since 3.3.0.
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
> release process.
>
> Best Regards,
> Wei-Chiu Chuang
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Bilwa,

I have commented on the jira's you mentioned. Based on the stability we can
plan this.But needs to be merged ASAP.



On Fri, Feb 19, 2021 at 5:20 PM bilwa st  wrote:

> Hi Brahma,
>
> Can we have below features in 3.3.1 release? We have been using these
> features for a long time. They are stable and tested in bigger clusters.
>
> 1. Container reuse - https://issues.apache.org/jira/browse/MAPREDUCE-6749
> 2. Speculative attempts should not run on the same node -
> https://issues.apache.org/jira/browse/MAPREDUCE-7169
>
> Thanks,
> Bilwa
>
> On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula 
> wrote:
>
>> Sorry for the late reply..
>>
>> I will come up with a plan.. Please let me know if anybody has some
>> features/improvements/bugs that need to be included.
>>
>> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan  wrote:
>>
>> > Hi Wei-Chiu,
>> >
>> > What will be the next steps here for 3.3.1 planning?
>> >
>> > Thanks
>> > Sunil
>> >
>> > On Mon, Feb 8, 2021 at 11:56 PM Stack  wrote:
>> >
>> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
>> > > > > >
>> > > wrote:
>> > >
>> > > >
>> > > > Regarding blockers : how about we have a little hackathon where we
>> > try
>> > > > and get things in. This means a promise of review time from the
>> people
>> > > with
>> > > > commit rights and other people who understand the code (Stack?)
>> > > >
>> > > >
>> > >
>> > > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing).
>> > > Thanks,
>> > > S
>> > >
>> > >
>> > >
>> > >
>> > > > -steve
>> > > >
>> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena 
>> wrote:
>> > > >
>> > > > > +1
>> > > > > Just to mention we would need to release hadoop-thirdparty too
>> > before.
>> > > > > Presently we are using the snapshot version of it.
>> > > > >
>> > > > > -Ayush
>> > > > >
>> > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang > >
>> > > > wrote:
>> > > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've
>> > > > > accumulated
>> > > > > > more than 400 changes in the branch-3.3. A number of
>> downstreamers
>> > > are
>> > > > > > eagerly waiting for 3.3.1 which addresses the guava version
>> > conflict
>> > > > > issue.
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
>> > > > > >
>> > > > > > We should start the release work for 3.3.1 before the diff
>> becomes
>> > > even
>> > > > > > larger.
>> > > > > >
>> > > > > > I believe there are  currently only two real blockers for a
>> 3.3.1
>> > > > (using
>> > > > > > this filter
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
>> > > > > > )
>> > > > > >
>> > > > > >
>> > > > > >   1. HDFS-15566 <
>> https://issues.apache.org/jira/browse/HDFS-15566>
>> > > > > >   2.
>> > > > > >  1. HADOOP-17112 <
>> > > > https://issues.apache.org/jira/browse/HADOOP-17112
>> > > > > >
>> > > > > > 2.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Is there anyone who would volunteer to be the 3.3.1 RM?
>> > > > > >
>> > > > > > Also, the HowToRelease wiki does not describe the ARM build
>> > process.
>> > > > > That's
>> > > > > > going to be important for future releases.
>> > > > >
>> > > > >
>> -
>> > > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>

-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.1 release

[Brahma Reddy Battula]

Sorry for the late reply..

I will come up with a plan.. Please let me know if anybody has some
features/improvements/bugs that need to be included.

On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan  wrote:

> Hi Wei-Chiu,
>
> What will be the next steps here for 3.3.1 planning?
>
> Thanks
> Sunil
>
> On Mon, Feb 8, 2021 at 11:56 PM Stack  wrote:
>
> > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
>  > >
> > wrote:
> >
> > >
> > > Regarding blockers : how about we have a little hackathon where we
> try
> > > and get things in. This means a promise of review time from the people
> > with
> > > commit rights and other people who understand the code (Stack?)
> > >
> > >
> >
> > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing).
> > Thanks,
> > S
> >
> >
> >
> >
> > > -steve
> > >
> > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena  wrote:
> > >
> > > > +1
> > > > Just to mention we would need to release hadoop-thirdparty too
> before.
> > > > Presently we are using the snapshot version of it.
> > > >
> > > > -Ayush
> > > >
> > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've
> > > > accumulated
> > > > > more than 400 changes in the branch-3.3. A number of downstreamers
> > are
> > > > > eagerly waiting for 3.3.1 which addresses the guava version
> conflict
> > > > issue.
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
> > > > >
> > > > > We should start the release work for 3.3.1 before the diff becomes
> > even
> > > > > larger.
> > > > >
> > > > > I believe there are  currently only two real blockers for a 3.3.1
> > > (using
> > > > > this filter
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
> > > > > )
> > > > >
> > > > >
> > > > >   1. HDFS-15566 <https://issues.apache.org/jira/browse/HDFS-15566>
> > > > >   2.
> > > > >  1. HADOOP-17112 <
> > > https://issues.apache.org/jira/browse/HADOOP-17112
> > > > >
> > > > > 2.
> > > > >
> > > > >
> > > > >
> > > > > Is there anyone who would volunteer to be the 3.3.1 RM?
> > > > >
> > > > > Also, the HowToRelease wiki does not describe the ARM build
> process.
> > > > That's
> > > > > going to be important for future releases.
> > > >
> > > > -
> > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > > >
> > > >
> > >
> >
>


-- 



--Brahma Reddy Battula

Re: [VOTE] Moving Ozone to a separated Apache project

[Brahma Reddy Battula]

+1


On Wed, 30 Sep 2020 at 6:06 AM, Wei-Chiu Chuang  wrote:

> +1
>
>
>
> Look forward to it.
>
>
>
> On Tue, Sep 29, 2020 at 4:48 PM Konstantin Shvachko 
>
> wrote:
>
>
>
> > +1
>
> >
>
> > Stay safe,
>
> > --Konstantin
>
> >
>
> > On Thu, Sep 24, 2020 at 10:59 PM Elek, Marton  wrote:
>
> >
>
> > > Hi all,
>
> > >
>
> > > Thank you for all the feedback and requests,
>
> > >
>
> > > As we discussed in the previous thread(s) [1], Ozone is proposed to be
> a
>
> > > separated Apache Top Level Project (TLP)
>
> > >
>
> > > The proposal with all the details, motivation and history is here:
>
> > >
>
> > >
>
> > >
>
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Hadoop+subproject+to+Apache+TLP+proposal
>
> > >
>
> > > This voting runs for 7 days and will be concluded at 2nd of October,
> 6AM
>
> > > GMT.
>
> > >
>
> > > Thanks,
>
> > > Marton Elek
>
> > >
>
> > > [1]:
>
> > >
>
> > >
>
> >
> https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
>
> > >
>
> > > -
>
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
> > >
>
> > >
>
> >
>
> --



--Brahma Reddy Battula

Re: [Virtual MEETUP]: Migration to Hadoop 3

[Brahma Reddy Battula]

One more update from me.

We didn't face any issues with YARN, for HDFS you can have a look at the
following jira's.

https://issues.apache.org/jira/browse/HDFS-13596
https://issues.apache.org/jira/browse/HDFS-14396
https://issues.apache.org/jira/browse/HDFS-14509

Following jira is incompatible for ACL commands.Only hadoop-3 clients will
work against hadoop-3 server during the upgrade.

https://issues.apache.org/jira/browse/HDFS-6984



On Wed, Aug 26, 2020 at 11:06 PM Brahma Reddy Battula 
wrote:

>
> Hi Eric,
>
> check the following references for the same.
>
> 01/02/2020 Didi talked about their large scale HDFS cluster upgrade
> experience.
>
> Slides:
> https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy
>
> Recording:
> https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl
>
> Didi studied two upgrade approaches from the community documentation:
> express upgrade and rolling upgrade. Rolling upgrade was selected.
>
> Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1
>
> https://techblog.yahoo.co.jp/entry/20191206786320/
>
> On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org 
> wrote:
>
>> Hello. Just a reminder that today I would like to invite you all to
>> discuss your
>> experiences migrating from Hadoop 2 to Hadoop 3.
>>
>> -Eric
>>
>> On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org <
>> epa...@apache.org> wrote:
>>
>> Hello everyone!
>>
>> We are considering migrating to Hadoop 3, and we would be very interested
>> to
>> hear about your experiences. If you have migrated from Hadoop 2 to Hadoop
>> 3
>> and can provide insights, please kindly consider attending the following:
>>
>> Date: Wednesday, Aug 26, 2020
>> Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M. GMT
>> Location: Zoom: https://cloudera.zoom.us/j/880548968
>>
>> Hope to see you there!
>>
>> Thank you!
>> Eric Payne
>> @ Verizon Media
>>
>> -------------
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [Virtual MEETUP]: Migration to Hadoop 3

[Brahma Reddy Battula]

Hi Eric,

check the following references for the same.

01/02/2020 Didi talked about their large scale HDFS cluster upgrade
experience.

Slides:
https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy

Recording:
https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl

Didi studied two upgrade approaches from the community documentation:
express upgrade and rolling upgrade. Rolling upgrade was selected.

Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1

https://techblog.yahoo.co.jp/entry/20191206786320/

On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org  wrote:

> Hello. Just a reminder that today I would like to invite you all to
> discuss your
> experiences migrating from Hadoop 2 to Hadoop 3.
>
> -Eric
>
> On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org <
> epa...@apache.org> wrote:
>
> Hello everyone!
>
> We are considering migrating to Hadoop 3, and we would be very interested
> to
> hear about your experiences. If you have migrated from Hadoop 2 to Hadoop 3
> and can provide insights, please kindly consider attending the following:
>
> Date: Wednesday, Aug 26, 2020
> Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M. GMT
> Location: Zoom: https://cloudera.zoom.us/j/880548968
>
> Hope to see you there!
>
> Thank you!
> Eric Payne
> @ Verizon Media
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

-- 



--Brahma Reddy Battula

Re: Mandarin Hadoop online sync this week

[Brahma Reddy Battula]

HI,

what you are planning for this week?

On Tue, Aug 25, 2020 at 10:18 PM Wei-Chiu Chuang  wrote:

> Hello,
>
> There hasn't been a Mandarin online sync for quite some time. I'd like to
> call for one this week:
>
> Date/time:
>
> 8/27 Thursday Beijing Time 1PM
> 8/26 Wednesday US Pacific Time 10PM
>
> Link:
> https://cloudera.zoom.us/j/880548968
>
> Past sync summary:
>
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit
>


-- 



--Brahma Reddy Battula

[ANNOUNCE] Apache Hadoop 3.3.0 release

[Brahma Reddy Battula]

Greetings all,

It gives me great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.3.0.

Apache Hadoop 3.3.0 is the first release of Apache Hadoop 3.3 line for the
year 2020,
which includes 2148 fixes since the previous Hadoop 3.2.0 release.


Of these fixes:
   - 525 in Hadoop Common
   - 804 in HDFS
   - 763 in YARN
   - 56 in MapReduce

Apache Hadoop 3.3.0 contains a number of significant features and
enhancements.A few of them are noted as below.


- *Support ARM:* This is the first release to support ARM architectures

- *Upgrade protobuf from 2.5.0 : *Protobuf upgraded to 3.7.1 as
protobuf-2.5.0 reached EOL.

- *S3A Enhancements : *Lots of enhancements to the S3A code including
Delegation Token support, better handling of 404 caching, S3guard
performance, resilience improvements

- *ABFS Enhancements :* Address issues which surface in the field and tune
things which need tuning, add more tests where appropriate.Improve docs,
especially troubleshooting.

- *Tencent Cloud COS File System Implementation: *Tencent cloud is top
2 cloud vendors in China market and the object store COS is widely
used among China’s cloud users. COSN filesytem to support Tencent
cloud COS natively in Hadoop.

-  *Java11 Runtime Support* : Java 11 runtime support is completed.

- *HDFS RBF stabilizatio**n*: HDFS Router now supports security. Also
contains many bug fixes and improvements.

- *DNS Resolution to support Nameservices to IP Support:*DFS clients can
use a single domain name to discover servers (namenodes/routers/observers)
instead of explicitly listing out all hosts in the config

*- Scheduling of opportunistic containers :  S*cheduling of opportunistic
container through the central RM (YARN-5220), through distributed
scheduling (YARN-2877), as well as the scheduling of containers based on
actual node utilization (YARN-1011) and the container
promotion/demotion (YARN-5085).

*- Application Catalog for YARN applications: **Application catalog
system which provides an editorial and search interface for YARN
applications.*This improves the usability of YARN for managing the
life cycle of applications.

* For major changes included in Hadoop 3.3 line, please refer to Hadoop3.3.0
main page [1].


* For more details about fixes in 3.3.0 release, please read the CHANGELOG
[2] and RELEASENOTES [3].

The release news is posted on the Hadoop website too, you can go to the
downloads section directly [4].

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! This release is a direct result of your great
contributions.

Many thanks to Vinayakumar B who helped in this release process.

[1] https://hadoop.apache.org/docs <https://hadoop.apache.org/docs/r3.2.0/>
[2]
https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/CHANGELOG.3.3.0.html
[3]
https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/RELEASENOTES.3.3.0.html
[4] https://hadoop.apache.org/rele <https://hadoop.apache.org/releases.html>
ases.html





Regards,
Brahma Reddy Battula

Re: [RESULT][VOTE] Rlease Apache Hadoop-3.3.0

[Brahma Reddy Battula]

Hi Stephen,

thanks for bringing this to my attention.

Looks it's late..I pushed the release tag ( which can't be reverted) and
updated the release date in the jira.

Can we plan this next release near future..?


On Wed, Jul 15, 2020 at 5:25 PM Stephen O'Donnell
 wrote:

> Hi All,
>
> Sorry for being a bit late to this, but I wonder if we have a potential
> blocker to this release.
>
> In Cloudera we have recently encountered a serious dataloss issue in HDFS
> surrounding snapshots. To hit the dataloss issue, you must have HDFS-13101
> and HDFS-15012 on the build (which branch-3.3.0 does). To prevent it, you
> must also have HDFS-15313 and unfortunately, this was only committed to
> trunk, so we need to cherry-pick it down the active branches.
>
> With data loss being a serious issue, should we pull this Jira into
> branch-3.3.0 and cut a new release candidate?
>
> Thanks,
>
> Stephen.
>
> On Tue, Jul 14, 2020 at 1:22 PM Brahma Reddy Battula 
> wrote:
>
> > Hi All,
> >
> > With 8 binding and 11 non-binding +1s and no -1s the vote for Apache
> > hadoop-3.3.0 Release
> > passes.
> >
> > Thank you everybody for contributing to the release, testing, and voting.
> >
> > Special thanks whoever verified the ARM Binary as this is the first
> release
> > to support the ARM in hadoop.
> >
> >
> > Binding +1s
> >
> > =====
> > Akira Ajisaka
> > Vinayakumar B
> > Inigo Goiri
> > Surendra Singh Lilhore
> > Masatake Iwasaki
> > Rakesh Radhakrishnan
> > Eric Badger
> > Brahma Reddy Battula
> >
> > Non-binding +1s
> >
> > =
> > Zhenyu Zheng
> > Sheng Liu
> > Yikun Jiang
> > Tianhua huang
> > Ayush Saxena
> > Hemanth Boyina
> > Bilwa S T
> > Takanobu Asanuma
> > Xiaoqiao He
> > CR Hota
> > Gergely Pollak
> >
> > I'm going to work on staging the release.
> >
> >
> > The voting thread is:
> >
> >  https://s.apache.org/hadoop-3.3.0-Release-vote-thread
> >
> >
> >
> > --Brahma Reddy Battula
> >
>


-- 



--Brahma Reddy Battula

[RESULT][VOTE] Rlease Apache Hadoop-3.3.0

[Brahma Reddy Battula]

Hi All,

With 8 binding and 11 non-binding +1s and no -1s the vote for Apache
hadoop-3.3.0 Release
passes.

Thank you everybody for contributing to the release, testing, and voting.

Special thanks whoever verified the ARM Binary as this is the first release
to support the ARM in hadoop.


Binding +1s

=
Akira Ajisaka
Vinayakumar B
Inigo Goiri
Surendra Singh Lilhore
Masatake Iwasaki
Rakesh Radhakrishnan
Eric Badger
Brahma Reddy Battula

Non-binding +1s

=
Zhenyu Zheng
Sheng Liu
Yikun Jiang
Tianhua huang
Ayush Saxena
Hemanth Boyina
Bilwa S T
Takanobu Asanuma
Xiaoqiao He
CR Hota
Gergely Pollak

I'm going to work on staging the release.


The voting thread is:

 https://s.apache.org/hadoop-3.3.0-Release-vote-thread



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0

[Brahma Reddy Battula]

Hi Xiaoqiao,

Thanks for bringing this to my attention.

It's too rare to occur this scenario and a couple of releases went after
HDFS-4882(broken jira).

So,You think it's fine to include in the next release? Or if you still
insist we can go for another RC...


On Mon, Jul 13, 2020 at 12:29 PM Xiaoqiao He  wrote:

> Thanks Brahma Reddy Battula for your great work here.
> Stephen fixed lease leak in namenode, and it is ready now:
> https://issues.apache.org/jira/browse/HDFS-14498.
> I think this affects 3.3.0-RC0. Would you check this?
> Sorry for reporting it so late.
>
> On Mon, Jul 13, 2020 at 2:53 PM Takanobu Asanuma 
> wrote:
>
>> +1(non-binding)
>>- verified checksums
>>- succeeded in building the package with OpenJDK 8
>>- started HDFS cluster with kerberos with OpenJDK 11
>>- verified Web UIs (NN, DN, Router)
>>- Ran some operations of Router-based Federation with Security
>>- Ran some operations of Erasure Coding
>>
>> Thanks,
>> Takanobu
>>
>> 
>> From: Bilwa S T 
>> Sent: Monday, July 13, 2020 2:23
>> To: Surendra Singh Lilhore; hemanth boyina
>> Cc: Brahma Reddy Battula; mapreduce-dev; Hdfs-dev; Hadoop Common; yarn-dev
>> Subject: RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0
>>
>> +1(non-binding)
>>
>> 1. Deployed 3 node cluster
>> 2. Browsed through Web UI (RM, NM)
>> 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort)
>> 4. Verified basic yarn commands
>>
>> Thanks,
>> Bilwa
>>
>> -Original Message-----
>> From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com]
>> Sent: 12 July 2020 18:32
>> To: hemanth boyina 
>> Cc: Iñigo Goiri ; Vinayakumar B <
>> vinayakum...@apache.org>; Brahma Reddy Battula ;
>> mapreduce-dev ; Hdfs-dev <
>> hdfs-...@hadoop.apache.org>; Hadoop Common ;
>> yarn-dev 
>> Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
>>
>> +1(binding)
>>
>> Deployed HDFS and Yarn Cluster
>> > Verified basic shell commands
>> > Ran some jobs
>> > Verified UI
>>
>> -Surendra
>>
>> On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina <
>> hemanthboyina...@gmail.com>
>> wrote:
>>
>> > +1(non-binding)
>> > Deployed Cluster with Namenodes and Router *)verified shell commands
>> > *)Executed various jobs *)Browsed UI's
>> >
>> >
>> > Thanks,
>> > HemanthBoyina
>> >
>> >
>> > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri,  wrote:
>> >
>> > > +1 (Binding)
>> > >
>> > > Deployed a cluster on Azure VMs with:
>> > > * 3 VMs with HDFS Namenodes and Routers
>> > > * 2 VMs with YARN Resource Managers
>> > > * 5 VMs with HDFS Datanodes and Node Managers
>> > >
>> > > Tests:
>> > > * Executed Tergagen+Terasort+Teravalidate.
>> > > * Executed wordcount.
>> > > * Browsed through the Web UI.
>> > >
>> > >
>> > >
>> > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B
>> > > 
>> > > wrote:
>> > >
>> > > > +1 (Binding)
>> > > >
>> > > > -Verified all checksums and Signatures.
>> > > > -Verified site, Release notes and Change logs
>> > > >   + May be changelog and release notes could be grouped based on
>> > > > the project at second level for better look (this needs to be
>> > > > supported
>> > from
>> > > > yetus)
>> > > > -Tested in x86 local 3-node docker cluster.
>> > > >   + Built from source with OpenJdk 8 and Ubuntu 18.04
>> > > >   + Deployed 3 node docker cluster
>> > > >   + Ran various Jobs (wordcount, Terasort, Pi, etc)
>> > > >
>> > > > No Issues reported.
>> > > >
>> > > > -Vinay
>> > > >
>> > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu 
>> > > wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages
>> > > > >
>> > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04
>> > > > > ARM/aarch64, openjdk-11-jdk
>> > > > >
>> > > > > - checked some web UIs (NN, DN, RM, NM)
>> > > > >
>> > > > > - Executed a wo

[VOTE] Release Apache Hadoop 3.3.0 - RC0

[Brahma Reddy Battula]

Hi folks,

This is the first release candidate for the first release of Apache
Hadoop 3.3.0
line.

It contains *1644[1]* fixed jira issues since 3.2.1 which include a lot of
features and improvements(read the full set of release notes).

Below feature additions are the highlights of the release.

- ARM Support
- Enhancements and new features on S3a,S3Guard,ABFS
- Java 11 Runtime support and TLS 1.3.
- Support Tencent Cloud COS File System.
- Added security to HDFS Router.
- Support non-volatile storage class memory(SCM) in HDFS cache directives
- Support Interactive Docker Shell for running Containers.
- Scheduling of opportunistic containers
- A pluggable device plugin framework to ease vendor plugin development

*The RC0 artifacts are at*: http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/

*First release to include ARM binary, Have a check.*
*RC tag is *release-3.3.0-RC0.


*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1271/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM IST.


I have done a few testing with my pseudo cluster. My +1 to start.



Regards,
Brahma Reddy Battula


1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.3.0) AND
fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER BY
fixVersion ASC

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Yes, All the blockers are closed.will cut RC soon..

On Thu, Jun 18, 2020 at 7:49 PM Adam Antal 
wrote:

> YARN-10314 is also merged. I don't see any blockers at this point.
> (Actually I couldn't see any jiras
> <
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%203.3.0%20ORDER%20BY%20priority%20DESC
> >
> targeted for 3.3.0).
>
> In the community sync yesterday we wanted to discuss the 3.3.0 release, but
> nobody had information about it in the call. Could you share the latest on
> the upcoming 3.3.0 release?
>
> Thanks,
> Adam
>
> On Mon, Jun 15, 2020 at 9:17 AM Ayush Saxena  wrote:
>
> > YARN-10314 also seems to be a blocker.
> >
> > https://issues.apache.org/jira/browse/YARN-10314
> >
> > We should wait for that as well, should get concluded in a day or two.
> >
> > -Ayush
> >
> > > On 15-Jun-2020, at 7:21 AM, Sheng Liu  wrote:
> > >
> > > The  HADOOP-17046 <https://issues.apache.org/jira/browse/HADOOP-17046
> >
> > has
> > > been merged :)
> > >
> > > Brahma Reddy Battula  于2020年6月4日周四 下午10:43写道：
> > >
> > >> Following blocker is pending for 3.3.0 release which is ready for
> > review.
> > >> Mostly we'll have RC soon.
> > >> https://issues.apache.org/jira/browse/HADOOP-17046
> > >>
> > >> Protobuf dependency was unexpected .
> > >>
> > >>> On Mon, Jun 1, 2020 at 7:11 AM Sheng Liu 
> > wrote:
> > >>>
> > >>> Hi folks,
> > >>>
> > >>> It looks like the 3.3.0 branch has been created for quite a while.
> Not
> > >> sure
> > >>> if there is remain block issue that need to be addressed before
> Hadoop
> > >>> 3.3.0 release publishing, maybe we can bring up to here and move the
> > >>> release forward ?
> > >>>
> > >>> Thank.
> > >>>
> > >>> Brahma Reddy Battula  于2020年3月25日周三 上午1:55写道：
> > >>>
> > >>>> thanks to all.
> > >>>>
> > >>>> will make this as optional..will update the wiki accordingly.
> > >>>>
> > >>>> On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B <
> > >> vinayakum...@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>>> Making ARM artifact optional, makes the release process simpler for
> > >> RM
> > >>>> and
> > >>>>> unblocks release process (if there is unavailability of ARM
> > >> resources).
> > >>>>>
> > >>>>> Still there are possible options to collaborate with RM ( as brahma
> > >>>>> mentioned earlier) and provide ARM artifact may be before or after
> > >>> vote.
> > >>>>> If feasible RM can decide to add ARM artifact by collaborating with
> > >>>> @Brahma
> > >>>>> Reddy Battula  or me to get the ARM artifact.
> > >>>>>
> > >>>>> -Vinay
> > >>>>>
> > >>>>> On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
> > >>>>>  wrote:
> > >>>>>
> > >>>>>> Thanks for the clarification Brahma. Can you update the proposal
> to
> > >>>> state
> > >>>>>> that it is optional (it may help to put the proposal on cwiki)?
> > >>>>>>
> > >>>>>> Also if we go ahead then the RM documentation should be clear this
> > >> is
> > >>>> an
> > >>>>>> optional step.
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula <
> > >>>> bra...@apache.org>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Sure, we can't make mandatory while voting and we can upload to
> > >>>>> downloads
> > >>>>>>> once release vote is passed.
> > >>>>>>>
> > >>>>>>> On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> > >>>>>>>  wrote:
> > >>>>>>>
> > >>>>>>>>> Sorry,didn't get you...do you mean, once release v

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Following blocker is pending for 3.3.0 release which is ready for review.
Mostly we'll have RC soon.
https://issues.apache.org/jira/browse/HADOOP-17046

Protobuf dependency was unexpected .

On Mon, Jun 1, 2020 at 7:11 AM Sheng Liu  wrote:

> Hi folks,
>
> It looks like the 3.3.0 branch has been created for quite a while. Not sure
> if there is remain block issue that need to be addressed before Hadoop
> 3.3.0 release publishing, maybe we can bring up to here and move the
> release forward ?
>
> Thank.
>
> Brahma Reddy Battula  于2020年3月25日周三 上午1:55写道：
>
> > thanks to all.
> >
> > will make this as optional..will update the wiki accordingly.
> >
> > On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B 
> > wrote:
> >
> > > Making ARM artifact optional, makes the release process simpler for RM
> > and
> > > unblocks release process (if there is unavailability of ARM resources).
> > >
> > > Still there are possible options to collaborate with RM ( as brahma
> > > mentioned earlier) and provide ARM artifact may be before or after
> vote.
> > > If feasible RM can decide to add ARM artifact by collaborating with
> > @Brahma
> > > Reddy Battula  or me to get the ARM artifact.
> > >
> > > -Vinay
> > >
> > > On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
> > >  wrote:
> > >
> > > > Thanks for the clarification Brahma. Can you update the proposal to
> > state
> > > > that it is optional (it may help to put the proposal on cwiki)?
> > > >
> > > > Also if we go ahead then the RM documentation should be clear this is
> > an
> > > > optional step.
> > > >
> > > >
> > > > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula <
> > bra...@apache.org>
> > > > wrote:
> > > > >
> > > > > Sure, we can't make mandatory while voting and we can upload to
> > > downloads
> > > > > once release vote is passed.
> > > > >
> > > > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> > > > >  wrote:
> > > > >
> > > > >>> Sorry,didn't get you...do you mean, once release voting is
> > > > >>> processed and upload by RM..?
> > > > >>
> > > > >> Yes, that is what I meant. I don’t want us to make more mandatory
> > work
> > > > for
> > > > >> the release manager because the job is hard enough already.
> > > > >>
> > > > >>
> > > > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula <
> > > bra...@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>> Sorry,didn't get you...do you mean, once release voting is
> > processed
> > > > and
> > > > >>> upload by RM..?
> > > > >>>
> > > > >>> FYI. There is docker image for ARM also which support all scripts
> > > > >>> (createrelease, start-build-env.sh, etc ).
> > > > >>>
> > > > >>> https://issues.apache.org/jira/browse/HADOOP-16797
> > > > >>>
> > > > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> > > > >>>  wrote:
> > > > >>>
> > > > >>>> Can ARM binaries be provided after the fact? We cannot increase
> > the
> > > > RM’s
> > > > >>>> burden by asking them to generate an extra set of binaries.
> > > > >>>>
> > > > >>>>
> > > > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> > > > bra...@apache.org>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>> + Dev mailing list.
> > > > >>>>>
> > > > >>>>> -- Forwarded message -
> > > > >>>>> From: Brahma Reddy Battula 
> > > > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
> > > > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > > > >>>>> To: junping_du 
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> thanks junping for your reply.
> > > > >>>>>
> > > > >>>>> bq.  I think mos

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Brahma Reddy Battula]

gt; > > incompatible types: com.google.protobuf.ServiceException cannot be
> > > > > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
> > > > > 17:01:19 2020/04/16 00:01:19 INFO: [INFO] 2 errors
> > > > >
> > > > >
> > > > > There's another error where Ozone uses the Hadoop RPC framework
> which
> > > > uses
> > > > > the hadoop.thirdparty protobuf.
> > > > >
> > > > > [ERROR] Failed to execute goal
> > > > > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > > > > (default-testCompile) on project hadoop-hdds-container-service:
> > > > Compilation
> > > > > failure
> > > > > [ERROR]
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/weichiu/sandbox/ozone/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java:[103,41]
> > > > > incompatible types: com.google.protobuf.BlockingService cannot be
> > > > converted
> > > > > to org.apache.hadoop.thirdparty.protobuf.BlockingService
> > > > >
> > > > > BlockingService scmDatanodeService =
> > > > > StorageContainerDatanodeProtocolService.
> > > > > newReflectiveBlockingService(
> > > > >     new
> > StorageContainerDatanodeProtocolServerSideTranslatorPB(
> > > > > server,
> Mockito.mock(ProtocolMessageMetrics.class)));
> > > > >
> > > > >
> > > > >
> > > > > Ratis probably breaks as well since it depends on the Hadoop RPC
> > > > framework
> > > > > too.
> > > > >
> > > > > On Tue, Apr 28, 2020 at 10:58 PM Vinayakumar B <
> > > vinayakum...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > hi Wei-Chiu,
> > > > > >
> > > > > > Can you elaborate on what failures you are facing related to
> > > relocated
> > > > > > protobuf classes.. ?
> > > > > >
> > > > > > IFAIK, if the issue with location of protobuf classes, still old
> > jar
> > > > > > protobuf-2.5.0.jar will be available in classpath. So downstream
> > > > > depending
> > > > > > on 2.5.0 version of protobuf still be able to access them.
> > > > > >
> > > > > > -vinay
> > > > > >
> > > > > > On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, <
> > weic...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> I'm sorry for coming to this late. I missed this message. It
> > should
> > > > have
> > > > > >> been a DISCUSS thread rather than NOTICE.
> > > > > >>
> > > > > >> Looks like this is inevitable. But we should make the downstream
> > > > > >> developers aware & make the update easier. As long as it is
> stated
> > > > > clearly
> > > > > >> how to update the code to support Hadoop 3.3, I am okay with
> that.
> > > > > >>
> > > > > >> Here's what I suggest:
> > > > > >> (1) label the jira incompatible (just updated the jira) and
> > updated
> > > > the
> > > > > >> release note to tell app developer how to update.
> > > > > >> (2) declare ProtobufHelper a public API HADOOP-17019
> > > > > >> <https://issues.apache.org/jira/browse/HADOOP-17019>
> > > > > >>
> > > > > >> Tez doesn't use the removed Token API, but there's code that
> > breaks
> > > > with
> > > > > >> the relocated protobuf class. The ProtobufHelper API will make
> > this
> > > > > >> transition much easier.
> > > > > >>
> > > > > >> Other downstreamers that break with the relocated protobuf
> > include:
> > > > > Ozone
> > > > > >> and HBase. but neither of them use the removed Token API.
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Jan 8, 2020 at 4:40 AM Vinayakumar B <
> > > vinayakum...@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi All,
> > > > > >>>
> > > > > >>>This mail is to notify about the Removal of following public
> > > APIs
> > > > > from
> > > > > >>> Hadoop Common.
> > > > > >>>
> > > > > >>>  ClassName: org.apache.hadoop.security.token.Token
> > > > > >>>  APIs:
> > > > > >>>  public Token(TokenProto tokenPB);
> > > > > >>>  public TokenProto toTokenProto();
> > > > > >>>
> > > > > >>>Reason: These APIs are having Generated protobuf classes in
> > the
> > > > > >>> signature. Right now due to protobuf upgrade in trunk (soon to
> be
> > > > 3.3.0
> > > > > >>> release) these APIs are breaking the downstream builds, even
> > though
> > > > > >>> downstreams dont use these APIs (just Loading Token class).
> > > > Downstreams
> > > > > >>> are
> > > > > >>> still referencing having older version (2.5.0) of protobuf,
> hence
> > > > build
> > > > > >>> is
> > > > > >>> being broken.
> > > > > >>>
> > > > > >>> These APIs were added for the internal
> purpose(HADOOP-12563),
> > > to
> > > > > >>> support serializing tokens using protobuf in UGI Credentials.
> > > > > >>> Same purpose can be achieved using the Helper classes without
> > > > > introducing
> > > > > >>> protobuf classes in API signatures.
> > > > > >>>
> > > > > >>> Token.java is marked as Evolving, so I believe APIs can be
> > changed
> > > > > >>> whenever
> > > > > >>> absolute necessary.
> > > > > >>>
> > > > > >>> Jira https://issues.apache.org/jira/browse/HADOOP-16621
> has
> > > been
> > > > > >>> reported to solve downstream build failure.
> > > > > >>>
> > > > > >>> So since this API was added for internal purpose easy approach
> to
> > > > solve
> > > > > >>> this is to remove APIs and use helper classes. Otherwise, as
> > > > mentioned
> > > > > in
> > > > > >>> HADOOP-16621, workaround will add unnecessary codes to be
> > > maintained.
> > > > > >>>
> > > > > >>> If anyone using these APIs outside hadoop project accidentally,
> > > > please
> > > > > >>> reply to this mail immediately.
> > > > > >>>
> > > > > >>> If no objection by next week, will go ahead with removal of
> above
> > > > said
> > > > > >>> APIs
> > > > > >>> in HADOOP-16621.
> > > > > >>>
> > > > > >>> -Vinay
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] making Ozone a separate Apache project

[Brahma Reddy Battula]

+1


On Sat, 16 May 2020 at 12:09 AM, Subru Krishnan  wrote:

> +1.
>
> Thanks,
> Subru
>
> On Thu, May 14, 2020 at 10:00 PM Akira Ajisaka 
> wrote:
>
> > +1
> >
> > -Akira
> >
> > On Wed, May 13, 2020 at 4:53 PM Elek, Marton  wrote:
> >
> > >
> > >
> > > I would like to start a discussion to make a separate Apache project
> for
> > > Ozone
> > >
> > >
> > >
> > > ### HISTORY [1]
> > >
> > >   * Apache Hadoop Ozone development started on a feature branch of
> > > Hadoop repository (HDFS-7240)
> > >
> > >   * In the October of 2017 a discussion has been started to merge it to
> > > the Hadoop main branch
> > >
> > >   * After a long discussion it's merged to Hadoop trunk at the March of
> > > 2018
> > >
> > >   * During the discussion of the merge, it was suggested multiple times
> > > to create a separated project for the Ozone. But at that time:
> > >  1). Ozone was tightly integrated with Hadoop/HDFS
> > >  2). There was an active plan to use Block layer of Ozone (HDDS or
> > > HDSL at that time) as the block level of HDFS
> > >  3). The community of Ozone was a subset of the HDFS community
> > >
> > >   * The first beta release of Ozone was just released. Seems to be a
> > > good time before the first GA to make a decision about the future.
> > >
> > >
> > >
> > > ### WHAT HAS BEEN CHANGED
> > >
> > >   During the last years Ozone became more and more independent both at
> > > the community and code side. The separation has been suggested again
> and
> > > again (for example by Owen [2] and Vinod [3])
> > >
> > >
> > >
> > >   From COMMUNITY point of view:
> > >
> > >
> > >* Fortunately more and more new contributors are helping Ozone.
> > > Originally the Ozone community was a subset of HDFS project. But now a
> > > bigger and bigger part of the community is related to Ozone only.
> > >
> > >* It seems to be easier to _build_ the community as a separated
> > project.
> > >
> > >* A new, younger project might have different practices
> > > (communication, commiter criteria, development style) compared to old,
> > > mature project
> > >
> > >* It's easier to communicate (and improve) these standards in a
> > > separated projects with clean boundaries
> > >
> > >* Separated project/brand can help to increase the adoption rate and
> > > attract more individual contributor (AFAIK it has been seen in
> Submarine
> > > after a similar move)
> > >
> > >   * Contribution process can be communicated more easily, we can make
> > > first time contribution more easy
> > >
> > >
> > >
> > >   From CODE point of view Ozone became more and more independent:
> > >
> > >
> > >   * Ozone has different release cycle
> > >
> > >   * Code is already separated from Hadoop code base
> > > (apache/hadoop-ozone.git)
> > >
> > >   * It has separated CI (github actions)
> > >
> > >   * Ozone uses different (more strict) coding style (zero toleration of
> > > unit test / checkstyle errors)
> > >
> > >   * The code itself became more and more independent from Hadoop on
> > > Maven level. Originally it was compiled together with the in-tree
> latest
> > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > > Configuration...)
> > >
> > >   * It starts to use multiple version of Hadoop (on client side)
> > >
> > >   * Volume of resolved issues are already very high on Ozone side
> (Ozone
> > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > > together in the last 2-3 months)
> > >
> > >
> > > Summary: Before the first Ozone GA release, It seems to be a good time
> > > to discuss the long-term future of Ozone. Managing it as a separated
> TLP
> > > project seems to have more benefits.
> > >
> > >
> > > Please let me know what your opinion is...
> > >
> > > Thanks a lot,
> > > Marton
> > >
> > >
> > >
> > >
> > >
> > > [1]: For more details, see:
> > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> > >
> > > [2]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> > >
> > > [3]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>
-- 



--Brahma Reddy Battula

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Thanks Akira and Surendra for pointing that out.

Looks I missed after creating the branch..



On Thu, 30 Apr 2020 at 10:16 AM, Akira Ajisaka  wrote:

> Hi Surendra,
>
> Updated the version to 3.3.1-SNAPSHOT in branch-3.3.
>
> Thanks,
> Akira
>
> On Wed, Apr 29, 2020 at 4:22 PM Surendra Singh Lilhore <
> surendralilh...@gmail.com> wrote:
>
>> Hi Brahma,
>>
>> Why the branch-3.3 & branch-3.3.0 pom version is same ?.
>>
>> In branch-3.3 pom version should be 3.3.1.
>>
>> Please correct me if I am wrong.
>>
>> -Surendra
>>
>>
>> On Sat, 25 Apr, 2020, 9:33 am Mingliang Liu,  wrote:
>>
>> > Brahma,
>> >
>> > What about https://issues.apache.org/jira/browse/HADOOP-17007?
>> >
>> > Thanks,
>> >
>> > On Fri, Apr 24, 2020 at 11:07 AM Brahma Reddy Battula <
>> bra...@apache.org>
>> > wrote:
>> >
>> > > Ok. Done. Branch created.
>> > >
>> > > Following blockers are pending, will closely track this.
>> > >
>> > > https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under
>> > discussion
>> > > )
>> > > https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available)
>> > > https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available)
>> > > https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available)
>> > >
>> > >
>> > > On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang
>> > >  wrote:
>> > >
>> > > > +1 we should have the branch ASAP.
>> > > >
>> > > > On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka > >
>> > > > wrote:
>> > > >
>> > > > > > Since blockers are not closed, I didn't cut the branch because
>> > > > > multiple branches might confuse or sombody might miss to commit.
>> > > > >
>> > > > > The current situation is already confusing. The 3.3.1 version
>> already
>> > > > > exists in JIRA, so some committers wrongly commit non-critical
>> issues
>> > > to
>> > > > > branch-3.3 and set the fix version to 3.3.1.
>> > > > > I think now we should cut branch-3.3.0 and freeze source code
>> except
>> > > the
>> > > > > blockers.
>> > > > >
>> > > > > -Akira
>> > > > >
>> > > > > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula <
>> > > bra...@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > >> Sure, I will do that.
>> > > > >>
>> > > > >> Since blockers are not closed, I didn't cut the branch because
>> > > > >> multiple branches might confuse or sombody might miss to
>> > commit.Shall
>> > > I
>> > > > >> wait till this weekend to create..?
>> > > > >>
>> > > > >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka <
>> aajis...@apache.org
>> > >
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Hi Brahma,
>> > > > >>>
>> > > > >>> Thank you for preparing the release.
>> > > > >>> Could you cut branch-3.3.0? I would like to backport some fixes
>> for
>> > > > >>> 3.3.1 and not for 3.3.0.
>> > > > >>>
>> > > > >>> Thanks and regards,
>> > > > >>> Akira
>> > > > >>>
>> > > > >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula <
>> > > > bra...@apache.org>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Hi All,
>> > > > >>>>
>> > > > >>>> we are down to two blockers issues now (YARN-10194 and
>> YARN-9848)
>> > > > which
>> > > > >>>> are in patch available state.Hopefully we can out the RC soon.
>> > > > >>>>
>> > > > >>>> thanks to @Prabhu Joseph 
>> > > > ,@masakate,@akira
>> > > > >>>> and @Wei-Chiu Chuang   and others for
>> > helping
>> > > > >>>> resloving the blockers.
>> > > > >>>>
>> >

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Ok. Done. Branch created.

Following blockers are pending, will closely track this.

https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under discussion )
https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available)
https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available)
https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available)


On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang
 wrote:

> +1 we should have the branch ASAP.
>
> On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka 
> wrote:
>
> > > Since blockers are not closed, I didn't cut the branch because
> > multiple branches might confuse or sombody might miss to commit.
> >
> > The current situation is already confusing. The 3.3.1 version already
> > exists in JIRA, so some committers wrongly commit non-critical issues to
> > branch-3.3 and set the fix version to 3.3.1.
> > I think now we should cut branch-3.3.0 and freeze source code except the
> > blockers.
> >
> > -Akira
> >
> > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula 
> > wrote:
> >
> >> Sure, I will do that.
> >>
> >> Since blockers are not closed, I didn't cut the branch because
> >> multiple branches might confuse or sombody might miss to commit.Shall I
> >> wait till this weekend to create..?
> >>
> >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka 
> >> wrote:
> >>
> >>> Hi Brahma,
> >>>
> >>> Thank you for preparing the release.
> >>> Could you cut branch-3.3.0? I would like to backport some fixes for
> >>> 3.3.1 and not for 3.3.0.
> >>>
> >>> Thanks and regards,
> >>> Akira
> >>>
> >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula <
> bra...@apache.org>
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> we are down to two blockers issues now (YARN-10194 and YARN-9848)
> which
> >>>> are in patch available state.Hopefully we can out the RC soon.
> >>>>
> >>>> thanks to @Prabhu Joseph 
> ,@masakate,@akira
> >>>> and @Wei-Chiu Chuang   and others for helping
> >>>> resloving the blockers.
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula <
> >>>> bra...@apache.org> wrote:
> >>>>
> >>>>>
> >>>>> @Prabhu Joseph 
> >>>>> >>> Have committed the YARN blocker YARN-10219 to trunk and
> >>>>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras -
> >>>>> YARN-10233 and HADOOP-16982
> >>>>> which i will help to review and commit. Thanks.
> >>>>>
> >>>>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as
> a
> >>>>> blockers. (without YARN-10233 we have given so many releases,it's
> not newly
> >>>>> introduced.).. Thanks
> >>>>>
> >>>>> @Vinod Kumar Vavilapalli  ,@adam Antal,
> >>>>>
> >>>>> I noted YARN-9848 as a blocker as you mentioned above.
> >>>>>
> >>>>> @All,
> >>>>>
> >>>>> Currently following four blockers are pending for 3.3.0 RC.
> >>>>>
> >>>>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
> >>>>> vino...@apache.org> wrote:
> >>>>>
> >>>>>> Looks like a really bad bug to me.
> >>>>>>
> >>>>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should
> >>>>>> also revert it in a 3.2 maintenance release too.
> >>>>>>
> >>>>>> Thanks
> >>>>>> +Vinod
> >>>>>>
> >>>>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal  .INVALID>
> >>>>>> wrote:
> >>>>>> >
> >>>>>> > Hi everyone,
> >>>>>> >
> >>>>>> > Sorry for coming a bit late with this, but there's also one jira
> >>>>>> that can
> >>>>>> > have potential impact on clusters and we should talk about it.
> >>>>>> >
> >>>&g

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Sure, I will do that.

Since blockers are not closed, I didn't cut the branch because
multiple branches might confuse or sombody might miss to commit.Shall I
wait till this weekend to create..?

On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka  wrote:

> Hi Brahma,
>
> Thank you for preparing the release.
> Could you cut branch-3.3.0? I would like to backport some fixes for 3.3.1
> and not for 3.3.0.
>
> Thanks and regards,
> Akira
>
> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula 
> wrote:
>
>> Hi All,
>>
>> we are down to two blockers issues now (YARN-10194 and YARN-9848) which
>> are in patch available state.Hopefully we can out the RC soon.
>>
>> thanks to @Prabhu Joseph  ,@masakate,@akira
>> and @Wei-Chiu Chuang   and others for helping
>> resloving the blockers.
>>
>>
>>
>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula 
>> wrote:
>>
>>>
>>> @Prabhu Joseph 
>>> >>> Have committed the YARN blocker YARN-10219 to trunk and
>>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras -
>>> YARN-10233 and HADOOP-16982
>>> which i will help to review and commit. Thanks.
>>>
>>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
>>> blockers. (without YARN-10233 we have given so many releases,it's not newly
>>> introduced.).. Thanks
>>>
>>> @Vinod Kumar Vavilapalli  ,@adam Antal,
>>>
>>> I noted YARN-9848 as a blocker as you mentioned above.
>>>
>>> @All,
>>>
>>> Currently following four blockers are pending for 3.3.0 RC.
>>>
>>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
>>>
>>>
>>>
>>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
>>> vino...@apache.org> wrote:
>>>
>>>> Looks like a really bad bug to me.
>>>>
>>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should
>>>> also revert it in a 3.2 maintenance release too.
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
>>>> wrote:
>>>> >
>>>> > Hi everyone,
>>>> >
>>>> > Sorry for coming a bit late with this, but there's also one jira that
>>>> can
>>>> > have potential impact on clusters and we should talk about it.
>>>> >
>>>> > Steven Rand found this problem earlier and commented to
>>>> > https://issues.apache.org/jira/browse/YARN-4946.
>>>> > The bug has impact on the RM state store: the RM does not delete apps
>>>> - see
>>>> > more details in his comment here:
>>>> >
>>>> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
>>>> > .
>>>> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
>>>> with
>>>> > the revert task).
>>>> >
>>>> > It might not be an actual blocker, but since there wasn't any
>>>> consensus
>>>> > about a follow up action, I thought we should decide how to proceed
>>>> before
>>>> > release 3.3.0.
>>>> >
>>>> > Regards,
>>>> > Adam
>>>> >
>>>> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
>>>> prabhujose.ga...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Thanks Brahma for the update.
>>>> >>
>>>> >> Have committed the YARN blocker YARN-10219 to trunk and
>>>> cherry-picked to
>>>> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
>>>> >> HADOOP-16982
>>>> >> which i will help to review and commit. Thanks.
>>>> >>
>>>> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
>>>> >>
>>>> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>>> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
>>>> ORDER
>>>> >> BY priority DESC
>>>> >>
>>>> >>
>>>> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
>>>> bra...@apache.org>
>>>> >> wrote:
>>>>

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Hi All,

we are down to two blockers issues now (YARN-10194 and YARN-9848) which are
in patch available state.Hopefully we can out the RC soon.

thanks to @Prabhu Joseph 
,@masakate,@akira and @Wei-Chiu
Chuang   and others for helping resloving the
blockers.



On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula 
wrote:

>
> @Prabhu Joseph 
> >>> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked
> to branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
> HADOOP-16982
> which i will help to review and commit. Thanks.
>
> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
> blockers. (without YARN-10233 we have given so many releases,it's not newly
> introduced.).. Thanks
>
> @Vinod Kumar Vavilapalli  ,@adam Antal,
>
> I noted YARN-9848 as a blocker as you mentioned above.
>
> @All,
>
> Currently following four blockers are pending for 3.3.0 RC.
>
> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
>
>
>
> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> Looks like a really bad bug to me.
>>
>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should also
>> revert it in a 3.2 maintenance release too.
>>
>> Thanks
>> +Vinod
>>
>> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > Sorry for coming a bit late with this, but there's also one jira that
>> can
>> > have potential impact on clusters and we should talk about it.
>> >
>> > Steven Rand found this problem earlier and commented to
>> > https://issues.apache.org/jira/browse/YARN-4946.
>> > The bug has impact on the RM state store: the RM does not delete apps -
>> see
>> > more details in his comment here:
>> >
>> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
>> > .
>> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
>> with
>> > the revert task).
>> >
>> > It might not be an actual blocker, but since there wasn't any consensus
>> > about a follow up action, I thought we should decide how to proceed
>> before
>> > release 3.3.0.
>> >
>> > Regards,
>> > Adam
>> >
>> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
>> prabhujose.ga...@gmail.com>
>> > wrote:
>> >
>> >> Thanks Brahma for the update.
>> >>
>> >> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked
>> to
>> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
>> >> HADOOP-16982
>> >> which i will help to review and commit. Thanks.
>> >>
>> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
>> >>
>> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
>> ORDER
>> >> BY priority DESC
>> >>
>> >>
>> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
>> bra...@apache.org>
>> >> wrote:
>> >>
>> >>> *Pending for 3.3.0 Release:*
>> >>>
>> >>> One Blocker(HADOOP-16963) confirmation and following jira's are open
>> as
>> >>> these needs to merged to other branches(I am tracking the same,
>> Ideally
>> >>> this can be closed and can raise seperate jira's to track).
>> >>>
>> >>>
>> >>> 1–4 of 4Refresh results
>> >>> <
>> >>>
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
>> >>>>
>> >>> Columns
>> >>> Patch InfoKeyTSummaryAssigneeReporterP
>> StatusResolutionUpdatedDueCreated
>> >>> <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353
>> >>> <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task]
>> >>> <https://issues.apache.org/jira/browse/HDFS-14353>
>> >>>
>> >>> HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure
>> >>> Coding:
>> >>> metrics xmitsInProgress become to neg

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

@Prabhu Joseph 
>>> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked
to branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
HADOOP-16982
which i will help to review and commit. Thanks.

Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
blockers. (without YARN-10233 we have given so many releases,it's not newly
introduced.).. Thanks

@Vinod Kumar Vavilapalli  ,@adam Antal,

I noted YARN-9848 as a blocker as you mentioned above.

@All,

Currently following four blockers are pending for 3.3.0 RC.

HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.



On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli 
wrote:

> Looks like a really bad bug to me.
>
> +1 for revert and +1 for making that a 3.3.0 blocker. I think should also
> revert it in a 3.2 maintenance release too.
>
> Thanks
> +Vinod
>
> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
> wrote:
> >
> > Hi everyone,
> >
> > Sorry for coming a bit late with this, but there's also one jira that can
> > have potential impact on clusters and we should talk about it.
> >
> > Steven Rand found this problem earlier and commented to
> > https://issues.apache.org/jira/browse/YARN-4946.
> > The bug has impact on the RM state store: the RM does not delete apps -
> see
> > more details in his comment here:
> >
> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
> > .
> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
> with
> > the revert task).
> >
> > It might not be an actual blocker, but since there wasn't any consensus
> > about a follow up action, I thought we should decide how to proceed
> before
> > release 3.3.0.
> >
> > Regards,
> > Adam
> >
> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
> prabhujose.ga...@gmail.com>
> > wrote:
> >
> >> Thanks Brahma for the update.
> >>
> >> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked to
> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
> >> HADOOP-16982
> >> which i will help to review and commit. Thanks.
> >>
> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
> >>
> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
> ORDER
> >> BY priority DESC
> >>
> >>
> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
> bra...@apache.org>
> >> wrote:
> >>
> >>> *Pending for 3.3.0 Release:*
> >>>
> >>> One Blocker(HADOOP-16963) confirmation and following jira's are open as
> >>> these needs to merged to other branches(I am tracking the same, Ideally
> >>> this can be closed and can raise seperate jira's to track).
> >>>
> >>>
> >>> 1–4 of 4Refresh results
> >>> <
> >>>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
> >>>>
> >>> Columns
> >>> Patch InfoKeyTSummaryAssigneeReporterP
> StatusResolutionUpdatedDueCreated
> >>> <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353
> >>> <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task]
> >>> <https://issues.apache.org/jira/browse/HDFS-14353>
> >>>
> >>> HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure
> >>> Coding:
> >>> metrics xmitsInProgress become to negative.
> >>> <https://issues.apache.org/jira/browse/HDFS-14353>
> >>> maobaolong
> >>> <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> >>> maobaolong
> >>> <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> >>> [image:
> >>> Major] REOPENED *Unresolved* 11/Apr/20   11/Mar/19 Actions
> >>> <
> >>>
> https://issues.apache.org/jira/rest/api/1.0/issues/13220750/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> >>>>
> >>> <https://issues.apache.org/jira/browse/HDFS-14788> HDFS-14788
> >>> <https://issues.apache.org/jira/browse/HDFS-14

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Thanks Hexiaoqiao for taking care of this jira.Hope other's can do the
needful.

On Sun, Apr 12, 2020 at 11:25 AM Xiaoqiao He  wrote:

> Hi Brahma,
>
> Thanks for your great works here.
> About HDFS-12862 <https://issues.apache.org/jira/browse/HDFS-12862>, I try
> to raise another JIRA (HDFS-15272) to track backport branch-3.1, close this
> one to decouple release progress.
>
> Thanks,
> Hexiaoqiao
>
> On Sun, Apr 12, 2020 at 2:49 AM Brahma Reddy Battula 
> wrote:
>
> > *Pending for 3.3.0 Release:*
> >
> > One Blocker(HADOOP-16963) confirmation and following jira's are open as
> > these needs to merged to other branches(I am tracking the same, Ideally
> > this can be closed and can raise seperate jira's to track).
> >
> >
> > 1–4 of 4Refresh results
> > <
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
> > >
> > Columns
> > Patch InfoKeyTSummaryAssigneeReporterP StatusResolutionUpdatedDueCreated
> > <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353
> > <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task]
> > <https://issues.apache.org/jira/browse/HDFS-14353>
> >
> > HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure
> > Coding:
> > metrics xmitsInProgress become to negative.
> > <https://issues.apache.org/jira/browse/HDFS-14353>
> > maobaolong
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> > maobaolong
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> > [image:
> > Major] REOPENED *Unresolved* 11/Apr/20   11/Mar/19 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13220750/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> > <https://issues.apache.org/jira/browse/HDFS-14788> HDFS-14788
> > <https://issues.apache.org/jira/browse/HDFS-14788> [image: Improvement]
> > <https://issues.apache.org/jira/browse/HDFS-14788>
> >
> > Use dynamic regex filter to ignore copy of source files in Distcp
> > <https://issues.apache.org/jira/browse/HDFS-14788>
> > Mukund Thakur
> > <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mukund-thakur
> > >
> > Mukund
> > Thakur
> > <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mukund-thakur
> > >
> > [image:
> > Major] OPEN *Unresolved* 11/Apr/20   28/Aug/19 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13253437/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> > <https://issues.apache.org/jira/browse/HDFS-12862> HDFS-12862
> > <https://issues.apache.org/jira/browse/HDFS-12862> [image: Bug]
> > <https://issues.apache.org/jira/browse/HDFS-12862>
> >
> > CacheDirective becomes invalid when NN restart or failover
> > <https://issues.apache.org/jira/browse/HDFS-12862>
> > Wang XL
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Wang%20XL>
> > Wang
> > XL <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Wang%20XL>
> > [image:
> > Major] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/17 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13121225/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> > <https://issues.apache.org/jira/browse/HDFS-14476> HDFS-14476
> > <https://issues.apache.org/jira/browse/HDFS-14476> [image: Improvement]
> > <https://issues.apache.org/jira/browse/HDFS-14476>
> >
> > lock too long when fix inconsistent blocks between disk and in-memory
> > <https://issues.apache.org/jira/browse/HDFS-14476>
> > Sean Chow
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=seanlook>
> > Sean
> > Chow <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=seanlook
> > >
> > [image:
> > Major] PATCH AVAILABLE *Unresolved* 12/Apr/20   07/May/19 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13231926/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> >
> >
> > 1–5 of 5Refresh results
&

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

ache.org/jira/secure/ViewProfile.jspa?name=gandras> Szilard
Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
[image:
Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
<https://issues.apache.org/jira/rest/api/1.0/issues/13271201/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin>
<https://issues.apache.org/jira/browse/YARN-10002> YARN-10002
<https://issues.apache.org/jira/browse/YARN-10002> [image: Improvement]
<https://issues.apache.org/jira/browse/YARN-10002>

Code cleanup and improvements in ConfigurationStoreBaseTest
<https://issues.apache.org/jira/browse/YARN-10002>
Benjamin Teke
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke> Szilard
Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
[image:
Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
<https://issues.apache.org/jira/rest/api/1.0/issues/13271215/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin>
<https://issues.apache.org/jira/browse/YARN-9998> YARN-9998
<https://issues.apache.org/jira/browse/YARN-9998> [image: Improvement]
<https://issues.apache.org/jira/browse/YARN-9998>

Code cleanup in LeveldbConfigurationStore
<https://issues.apache.org/jira/browse/YARN-9998>
Benjamin Teke
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke> Szilard
Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
[image:
Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
<https://issues.apache.org/jira/rest/api/1.0/issues/13271202/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin>
<https://issues.apache.org/jira/browse/YARN-9354> YARN-9354
<https://issues.apache.org/jira/browse/YARN-9354> [image: Improvement]
<https://issues.apache.org/jira/browse/YARN-9354>

Resources sh <https://issues.apache.org/jira/browse/YARN-9354>

On Thu, Apr 9, 2020 at 11:41 PM Brahma Reddy Battula 
wrote:

> thanks a lot Akira and Prabhu for ping.
>
> All the blockers/critical issues are fixed. Recently two are reported
> again like HADOOP-16955 and HADOOP-16963 waiting for confirmation.
>
> @prabhu, Those 156 jira's are non-blocker issues which needs to moved to
> 3.4.0. and filter for blocker/critical is following.
>
> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
> BY priority DESC
>
>
> On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph 
> wrote:
>
>> Hi Brahma,
>>
>>  Please let me know if i can be of any help in review and commit /
>> clear the pending issues in YARN and MapReduce targeted for 3.3.0.
>> I see there are 156 YARN + MapReduce Unresolved Jiras which are either
>> with
>> target or fix version set to 3.3.0.
>>
>> project in (YARN, MAPREDUCE) AND resolution = Unresolved AND (cf[12310320]
>> = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC
>>
>> Thanks,
>> Prabhu Joseph
>>
>> On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka  wrote:
>>
>> > Hi Brahma,
>> >
>> > How is this issue going?
>> > If there are some blockers, I can help you.
>> >
>> > -Akira
>> >
>> > On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula > >
>> > wrote:
>> >
>> > > Hi All,
>> > >
>> > > branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.
>> > >
>> > > while committing jira's please set the proper fix version and if
>> there is
>> > > any blocker/critical please commit to branch-3.3 or let me know.
>> > >
>> > > Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
>> > > communication in another thread[2]).
>> > >
>> > >
>> > > 1.https://github.com/apache/hadoop/commits/branch-3.3
>> > > 2.
>> https://lists.apache.org/list.html?common-...@hadoop.apache.org:2020-3
>> > >
>> > >
>> > >
>> > >
>> > > --Brahma Reddy Battula
>> > >
>> >
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

thanks a lot Akira and Prabhu for ping.

All the blockers/critical issues are fixed. Recently two are reported again
like HADOOP-16955 and HADOOP-16963 waiting for confirmation.

@prabhu, Those 156 jira's are non-blocker issues which needs to moved to
3.4.0. and filter for blocker/critical is following.

project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
BY priority DESC


On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph 
wrote:

> Hi Brahma,
>
>  Please let me know if i can be of any help in review and commit /
> clear the pending issues in YARN and MapReduce targeted for 3.3.0.
> I see there are 156 YARN + MapReduce Unresolved Jiras which are either with
> target or fix version set to 3.3.0.
>
> project in (YARN, MAPREDUCE) AND resolution = Unresolved AND (cf[12310320]
> = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC
>
> Thanks,
> Prabhu Joseph
>
> On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka  wrote:
>
> > Hi Brahma,
> >
> > How is this issue going?
> > If there are some blockers, I can help you.
> >
> > -Akira
> >
> > On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula 
> > wrote:
> >
> > > Hi All,
> > >
> > > branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.
> > >
> > > while committing jira's please set the proper fix version and if there
> is
> > > any blocker/critical please commit to branch-3.3 or let me know.
> > >
> > > Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
> > > communication in another thread[2]).
> > >
> > >
> > > 1.https://github.com/apache/hadoop/commits/branch-3.3
> > > 2.
> https://lists.apache.org/list.html?common-...@hadoop.apache.org:2020-3
> > >
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>


-- 



--Brahma Reddy Battula

Re: JENKINS is not posting build results in jiras

[Brahma Reddy Battula]

Thanks Akira!!

On Wed, Mar 25, 2020 at 8:22 AM Sunil Govindan  wrote:

> Thanks Akira for helping.
>
> Best Regards,
> Sunil
>
> On Wed, Mar 25, 2020 at 6:41 AM Akira Ajisaka  wrote:
>
> > Sorry it was the root cause. Reverted.
> > Now the precommit job can add comments to GitHub.
> > https://github.com/apache/hadoop/pull/1820#issuecomment-603545202
> >
> > Akira
> >
> > On Tue, Mar 24, 2020 at 8:53 PM Akira Ajisaka 
> wrote:
> >
> > > A few days ago I upgraded the build image and maybe it is the root
> cause.
> > > Created a pull request to revert the change to see if it is really the
> > > cause: https://github.com/apache/hadoop/pull/1911
> > >
> > > Thanks,
> > > Akira
> > >
> > > On Tue, Mar 24, 2020 at 8:21 PM Akira Ajisaka 
> > wrote:
> > >
> > >> Thanks Sunil for the report.
> > >>
> > >> The precommit jobs are failing to add comments to GitHub also.
> > >> I'm looking into the cause, and any help is appreciated.
> > >>
> > >> Thanks,
> > >> Akira
> > >>
> > >>
> > >> On Mon, Mar 23, 2020 at 7:39 PM Sunil Govindan 
> > wrote:
> > >>
> > >>> Hello
> > >>>
> > >>> From last few days, jenkins results are not getting published in
> jiras.
> > >>>
> > >>> For eg,
> > https://builds.apache.org/job/PreCommit-YARN-Build/25735/console
> > >>> has
> > >>> results. but I cannot see the same in YARN-9879. Similarly in the
> cases
> > >>> of YARN-10198 etc.
> > >>>
> > >>> I could see a message in console o/p as "Adding comment to JIRA". But
> > its
> > >>> not happening.
> > >>>
> > >>> Any ideas?
> > >>>
> > >>> Thanks
> > >>> Sunil
> > >>>
> > >>
> >
>


-- 



--Brahma Reddy Battula

[Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Hi All,

branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.

while committing jira's please set the proper fix version and if there is
any blocker/critical please commit to branch-3.3 or let me know.

Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
communication in another thread[2]).


1.https://github.com/apache/hadoop/commits/branch-3.3
2.https://lists.apache.org/list.html?common-...@hadoop.apache.org:2020-3




--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

*Important update: *branch-3.3 has created. and now trunk is to target to
3.4.0 ( As per above plan). ( will send in another mail with subject and
post in slack group also.)

Please merge any critical or blocker issues to branch-3.3 also.

Thanks.

On Tue, Mar 24, 2020 at 11:34 PM Brahma Reddy Battula 
wrote:

>
> Critical and blockers issues came down[1].
>
> Planning the cut branch-3.3 this weekend and start release vote on next
> weekend.
>
>
>
> 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
> BY priority DESC
>
>
>
> On Thu, Feb 27, 2020 at 12:42 AM Brahma Reddy Battula 
> wrote:
>
>> Hi All,
>>
>> Inline with the original 3.3.0 communication proposal dated 8th Jan 2020,
>> I would like to provide more updates[1].
>>
>> We are approaching previously proposed code freeze date (March 10,2020).
>> So I would like to cut 3.3 branch on *10th March* and point existing *trunk
>> to 3.4* if there are no issues.
>>
>> *Current Release Plan:*
>>
>> *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's
>> almost done)
>> *Code freeze Date *: blockers/critical only, no improvements and
>> blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues
>> are there which are very old and I am tracking same.)
>> *Release Date*: March 15, 2020.
>>
>> *Please let me know if I missed anything.*
>>
>> 1.
>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
>> 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
>> BY priority DESC
>>
>>
>> On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula <
>> brbapa...@gmail.com> wrote:
>>
>>>
>>> Wiki was updated for 3.3
>>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
>>> .
>>>
>>>
>>> >I'll move out anything that isn't needed.
>>>
>>> thanks steve.
>>>
>>> > We need to fix the shaded protobuf in
>>> > Token issue to even get spark to compile.
>>>
>>> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621
>>>
>>> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran
>>>  wrote:
>>>
>>>> >
>>>> > 2. Features close to finish:
>>>> >
>>>> >
>>>> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
>>>> owner
>>>> > :  Steve Loughran)
>>>> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
>>>> > fixes. ( owner :  Steve Loughran)
>>>> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features.
>>>> (
>>>> > owner :  Steve Loughran)
>>>> >
>>>> > I'll move out anything that isn't needed.
>>>>
>>>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing
>>>> and
>>>> I've got some provisional tuning to do. That is -if things didn't work
>>>> in
>>>> the test/production deployments, I'd know about the regressions (e.g.
>>>> HADOOP-16751).
>>>>
>>>> This is S3A and ABFS code -no idea about the rest, and inevitably the
>>>> big
>>>> JAR changes will have surprises. We need to fix the shaded protobuf in
>>>> Token issue to even get spark to compile.
>>>>
>>>> -Steve
>>>>
>>>> >
>>>> >
>>>>
>>>
>>>
>>> --
>>> --Brahma Reddy Battula
>>>
>>
>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Critical and blockers issues came down[1].

Planning the cut branch-3.3 this weekend and start release vote on next
weekend.



1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
BY priority DESC



On Thu, Feb 27, 2020 at 12:42 AM Brahma Reddy Battula 
wrote:

> Hi All,
>
> Inline with the original 3.3.0 communication proposal dated 8th Jan 2020,
> I would like to provide more updates[1].
>
> We are approaching previously proposed code freeze date (March 10,2020).
> So I would like to cut 3.3 branch on *10th March* and point existing *trunk
> to 3.4* if there are no issues.
>
> *Current Release Plan:*
>
> *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's
> almost done)
> *Code freeze Date *: blockers/critical only, no improvements and
> blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues
> are there which are very old and I am tracking same.)
> *Release Date*: March 15, 2020.
>
> *Please let me know if I missed anything.*
>
> 1.
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
> 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
> BY priority DESC
>
>
> On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula 
> wrote:
>
>>
>> Wiki was updated for 3.3
>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0.
>>
>>
>> >I'll move out anything that isn't needed.
>>
>> thanks steve.
>>
>> > We need to fix the shaded protobuf in
>> > Token issue to even get spark to compile.
>>
>> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621
>>
>> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran 
>> wrote:
>>
>>> >
>>> > 2. Features close to finish:
>>> >
>>> >
>>> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
>>> owner
>>> > :  Steve Loughran)
>>> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
>>> > fixes. ( owner :  Steve Loughran)
>>> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
>>> > owner :  Steve Loughran)
>>> >
>>> > I'll move out anything that isn't needed.
>>>
>>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing
>>> and
>>> I've got some provisional tuning to do. That is -if things didn't work in
>>> the test/production deployments, I'd know about the regressions (e.g.
>>> HADOOP-16751).
>>>
>>> This is S3A and ABFS code -no idea about the rest, and inevitably the big
>>> JAR changes will have surprises. We need to fix the shaded protobuf in
>>> Token issue to even get spark to compile.
>>>
>>> -Steve
>>>
>>> >
>>> >
>>>
>>
>>
>> --
>> --Brahma Reddy Battula
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

thanks to all.

will make this as optional..will update the wiki accordingly.

On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B 
wrote:

> Making ARM artifact optional, makes the release process simpler for RM  and
> unblocks release process (if there is unavailability of ARM resources).
>
> Still there are possible options to collaborate with RM ( as brahma
> mentioned earlier) and provide ARM artifact may be before or after vote.
> If feasible RM can decide to add ARM artifact by collaborating with @Brahma
> Reddy Battula  or me to get the ARM artifact.
>
> -Vinay
>
> On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
>  wrote:
>
> > Thanks for the clarification Brahma. Can you update the proposal to state
> > that it is optional (it may help to put the proposal on cwiki)?
> >
> > Also if we go ahead then the RM documentation should be clear this is an
> > optional step.
> >
> >
> > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula 
> > wrote:
> > >
> > > Sure, we can't make mandatory while voting and we can upload to
> downloads
> > > once release vote is passed.
> > >
> > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> > >  wrote:
> > >
> > >>> Sorry,didn't get you...do you mean, once release voting is
> > >>> processed and upload by RM..?
> > >>
> > >> Yes, that is what I meant. I don’t want us to make more mandatory work
> > for
> > >> the release manager because the job is hard enough already.
> > >>
> > >>
> > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula <
> bra...@apache.org>
> > >> wrote:
> > >>>
> > >>> Sorry,didn't get you...do you mean, once release voting is processed
> > and
> > >>> upload by RM..?
> > >>>
> > >>> FYI. There is docker image for ARM also which support all scripts
> > >>> (createrelease, start-build-env.sh, etc ).
> > >>>
> > >>> https://issues.apache.org/jira/browse/HADOOP-16797
> > >>>
> > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> > >>>  wrote:
> > >>>
> > >>>> Can ARM binaries be provided after the fact? We cannot increase the
> > RM’s
> > >>>> burden by asking them to generate an extra set of binaries.
> > >>>>
> > >>>>
> > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> > bra...@apache.org>
> > >>>> wrote:
> > >>>>>
> > >>>>> + Dev mailing list.
> > >>>>>
> > >>>>> -- Forwarded message -
> > >>>>> From: Brahma Reddy Battula 
> > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
> > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > >>>>> To: junping_du 
> > >>>>>
> > >>>>>
> > >>>>> thanks junping for your reply.
> > >>>>>
> > >>>>> bq.  I think most of us in Hadoop community doesn't want to
> have
> > >>>> biased
> > >>>>> on ARM or any other platforms.
> > >>>>>
> > >>>>> Yes, release voting will be based on the source code.AFAIK,Binary
> we
> > >> are
> > >>>>> providing for user to easy to download and verify.
> > >>>>>
> > >>>>> bq. The only thing I try to understand is how much complexity
> get
> > >>>>> involved for our RM work. Does that potentially become a blocker
> for
> > >>>> future
> > >>>>> releases? And how we can get rid of this risk.
> > >>>>>
> > >>>>> As I mentioned earlier, RM need to access the ARM machine(it will
> be
> > >>>>> donated and current qbt also using one ARM machine) and build tar
> > using
> > >>>> the
> > >>>>> keys. As it can be common machine, RM can delete his keys once
> > release
> > >>>>> approved.
> > >>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM
> > >> machine)
> > >>>>>
> > >>>>> bq.   If you can list the concrete work that RM need to do
> extra
> > >> for
> > >>>>> ARM release, that would hel

Re: [RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Brahma Reddy Battula]

My late +1.. I missed this..


On Thu, 19 Mar 2020 at 12:43 AM, Vinayakumar B 
wrote:

> Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s
>
> Thanks everyone for verification.
> I will push out bits soon.
>
> -Vinay
>
>
> On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore <
> surendralilh...@gmail.com> wrote:
>
> > +1
> >
> > -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0
> > -Verified artifacts available in repo.
> >
> >
> > Thanks
> > Surendra
> >
> >
> > On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka 
> wrote:
> >
> >> +1
> >>
> >> - Verified signatures and checksums
> >> - Built jars and docs from source
> >> - Built hadoop trunk with hadoop-thirdparty 1.0.0
> >> - Checked rat files and documents
> >> - Checked LICENSE and NOTICE files
> >>
> >> Thanks,
> >> Akira
> >>
> >> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B 
> >> wrote:
> >>
> >> > Hi folks,
> >> >
> >> > Thanks to everyone's help on this release.
> >> >
> >> > I have re-created a release candidate (RC1) for Apache Hadoop
> Thirdparty
> >> > 1.0.0.
> >> >
> >> > RC Release artifacts are available at :
> >> >
> >> >
> >>
> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
> >> >
> >> > Maven artifacts are available in staging repo:
> >> >
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
> >> >
> >> > The RC tag in git is here:
> >> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
> >> >
> >> > And my public key is at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm
> >> IST.*
> >> >
> >> > For the testing, I have verified Hadoop trunk compilation with
> >> >"-DdistMgmtSnapshotsUrl=
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
> >> >  -Dhadoop-thirdparty-protobuf.version=1.0.0"
> >> >
> >> > My +1 to start.
> >> >
> >> > -Vinay
> >> >
> >>
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Sure, I will update in cwiki,Once it's concluded here..Thanks a lot arpit...

On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
 wrote:

> Thanks for the clarification Brahma. Can you update the proposal to state
> that it is optional (it may help to put the proposal on cwiki)?
>
> Also if we go ahead then the RM documentation should be clear this is an
> optional step.
>
>
> > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula 
> wrote:
> >
> > Sure, we can't make mandatory while voting and we can upload to downloads
> > once release vote is passed.
> >
> > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> >  wrote:
> >
> >>> Sorry,didn't get you...do you mean, once release voting is
> >>> processed and upload by RM..?
> >>
> >> Yes, that is what I meant. I don’t want us to make more mandatory work
> for
> >> the release manager because the job is hard enough already.
> >>
> >>
> >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
> >> wrote:
> >>>
> >>> Sorry,didn't get you...do you mean, once release voting is processed
> and
> >>> upload by RM..?
> >>>
> >>> FYI. There is docker image for ARM also which support all scripts
> >>> (createrelease, start-build-env.sh, etc ).
> >>>
> >>> https://issues.apache.org/jira/browse/HADOOP-16797
> >>>
> >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> >>>  wrote:
> >>>
> >>>> Can ARM binaries be provided after the fact? We cannot increase the
> RM’s
> >>>> burden by asking them to generate an extra set of binaries.
> >>>>
> >>>>
> >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> bra...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>> + Dev mailing list.
> >>>>>
> >>>>> -- Forwarded message -
> >>>>> From: Brahma Reddy Battula 
> >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
> >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> >>>>> To: junping_du 
> >>>>>
> >>>>>
> >>>>> thanks junping for your reply.
> >>>>>
> >>>>> bq.  I think most of us in Hadoop community doesn't want to have
> >>>> biased
> >>>>> on ARM or any other platforms.
> >>>>>
> >>>>> Yes, release voting will be based on the source code.AFAIK,Binary we
> >> are
> >>>>> providing for user to easy to download and verify.
> >>>>>
> >>>>> bq. The only thing I try to understand is how much complexity get
> >>>>> involved for our RM work. Does that potentially become a blocker for
> >>>> future
> >>>>> releases? And how we can get rid of this risk.
> >>>>>
> >>>>> As I mentioned earlier, RM need to access the ARM machine(it will be
> >>>>> donated and current qbt also using one ARM machine) and build tar
> using
> >>>> the
> >>>>> keys. As it can be common machine, RM can delete his keys once
> release
> >>>>> approved.
> >>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM
> >> machine)
> >>>>>
> >>>>> bq.   If you can list the concrete work that RM need to do extra
> >> for
> >>>>> ARM release, that would help us to better understand.
> >>>>>
> >>>>> I can write and update for future reference.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >>>>>
> >>>>>> Hi Brahma,
> >>>>>>   I think most of us in Hadoop community doesn't want to have biased
> >>>> on
> >>>>>> ARM or any other platforms.
> >>>>>>   The only thing I try to understand is how much complexity get
> >>>>>> involved for our RM work. Does that potentially become a blocker for
> >>>> future
> >>>>>> releases? And how we can get rid of this risk.
> >>>>>>If you can li

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Sure, we can't make mandatory while voting and we can upload to downloads
once release vote is passed.

On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
 wrote:

> > Sorry,didn't get you...do you mean, once release voting is
> > processed and upload by RM..?
>
> Yes, that is what I meant. I don’t want us to make more mandatory work for
> the release manager because the job is hard enough already.
>
>
> > On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
> wrote:
> >
> > Sorry,didn't get you...do you mean, once release voting is processed and
> > upload by RM..?
> >
> > FYI. There is docker image for ARM also which support all scripts
> > (createrelease, start-build-env.sh, etc ).
> >
> > https://issues.apache.org/jira/browse/HADOOP-16797
> >
> > On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> >  wrote:
> >
> >> Can ARM binaries be provided after the fact? We cannot increase the RM’s
> >> burden by asking them to generate an extra set of binaries.
> >>
> >>
> >>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula 
> >> wrote:
> >>>
> >>> + Dev mailing list.
> >>>
> >>> -- Forwarded message -
> >>> From: Brahma Reddy Battula 
> >>> Date: Tue, Mar 17, 2020 at 10:31 PM
> >>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> >>> To: junping_du 
> >>>
> >>>
> >>> thanks junping for your reply.
> >>>
> >>> bq.  I think most of us in Hadoop community doesn't want to have
> >> biased
> >>> on ARM or any other platforms.
> >>>
> >>> Yes, release voting will be based on the source code.AFAIK,Binary we
> are
> >>> providing for user to easy to download and verify.
> >>>
> >>> bq. The only thing I try to understand is how much complexity get
> >>> involved for our RM work. Does that potentially become a blocker for
> >> future
> >>> releases? And how we can get rid of this risk.
> >>>
> >>> As I mentioned earlier, RM need to access the ARM machine(it will be
> >>> donated and current qbt also using one ARM machine) and build tar using
> >> the
> >>> keys. As it can be common machine, RM can delete his keys once release
> >>> approved.
> >>> Can be sorted out as I mentioned earlier.(For accessing the ARM
> machine)
> >>>
> >>> bq.   If you can list the concrete work that RM need to do extra
> for
> >>> ARM release, that would help us to better understand.
> >>>
> >>> I can write and update for future reference.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >>>
> >>>> Hi Brahma,
> >>>>    I think most of us in Hadoop community doesn't want to have biased
> >> on
> >>>> ARM or any other platforms.
> >>>>The only thing I try to understand is how much complexity get
> >>>> involved for our RM work. Does that potentially become a blocker for
> >> future
> >>>> releases? And how we can get rid of this risk.
> >>>> If you can list the concrete work that RM need to do extra for ARM
> >>>> release, that would help us to better understand.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Junping
> >>>>
> >>>> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >>>>
> >>>>> If you can provide ARM release for future releases, I'm fine with
> that.
> >>>>>
> >>>>> Thanks,
> >>>>> Akira
> >>>>>
> >>>>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
> >> bra...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> thanks Akira.
> >>>>>>
> >>>>>> Currently only problem is dedicated ARM for future RM.This i want to
> >>>>> sort
> >>>>>> out like below,if you've some other,please let me know.
> >>>>>>
> >>>>>> i) Single machine and share cred to future RM ( as we can delete
> keys
> >>>>> once
> >>>>>> release is over).
> >>>>>> ii) Creating the 

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Sorry,didn't get you...do you mean, once release voting is processed and
upload by RM..?

FYI. There is docker image for ARM also which support all scripts
(createrelease, start-build-env.sh, etc ).

https://issues.apache.org/jira/browse/HADOOP-16797

On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
 wrote:

> Can ARM binaries be provided after the fact? We cannot increase the RM’s
> burden by asking them to generate an extra set of binaries.
>
>
> > On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula 
> wrote:
> >
> > + Dev mailing list.
> >
> > -- Forwarded message -
> > From: Brahma Reddy Battula 
> > Date: Tue, Mar 17, 2020 at 10:31 PM
> > Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > To: junping_du 
> >
> >
> > thanks junping for your reply.
> >
> > bq.  I think most of us in Hadoop community doesn't want to have
> biased
> > on ARM or any other platforms.
> >
> > Yes, release voting will be based on the source code.AFAIK,Binary we are
> > providing for user to easy to download and verify.
> >
> > bq. The only thing I try to understand is how much complexity get
> > involved for our RM work. Does that potentially become a blocker for
> future
> > releases? And how we can get rid of this risk.
> >
> > As I mentioned earlier, RM need to access the ARM machine(it will be
> > donated and current qbt also using one ARM machine) and build tar using
> the
> > keys. As it can be common machine, RM can delete his keys once release
> > approved.
> > Can be sorted out as I mentioned earlier.(For accessing the ARM machine)
> >
> > bq.   If you can list the concrete work that RM need to do extra for
> > ARM release, that would help us to better understand.
> >
> > I can write and update for future reference.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >
> >> Hi Brahma,
> >> I think most of us in Hadoop community doesn't want to have biased
> on
> >> ARM or any other platforms.
> >> The only thing I try to understand is how much complexity get
> >> involved for our RM work. Does that potentially become a blocker for
> future
> >> releases? And how we can get rid of this risk.
> >>  If you can list the concrete work that RM need to do extra for ARM
> >> release, that would help us to better understand.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >>
> >>> If you can provide ARM release for future releases, I'm fine with that.
> >>>
> >>> Thanks,
> >>> Akira
> >>>
> >>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
> bra...@apache.org>
> >>> wrote:
> >>>
> >>>> thanks Akira.
> >>>>
> >>>> Currently only problem is dedicated ARM for future RM.This i want to
> >>> sort
> >>>> out like below,if you've some other,please let me know.
> >>>>
> >>>> i) Single machine and share cred to future RM ( as we can delete keys
> >>> once
> >>>> release is over).
> >>>> ii) Creating the jenkins project ( may be we need to discuss in the
> >>>> board..)
> >>>> iii) I can provide ARM release for future releases.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
> >>> wrote:
> >>>>
> >>>>> Hi Brahma,
> >>>>>
> >>>>> I think we cannot do any of your proposed actions.
> >>>>>
> >>>>>
> >>>>
> >>>
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> >>>>>> Strictly speaking, releases must be verified on hardware owned and
> >>>>> controlled by the committer. That means hardware the committer has
> >>>> physical
> >>>>> possession and control of and exclusively full
> >>> administrative/superuser
> >>>>> access to. That's because only such hardware is qualified to hold a
> >>> PGP
> >>>>> private key, and the release should be verified on the machine the
> >>>> private
> >>>>> key

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Thanks Masatake!!

I was aware of this thread which you given for reference as I am the source
to discuss this(as I verified binary and given some comments). Please check
following for same.

https://lists.apache.org/list.html?common-...@hadoop.apache.org:2017-7


AFAIK, that discussion whether we should vote ton he binary or not.Even
Andrew discussed with legal team [1] and finally it was concluded that vote
should only on source I think.

1. https://issues.apache.org/jira/browse/LEGAL-323


On Tue, Mar 17, 2020 at 11:23 AM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> This thread seems to be relevant.
>
> https://lists.apache.org/thread.html/0d2a1b39f7e890c4f40be5fd92f107fbf048b936005901b7b53dd0f1%40%3Ccommon-dev.hadoop.apache.org%3E
>
>  > Convenience binary artifacts are not official release artifacts and thus
>  > are not voted on. However, since they are distributed by Apache, they
> are
>  > still subject to the same distribution requirements as official release
>  > artifacts. This means they need to have a LICENSE and NOTICE file,
> follow
>  > ASF licensing rules, etc. The PMC needs to ensure that binary artifacts
>  > meet these requirements.
>  >
>  > However, being a "convenience" artifact doesn't mean it isn't important.
>  > The appropriate level of quality for binary artifacts is left up to the
>  > project. An OpenOffice person mentioned the quality of their binary
>  > artifacts is super important since very few of their users will compile
>  > their own office suite.
>  >
>  > I don't know if we've discussed the topic of binary artifact quality in
>  > Hadoop. My stance is that if we're going to publish something, it
> should be
>  > good, or we shouldn't publish it at all. I think we do want to publish
>  > binary tarballs (it's the easiest way for new users to get started with
>  > Hadoop), so it's fair to consider them when evaluating a release.
>
> Just providing build machine to RM would not be enough if
> PMC need to ensure that binary artifiacts meet these requirements.
>
> Thanks,
> Masatake Iwasaki
>
> On 3/17/20 14:11, 俊平堵 wrote:
> > Hi Brahma,
> >   I think most of us in Hadoop community doesn't want to have biased
> on
> > ARM or any other platforms.
> >   The only thing I try to understand is how much complexity get
> involved
> > for our RM work. Does that potentially become a blocker for future
> > releases? And how we can get rid of this risk.
> >If you can list the concrete work that RM need to do extra for ARM
> > release, that would help us to better understand.
> >
> > Thanks,
> >
> > Junping
> >
> > Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >
> >> If you can provide ARM release for future releases, I'm fine with that.
> >>
> >> Thanks,
> >> Akira
> >>
> >> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula  >
> >> wrote:
> >>
> >>> thanks Akira.
> >>>
> >>> Currently only problem is dedicated ARM for future RM.This i want to
> sort
> >>> out like below,if you've some other,please let me know.
> >>>
> >>> i) Single machine and share cred to future RM ( as we can delete keys
> >> once
> >>> release is over).
> >>> ii) Creating the jenkins project ( may be we need to discuss in the
> >>> board..)
> >>> iii) I can provide ARM release for future releases.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
> >> wrote:
> >>>> Hi Brahma,
> >>>>
> >>>> I think we cannot do any of your proposed actions.
> >>>>
> >>>>
> >>
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> >>>>> Strictly speaking, releases must be verified on hardware owned and
> >>>> controlled by the committer. That means hardware the committer has
> >>> physical
> >>>> possession and control of and exclusively full
> administrative/superuser
> >>>> access to. That's because only such hardware is qualified to hold a
> PGP
> >>>> private key, and the release should be verified on the machine the
> >>> private
> >>>> key lives on or on a machine as trusted as that.
> >>>>
> >>>> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> >>>>> Private keys MUST NOT be stored on an

Fwd: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

+ Dev mailing list.

-- Forwarded message -
From: Brahma Reddy Battula 
Date: Tue, Mar 17, 2020 at 10:31 PM
Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
To: junping_du 


thanks junping for your reply.

bq.  I think most of us in Hadoop community doesn't want to have biased
on ARM or any other platforms.

Yes, release voting will be based on the source code.AFAIK,Binary we are
providing for user to easy to download and verify.

bq. The only thing I try to understand is how much complexity get
involved for our RM work. Does that potentially become a blocker for future
releases? And how we can get rid of this risk.

As I mentioned earlier, RM need to access the ARM machine(it will be
donated and current qbt also using one ARM machine) and build tar using the
keys. As it can be common machine, RM can delete his keys once release
approved.
Can be sorted out as I mentioned earlier.(For accessing the ARM machine)

bq.   If you can list the concrete work that RM need to do extra for
ARM release, that would help us to better understand.

I can write and update for future reference.









On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:

> Hi Brahma,
>  I think most of us in Hadoop community doesn't want to have biased on
> ARM or any other platforms.
>  The only thing I try to understand is how much complexity get
> involved for our RM work. Does that potentially become a blocker for future
> releases? And how we can get rid of this risk.
>   If you can list the concrete work that RM need to do extra for ARM
> release, that would help us to better understand.
>
> Thanks,
>
> Junping
>
> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
>
>> If you can provide ARM release for future releases, I'm fine with that.
>>
>> Thanks,
>> Akira
>>
>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula 
>> wrote:
>>
>> > thanks Akira.
>> >
>> > Currently only problem is dedicated ARM for future RM.This i want to
>> sort
>> > out like below,if you've some other,please let me know.
>> >
>> > i) Single machine and share cred to future RM ( as we can delete keys
>> once
>> > release is over).
>> > ii) Creating the jenkins project ( may be we need to discuss in the
>> > board..)
>> > iii) I can provide ARM release for future releases.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
>> wrote:
>> >
>> > > Hi Brahma,
>> > >
>> > > I think we cannot do any of your proposed actions.
>> > >
>> > >
>> >
>> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
>> > > > Strictly speaking, releases must be verified on hardware owned and
>> > > controlled by the committer. That means hardware the committer has
>> > physical
>> > > possession and control of and exclusively full
>> administrative/superuser
>> > > access to. That's because only such hardware is qualified to hold a
>> PGP
>> > > private key, and the release should be verified on the machine the
>> > private
>> > > key lives on or on a machine as trusted as that.
>> > >
>> > > https://www.apache.org/dev/release-distribution.html#sigs-and-sums
>> > > > Private keys MUST NOT be stored on any ASF machine. Likewise,
>> > signatures
>> > > for releases MUST NOT be created on ASF machines.
>> > >
>> > > We need to have dedicated physical ARM machines for each release
>> manager,
>> > > and now it is not feasible.
>> > > If you provide an unofficial ARM binary release in some repository,
>> > that's
>> > > okay.
>> > >
>> > > -Akira
>> > >
>> > > On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula <
>> bra...@apache.org>
>> > > wrote:
>> > >
>> > >> Hello folks,
>> > >>
>> > >> As currently trunk will support ARM based compilation and qbt(1) is
>> > >> running
>> > >> from several months with quite stable, hence planning to propose ARM
>> > >> binary
>> > >> this time.
>> > >>
>> > >> ( Note : As we'll know voting will be based on the source,so this
>> will
>> > not
>> > >> issue.)
>> > >>
>> > >> *Proposed Change:*
>> > >> Currently in downloads we are keeping only x86 binary(2),Can we keep
>> ARM
>> > >> binary also.?
>> > >>
>> > >> *Actions:*
>> > >> a) *Dedicated* *Machine*:
>> > >>i) Dedicated ARM machine will be donated which I confirmed
>> > >>ii) Or can use jenkins ARM machine itself which is currently
>> used
>> > >> for ARM
>> > >> b) *Automate Release:* How about having one release project in
>> > jenkins..?
>> > >> So that future RM's just trigger the jenkin project.
>> > >>
>> > >> Please let me know your thoughts on this.
>> > >>
>> > >>
>> > >> 1.
>> > >>
>> > >>
>> >
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> > >> 2.https://hadoop.apache.org/releases.html
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --Brahma Reddy Battula
>> > >>
>> > >
>> >
>> > --
>> >
>> >
>> >
>> > --Brahma Reddy Battula
>> >
>>
>

-- 



--Brahma Reddy Battula


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

thanks Akira.

Currently only problem is dedicated ARM for future RM.This i want to sort
out like below,if you've some other,please let me know.

i) Single machine and share cred to future RM ( as we can delete keys once
release is over).
ii) Creating the jenkins project ( may be we need to discuss in the board..)
iii) I can provide ARM release for future releases.







On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka  wrote:

> Hi Brahma,
>
> I think we cannot do any of your proposed actions.
>
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> > Strictly speaking, releases must be verified on hardware owned and
> controlled by the committer. That means hardware the committer has physical
> possession and control of and exclusively full administrative/superuser
> access to. That's because only such hardware is qualified to hold a PGP
> private key, and the release should be verified on the machine the private
> key lives on or on a machine as trusted as that.
>
> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> > Private keys MUST NOT be stored on any ASF machine. Likewise, signatures
> for releases MUST NOT be created on ASF machines.
>
> We need to have dedicated physical ARM machines for each release manager,
> and now it is not feasible.
> If you provide an unofficial ARM binary release in some repository, that's
> okay.
>
> -Akira
>
> On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula 
> wrote:
>
>> Hello folks,
>>
>> As currently trunk will support ARM based compilation and qbt(1) is
>> running
>> from several months with quite stable, hence planning to propose ARM
>> binary
>> this time.
>>
>> ( Note : As we'll know voting will be based on the source,so this will not
>> issue.)
>>
>> *Proposed Change:*
>> Currently in downloads we are keeping only x86 binary(2),Can we keep ARM
>> binary also.?
>>
>> *Actions:*
>> a) *Dedicated* *Machine*:
>>i) Dedicated ARM machine will be donated which I confirmed
>>ii) Or can use jenkins ARM machine itself which is currently used
>> for ARM
>> b) *Automate Release:* How about having one release project in jenkins..?
>> So that future RM's just trigger the jenkin project.
>>
>> Please let me know your thoughts on this.
>>
>>
>> 1.
>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> 2.https://hadoop.apache.org/releases.html
>>
>>
>>
>>
>>
>>
>> --Brahma Reddy Battula
>>
>

-- 



--Brahma Reddy Battula

[DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Hello folks,

As currently trunk will support ARM based compilation and qbt(1) is running
from several months with quite stable, hence planning to propose ARM binary
this time.

( Note : As we'll know voting will be based on the source,so this will not
issue.)

*Proposed Change:*
Currently in downloads we are keeping only x86 binary(2),Can we keep ARM
binary also.?

*Actions:*
a) *Dedicated* *Machine*:
   i) Dedicated ARM machine will be donated which I confirmed
   ii) Or can use jenkins ARM machine itself which is currently used
for ARM
b) *Automate Release:* How about having one release project in jenkins..?
So that future RM's just trigger the jenkin project.

Please let me know your thoughts on this.


1.
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
2.https://hadoop.apache.org/releases.html






--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Hi All,

Inline with the original 3.3.0 communication proposal dated 8th Jan 2020, I
would like to provide more updates[1].

We are approaching previously proposed code freeze date (March 10,2020). So
I would like to cut 3.3 branch on *10th March* and point existing *trunk to
3.4* if there are no issues.

*Current Release Plan:*

*Feature freeze Date *: all features to merge by Feb 28, 2020.( it's almost
done)
*Code freeze Date *: blockers/critical only, no improvements and
blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues
are there which are very old and I am tracking same.)
*Release Date*: March 15, 2020.

*Please let me know if I missed anything.*

1. https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
BY priority DESC


On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula 
wrote:

>
> Wiki was updated for 3.3
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0.
>
>
> >I'll move out anything that isn't needed.
>
> thanks steve.
>
> > We need to fix the shaded protobuf in
> > Token issue to even get spark to compile.
>
> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621
>
> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran 
> wrote:
>
>> >
>> > 2. Features close to finish:
>> >
>> >
>> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
>> owner
>> > :  Steve Loughran)
>> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
>> > fixes. ( owner :  Steve Loughran)
>> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
>> > owner :  Steve Loughran)
>> >
>> > I'll move out anything that isn't needed.
>>
>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing and
>> I've got some provisional tuning to do. That is -if things didn't work in
>> the test/production deployments, I'd know about the regressions (e.g.
>> HADOOP-16751).
>>
>> This is S3A and ABFS code -no idea about the rest, and inevitably the big
>> JAR changes will have surprises. We need to fix the shaded protobuf in
>> Token issue to even get spark to compile.
>>
>> -Steve
>>
>> >
>> >
>>
>
>
> --
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Wiki was updated for 3.3
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0.


>I'll move out anything that isn't needed.

thanks steve.

> We need to fix the shaded protobuf in
> Token issue to even get spark to compile.

Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621

On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran 
wrote:

> >
> > 2. Features close to finish:
> >
> >
> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
> owner
> > :  Steve Loughran)
> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
> > fixes. ( owner :  Steve Loughran)
> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
> > owner :  Steve Loughran)
> >
> > I'll move out anything that isn't needed.
>
> FWIW, most of these are in CDP 1.x, so there's been reasonable testing and
> I've got some provisional tuning to do. That is -if things didn't work in
> the test/production deployments, I'd know about the regressions (e.g.
> HADOOP-16751).
>
> This is S3A and ABFS code -no idea about the rest, and inevitably the big
> JAR changes will have surprises. We need to fix the shaded protobuf in
> Token issue to even get spark to compile.
>
> -Steve
>
> >
> >
>


-- 
--Brahma Reddy Battula

[jira] [Created] (YARN-10089) [Rollingupragde] PhysicalResource be always null (RMNode should be updated NM registeration))

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created YARN-10089:
---

 Summary: [Rollingupragde] PhysicalResource be always null (RMNode 
should be updated NM registeration))
 Key: YARN-10089
 URL: https://issues.apache.org/jira/browse/YARN-10089
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Brahma Reddy Battula


PhysicalResource will be null always, in following scenario

i) Upgrade RM from 2.7 to 3.0.

ii) Upgrade NM from 2.7 to 3.0.

Here when NM re-register,as RMContext already have this nodeID so it will not 
added again as httpport also same hence "PhysicalResource" will be always null 
in the upgraded cluster till RM restart.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5542) Scheduling of opportunistic containers

[Brahma Reddy Battula (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved YARN-5542.

Fix Version/s: 3.3.0
   Resolution: Fixed

Resolving all of this subtasks are closed.

> Scheduling of opportunistic containers
> --
>
> Key: YARN-5542
> URL: https://issues.apache.org/jira/browse/YARN-5542
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Konstantinos Karanasos
>Priority: Major
> Fix For: 3.3.0
>
>
> This JIRA groups all efforts related to the scheduling of opportunistic 
> containers. 
> It includes the scheduling of opportunistic container through the central RM 
> (YARN-5220), through distributed scheduling (YARN-2877), as well as the 
> scheduling of containers based on actual node utilization (YARN-1011) and the 
> container promotion/demotion (YARN-5085).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Hi All,



To continue a faster cadence of releases to accommodate more features,we could 
plan a Hadoop 3.3 release around March Mid.



To start the process sooner, and to establish a timeline, I propose to target 
Hadoop 3.3.0 release by March Mid 2020. (About 2 months from now).



I would also would like to take this opportunity to come up with a detailed 
plan.



Feature Freeze Date : All features should be merged by Feb 28, 2020.



Code Freeze Date : blockers/critical only, no improvements and non 
blocker/critical bug-fixes March 10, 2020.



Release Date: March 15, 2020



I have tried to come up with a list of features on my radar which could be 
candidates



1. Merged & Completed features:

  *   HDFS-13891 HDFS RBF stabilization phase 1. (Owner: Brahma)
  *   HDFS-12345: Scale testing HDFS NameNode with real metadata and workloads 
(Dynamometer) (owner: Erik Krogen)
  *   HDFS-13762: Support non-volatile storage class memory(SCM) in HDFS cache 
directives ( owner: Feilong He)
  *   HADOOP-16095 : Support impersonation for AuthenticationFilter (owner:  
Eric Yang)
  *   YARN-7129: Application Catalog for YARN applications (Owner: Eric Yang)
  *   YARN-5542: Scheduling of opportunistic containers (owner: Konstantinos 
Karanasos)
  *   YARN-9473: Support Vector Engine ( a new accelerator hardware) based on 
pluggable device framework. (owner :Peter Bacsko)
  *   YARN-9264: [Umbrella] Follow-up on IntelOpenCL FPGA plugin. (owner: Peter 
Bacsko)
  *   YARN-9145: [Umbrella] Dynamically add or remove auxiliary services



2. Features close to finish:



  *   HADOOP-13363 Upgrade protobuf from 2.5.0 to something newer (Owner: Vinay)
  *YARN-1011: Schedule containers based on utilization of currently 
allocated containers
  *   (Owner: Haibo Chen)
  *YARN-9698: [Umbrella] Tools to help migration from Fair Scheduler to 
Capacity Scheduler. (Owner: Weiwei Yang)
  *YARN-9050: [Umbrella] Usability improvements for scheduler activities. 
(Owner: Tao Yang)
  *YARN-8851:[Umbrella] A pluggable device plugin framework to ease vendor 
plugin development (owner: Zhankun Tang)
  *   YARN-9014: runC container runtime (owner : Eric Badger)
  *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( owner :  
Steve Loughran)
  *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes. ( 
owner :  Steve Loughran)
  *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. ( owner : 
 Steve Loughran)
  *HADOOP-15338: Support Java 11 LTS in Hadoop (owner: Akira Ajisaka)



3. Summary of Issues Status



There are 1781 issues are fixed in 3.3.0(1) which very big number.



13 Blocker and critical issues are open(2),I will followup owners to get status 
on each of them to get in by code Freeze date.



Please let me know if I missed any features targeted to 3.3 per this timeline. 
I would like to volunteer myself as release manager of 3.3.0 release.



Please let me know if you have any suggestions.

Reference:


   1) project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND 
fixVersion = 3.3.0

   2) project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, 
Critical) AND resolution = Unresolved AND "TargetVersion/s" = 3.3.0 
ORDER BY priority DESC



Note:

i) added the owners based on the jira assignee and reporter.. Please correct me
ii) will update cwiki


Regards,
Brahma Reddy Battula

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Brahma Reddy Battula]

gt; Thoughts?
> > > >> > > >
> > > >> > > > .. Owen
> > > >> > > >
> > > >> > > > On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B <
> > > >> vinayakum...@apache.org
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > >> Hi All,
> > > >> > > >>
> > > >> > > >>I wanted to discuss about the separate repo for thirdparty
> > > >> > > dependencies
> > > >> > > >> which we need to shaded and include in Hadoop component's
> jars.
> > > >> > > >>
> > > >> > > >>Apologies for the big text ahead, but this needs clear
> > > >> > explanation!!
> > > >> > > >>
> > > >> > > >>Right now most needed such dependency is protobuf.
> Protobuf
> > > >> > > dependency
> > > >> > > >> was not upgraded from 2.5.0 onwards with the fear that
> > downstream
> > > >> > > builds,
> > > >> > > >> which depends on transitive dependency protobuf coming from
> > > >> hadoop's
> > > >> > > jars,
> > > >> > > >> may fail with the upgrade. Apparently protobuf does not
> > guarantee
> > > >> > source
> > > >> > > >> compatibility, though it guarantees wire compatibility
> between
> > > >> > versions.
> > > >> > > >> Because of this behavior, version upgrade may cause breakage
> in
> > > >> known
> > > >> > > and
> > > >> > > >> unknown (private?) downstreams.
> > > >> > > >>
> > > >> > > >>So to tackle this, we came up the following proposal in
> > > >> > HADOOP-13363.
> > > >> > > >>
> > > >> > > >>Luckily, As far as I know, no APIs, either public to user
> or
> > > >> > between
> > > >> > > >> Hadoop processes, is not directly using protobuf classes in
> > > >> > signatures.
> > > >> > > >> (If
> > > >> > > >> any exist, please let us know).
> > > >> > > >>
> > > >> > > >>Proposal:
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>1. Create a artifact(s) which contains shaded
> dependencies.
> > > All
> > > >> > such
> > > >> > > >> shading/relocation will be with known prefix
> > > >> > > >> **org.apache.hadoop.thirdparty.**.
> > > >> > > >>2. Right now protobuf jar (ex:
> > > >> > > o.a.h.thirdparty:hadoop-shaded-protobuf)
> > > >> > > >> to start with, all **com.google.protobuf** classes will be
> > > >> relocated
> > > >> > as
> > > >> > > >> **org.apache.hadoop.thirdparty.com.google.protobuf**.
> > > >> > > >>3. Hadoop modules, which needs protobuf as dependency,
> will
> > > add
> > > >> > this
> > > >> > > >> shaded artifact as dependency (ex:
> > > >> > > >> o.a.h.thirdparty:hadoop-shaded-protobuf).
> > > >> > > >>4. All previous usages of "com.google.protobuf" will be
> > > >> relocated
> > > >> > to
> > > >> > > >> "org.apache.hadoop.thirdparty.com.google.protobuf" in the
> code
> > > and
> > > >> > will
> > > >> > > be
> > > >> > > >> committed. Please note, this replacement is One-Time directly
> > in
> > > >> > source
> > > >> > > >> code, NOT during compile and package.
> > > >> > > >>5. Once all usages of "com.google.protobuf" is relocated,
> > then
> > > >> > hadoop
> > > >> > > >> dont care about which version of original  "protobuf-java" is
> > in
> > > >> > > >> dependency.
> > > >> > > >>6. Just keep "protobuf-java:2.5.0" in dependency tree not
> to
> > > >> break
> > > >> > > the
> > > >> > > >> downstreams. But hadoop will be originally using the latest
> > > >> protobuf
> > > >> > > >> present in "o.a.h.thirdparty:hadoop-shaded-protobuf".
> > > >> > > >>
> > > >> > > >>7. Coming back to separate repo, Following are most
> > > appropriate
> > > >> > > reasons
> > > >> > > >> of keeping shaded dependency artifact in separate repo
> instead
> > of
> > > >> > > >> submodule.
> > > >> > > >>
> > > >> > > >>  7a. These artifacts need not be built all the time. It
> > needs
> > > >> to
> > > >> > be
> > > >> > > >> built only when there is a change in the dependency version
> or
> > > the
> > > >> > build
> > > >> > > >> process.
> > > >> > > >>  7b. If added as "submodule in Hadoop repo",
> > > >> > > maven-shade-plugin:shade
> > > >> > > >> will execute only in package phase. That means, "mvn compile"
> > or
> > > >> "mvn
> > > >> > > >> test-compile" will not be failed as this artifact will not
> have
> > > >> > > relocated
> > > >> > > >> classes, instead it will have original classes, resulting in
> > > >> > compilation
> > > >> > > >> failure. Workaround, build thirdparty submodule first and
> > exclude
> > > >> > > >> "thirdparty" submodule in other executions. This will be a
> > > complex
> > > >> > > process
> > > >> > > >> compared to keeping in a separate repo.
> > > >> > > >>
> > > >> > > >>  7c. Separate repo, will be a subproject of Hadoop, using
> > the
> > > >> > same
> > > >> > > >> HADOOP jira project, with different versioning prefixed with
> > > >> > > "thirdparty-"
> > > >> > > >> (ex: thirdparty-1.0.0).
> > > >> > > >>  7d. Separate will have same release process as Hadoop.
> > > >> > > >>
> > > >> > > >>HADOOP-13363 (
> > > >> https://issues.apache.org/jira/browse/HADOOP-13363)
> > > >> > > is
> > > >> > > >> an
> > > >> > > >> umbrella jira tracking the changes to protobuf upgrade.
> > > >> > > >>
> > > >> > > >>PR (https://github.com/apache/hadoop-thirdparty/pull/1)
> has
> > > >> been
> > > >> > > >> raised
> > > >> > > >> for separate repo creation in (HADOOP-16595 (
> > > >> > > >> https://issues.apache.org/jira/browse/HADOOP-16595)
> > > >> > > >>
> > > >> > > >>Please provide your inputs for the proposal and review the
> > PR
> > > >> to
> > > >> > > >> proceed with the proposal.
> > > >> > > >>
> > > >> > > >>
> > > >> > > >-Thanks,
> > > >> > > >>Vinay
> > > >> > > >>
> > > >> > > >> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli <
> > > >> > > >> vino...@apache.org>
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> > Moving the thread to the dev lists.
> > > >> > > >> >
> > > >> > > >> > Thanks
> > > >> > > >> > +Vinod
> > > >> > > >> >
> > > >> > > >> > > On Sep 23, 2019, at 11:43 PM, Vinayakumar B <
> > > >> > > vinayakum...@apache.org>
> > > >> > > >> > wrote:
> > > >> > > >> > >
> > > >> > > >> > > Thanks Marton,
> > > >> > > >> > >
> > > >> > > >> > > Current created 'hadoop-thirdparty' repo is empty right
> > now.
> > > >> > > >> > > Whether to use that repo  for shaded artifact or not will
> > be
> > > >> > > >> monitored in
> > > >> > > >> > > HADOOP-13363 umbrella jira. Please feel free to join the
> > > >> > discussion.
> > > >> > > >> > >
> > > >> > > >> > > There is no existing codebase is being moved out of
> hadoop
> > > >> repo.
> > > >> > So
> > > >> > > I
> > > >> > > >> > think
> > > >> > > >> > > right now we are good to go.
> > > >> > > >> > >
> > > >> > > >> > > -Vinay
> > > >> > > >> > >
> > > >> > > >> > > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek <
> > > e...@apache.org>
> > > >> > > wrote:
> > > >> > > >> > >
> > > >> > > >> > >>
> > > >> > > >> > >> I am not sure if it's defined when is a vote required.
> > > >> > > >> > >>
> > > >> > > >> > >> https://www.apache.org/foundation/voting.html
> > > >> > > >> > >>
> > > >> > > >> > >> Personally I think it's a big enough change to send a
> > > >> > notification
> > > >> > > to
> > > >> > > >> > the
> > > >> > > >> > >> dev lists with a 'lazy consensus'  closure
> > > >> > > >> > >>
> > > >> > > >> > >> Marton
> > > >> > > >> > >>
> > > >> > > >> > >> On 2019/09/23 17:46:37, Vinayakumar B <
> > > >> vinayakum...@apache.org>
> > > >> > > >> wrote:
> > > >> > > >> > >>> Hi,
> > > >> > > >> > >>>
> > > >> > > >> > >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may
> > be
> > > >> more
> > > >> > in
> > > >> > > >> > >> future)
> > > >> > > >> > >>> will be kept as a shaded artifact in a separate repo,
> > which
> > > >> will
> > > >> > > be
> > > >> > > >> > >>> referred as dependency in hadoop modules.  This
> approach
> > > >> avoids
> > > >> > > >> shading
> > > >> > > >> > >> of
> > > >> > > >> > >>> every submodule during build.
> > > >> > > >> > >>>
> > > >> > > >> > >>> So question is does any VOTE required before asking to
> > > >> create a
> > > >> > > git
> > > >> > > >> > repo?
> > > >> > > >> > >>>
> > > >> > > >> > >>> On selfserve platform
> > > >> > > https://gitbox.apache.org/setup/newrepo.html
> > > >> > > >> > >>> I can access see that, requester should be PMC.
> > > >> > > >> > >>>
> > > >> > > >> > >>> Wanted to confirm here first.
> > > >> > > >> > >>>
> > > >> > > >> > >>> -Vinay
> > > >> > > >> > >>>
> > > >> > > >> > >>
> > > >> > > >> > >>
> > > >> > >
> > > -
> > > >> > > >> > >> To unsubscribe, e-mail:
> > > private-unsubscr...@hadoop.apache.org
> > > >> > > >> > >> For additional commands, e-mail:
> > > >> private-h...@hadoop.apache.org
> > > >> > > >> > >>
> > > >> > > >> > >>
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 2019 Release Planning

[Brahma Reddy Battula]

Thanks Akira for resuming this..

 I am interested on 3.3 release ..will act as RM .will update the wiki as
well..



On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota 
wrote:

> I'm interested in doing a release of hadoop.
> The version we need an RM is 3.1.3 right? What's the target date for that?
>
> Thanks,
> Gabor
>
> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka  wrote:
>
> > Thank you Wangda.
> >
> > Now it's 2020. Let's release Hadoop 3.3.0.
> > I created a wiki page for tracking blocker/critical issues for 3.3.0 and
> > I'll check the issues in the list.
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
> > If you find blocker/critical issues in trunk, please set the target
> version
> > to 3.3.0 for tracking.
> >
> > > We still need RM for 3.3.0 and 3.1.3.
> > I can work as a release manager for 3.3.0. Is there anyone who wants to
> be
> > a RM?
> >
> > Thanks and regards,
> > Akira
> >
> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
> > wrote:
> >
> > > Thanks Wangda for bring this up!
> > >
> > > I ran the submarine 0.2.0 release before with a lot of help from folks
> > > especially Sunil. :D
> > > And this time I would like to help to release the 3.1.4. Thanks!
> > >
> > > BR,
> > > Zhankun
> > >
> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
> > >
> > > > Hi Wangda,
> > > > Thanks for bringing this up!
> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is
> a
> > > > problem.
> > > > Hope commiters watch and review these issues, Thanks
> > > > https://issues.apache.org/jira/browse/HDFS-13596
> > > > https://issues.apache.org/jira/browse/HDFS-14396
> > > >
> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > Hope this email finds you well
> > > > >
> > > > > I want to hear your thoughts about what should be the release plan
> > for
> > > > > 2019.
> > > > >
> > > > > In 2018, we released:
> > > > > - 1 maintenance release of 2.6
> > > > > - 3 maintenance releases of 2.7
> > > > > - 3 maintenance releases of 2.8
> > > > > - 3 releases of 2.9
> > > > > - 4 releases of 3.0
> > > > > - 2 releases of 3.1
> > > > >
> > > > > Total 16 releases in 2018.
> > > > >
> > > > > In 2019, by far we only have two releases:
> > > > > - 1 maintenance release of 3.1
> > > > > - 1 minor release of 3.2.
> > > > >
> > > > > However, the community put a lot of efforts to stabilize features
> of
> > > > > various release branches.
> > > > > There're:
> > > > > - 217 fixed patches in 3.1.3 [1]
> > > > > - 388 fixed patches in 3.2.1 [2]
> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> > > > >
> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and
> do a
> > > > minor
> > > > > release for 3.3.0.
> > > > >
> > > > > In addition, I saw community discussion to do a 2.8.6 release for
> > > > security
> > > > > fixes.
> > > > >
> > > > > Any other releases? I think there're release plans for Ozone as
> well.
> > > And
> > > > > please add your thoughts.
> > > > >
> > > > > Volunteers welcome! If you have interests to run a release as
> Release
> > > > > Manager (or co-Resource Manager), please respond to this email
> thread
> > > so
> > > > we
> > > > > can coordinate.
> > > > >
> > > > > Thanks,
> > > > > Wangda Tan
> > > > >
> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.1.3
> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.2.1
> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.3.0
> > > > >
> > > >
> > >
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Brahma Reddy Battula]

t; classes, instead it will have original classes, resulting in
> >> > compilation
> >> > > >> failure. Workaround, build thirdparty submodule first and exclude
> >> > > >> "thirdparty" submodule in other executions. This will be a
> complex
> >> > > process
> >> > > >> compared to keeping in a separate repo.
> >> > > >>
> >> > > >>  7c. Separate repo, will be a subproject of Hadoop, using the
> >> > same
> >> > > >> HADOOP jira project, with different versioning prefixed with
> >> > > "thirdparty-"
> >> > > >> (ex: thirdparty-1.0.0).
> >> > > >>  7d. Separate will have same release process as Hadoop.
> >> > > >>
> >> > > >>HADOOP-13363 (
> >> https://issues.apache.org/jira/browse/HADOOP-13363)
> >> > > is
> >> > > >> an
> >> > > >> umbrella jira tracking the changes to protobuf upgrade.
> >> > > >>
> >> > > >>PR (https://github.com/apache/hadoop-thirdparty/pull/1) has
> >> been
> >> > > >> raised
> >> > > >> for separate repo creation in (HADOOP-16595 (
> >> > > >> https://issues.apache.org/jira/browse/HADOOP-16595)
> >> > > >>
> >> > > >>Please provide your inputs for the proposal and review the PR
> >> to
> >> > > >> proceed with the proposal.
> >> > > >>
> >> > > >>
> >> > > >-Thanks,
> >> > > >>Vinay
> >> > > >>
> >> > > >> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli <
> >> > > >> vino...@apache.org>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Moving the thread to the dev lists.
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > +Vinod
> >> > > >> >
> >> > > >> > > On Sep 23, 2019, at 11:43 PM, Vinayakumar B <
> >> > > vinayakum...@apache.org>
> >> > > >> > wrote:
> >> > > >> > >
> >> > > >> > > Thanks Marton,
> >> > > >> > >
> >> > > >> > > Current created 'hadoop-thirdparty' repo is empty right now.
> >> > > >> > > Whether to use that repo  for shaded artifact or not will be
> >> > > >> monitored in
> >> > > >> > > HADOOP-13363 umbrella jira. Please feel free to join the
> >> > discussion.
> >> > > >> > >
> >> > > >> > > There is no existing codebase is being moved out of hadoop
> >> repo.
> >> > So
> >> > > I
> >> > > >> > think
> >> > > >> > > right now we are good to go.
> >> > > >> > >
> >> > > >> > > -Vinay
> >> > > >> > >
> >> > > >> > > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek <
> e...@apache.org>
> >> > > wrote:
> >> > > >> > >
> >> > > >> > >>
> >> > > >> > >> I am not sure if it's defined when is a vote required.
> >> > > >> > >>
> >> > > >> > >> https://www.apache.org/foundation/voting.html
> >> > > >> > >>
> >> > > >> > >> Personally I think it's a big enough change to send a
> >> > notification
> >> > > to
> >> > > >> > the
> >> > > >> > >> dev lists with a 'lazy consensus'  closure
> >> > > >> > >>
> >> > > >> > >> Marton
> >> > > >> > >>
> >> > > >> > >> On 2019/09/23 17:46:37, Vinayakumar B <
> >> vinayakum...@apache.org>
> >> > > >> wrote:
> >> > > >> > >>> Hi,
> >> > > >> > >>>
> >> > > >> > >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be
> >> more
> >> > in
> >> > > >> > >> future)
> >> > > >> > >>> will be kept as a shaded artifact in a separate repo, which
> >> will
> >> > > be
> >> > > >> > >>> referred as dependency in hadoop modules.  This approach
> >> avoids
> >> > > >> shading
> >> > > >> > >> of
> >> > > >> > >>> every submodule during build.
> >> > > >> > >>>
> >> > > >> > >>> So question is does any VOTE required before asking to
> >> create a
> >> > > git
> >> > > >> > repo?
> >> > > >> > >>>
> >> > > >> > >>> On selfserve platform
> >> > > https://gitbox.apache.org/setup/newrepo.html
> >> > > >> > >>> I can access see that, requester should be PMC.
> >> > > >> > >>>
> >> > > >> > >>> Wanted to confirm here first.
> >> > > >> > >>>
> >> > > >> > >>> -Vinay
> >> > > >> > >>>
> >> > > >> > >>
> >> > > >> > >>
> >> > >
> -
> >> > > >> > >> To unsubscribe, e-mail:
> private-unsubscr...@hadoop.apache.org
> >> > > >> > >> For additional commands, e-mail:
> >> private-h...@hadoop.apache.org
> >> > > >> > >>
> >> > > >> > >>
> >> > > >> >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >

-- 



--Brahma Reddy Battula

RE: [VOTE] Moving Submarine to a separate Apache project proposal

[Brahma Reddy Battula]

+1, Thanks for Wangda's proposal.

I am interested to participate in this project. Please include me


-Original Message-
From: Wanqiang Ji [mailto:wanqiang...@gmail.com] 
Sent: Wednesday, September 04, 2019 6:53 PM
To: Wangda Tan 
Cc: submarine-dev ; yarn-dev 
; Hdfs-dev ; 
mapreduce-dev ; Hadoop Common 
; private 
Subject: Re: [VOTE] Moving Submarine to a separate Apache project proposal

+1

Thanks for Wangda's proposal.

It is indeed amazing to see the growth and development of submarine. As the TLP 
will attract more developers to join.
I will put more energy into it and contribute more feature. Look forward to the 
next change in submarine.

Thanks,
Wanqiang Ji

On Wed, Sep 4, 2019 at 3:09 PM Bibin Chundatt 
wrote:

> +1
> Thank you  for the proposal.
> I am interested in project. Please include me as well in the project.
>
> Thanks,
> Bibin
>
> On Tue, Sep 3, 2019 at 6:41 PM Ayush Saxena 
> wrote:
>
> > +1
> > Thanx for the proposal.
> >
> > I would even like to participate in the project.
> > Please add me as well.
> >
> > -Ayush
> >
> >
> > > On 03-Sep-2019, at 6:00 PM, Vinayakumar B 
> > > 
> > wrote:
> > >
> > > +1
> > >
> > > Thanks for the proposal.
> > > Its very interesting project and looks very promising one by 
> > > looking at
> > the
> > > participations from various companies and the speed of development.
> > >
> > > I would also like to participate in the project.
> > > Please add me as well.
> > >
> > > Thanks,
> > > -Vinay
> > >
> > > On Tue, 3 Sep 2019, 12:38 pm Rakesh Radhakrishnan, 
> > >  >
> > > wrote:
> > >
> > >> +1, Thanks for the proposal.
> > >>
> > >> I am interested to participate in this project. Please include me 
> > >> as
> > well
> > >> in the project.
> > >>
> > >> Thanks,
> > >> Rakesh
> > >>
> > >> On Tue, Sep 3, 2019 at 11:59 AM zhankun tang 
> > >> 
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Thanks for Wangda's proposal.
> > >>>
> > >>> The submarine project is born within Hadoop, but not limited to
> Hadoop.
> > >> It
> > >>> began with a trainer on YARN but it quickly realized that only a
> > trainer
> > >> is
> > >>> not enough to meet the AI platform requirements. But now there's 
> > >>> no user-friendly open-source solution covers the whole AI 
> > >>> pipeline like
> > data
> > >>> engineering, training, and serving. And the underlying data
> > >> infrastructure
> > >>> itself is also evolving, for instance, many people love k8s. Not
> > >> mentioning
> > >>> there're many AI domain problems in this area to be solved.
> > >>> It's almost for sure that building such an ML platform would 
> > >>> utilize various other open-source components taking ML into 
> > >>> consideration initially.
> > >>>
> > >>> I see submarine grows rapidly towards an enterprise-grade ML 
> > >>> platform
> > >> which
> > >>> could potentially enable AI ability for data engineer and scientist.
> > This
> > >>> is an exciting thing for both the community and the industry.
> > >>>
> > >>> BR,
> > >>> Zhankun
> > >>>
> > >>>
> >  On Tue, 3 Sep 2019 at 13:34, Xun Liu  wrote:
> > 
> >  +1
> > 
> >  Hello everyone, I am a member of the submarine development team.
> >  I have been contributing to submarine for more than a year.
> >  I have seen the progress of submarine development very fast.
> >  In more than a year, there are 9 long-term developers of 
> >  different companies. Contributing, submarine cumulative code 
> >  has more than 200,000 lines of code, is
> > >> growing
> >  very fast,
> >  and is used in the production environment of multiple companies.
> > 
> >  In the submarine development group, there are 5 PMCs and 
> >  7committer
> > >>> members
> >  from Hadoop, spark, zeppelin projects.
> >  They are very familiar with the development process and
> specifications
> > >> of
> >  the apache community,
> >  and can well grasp the project development progress and project
> > >> quality.
> >  So I recommend submarine to be a TLP project directly.
> > 
> >  We will continue to contribute to the submarine project. :-)
> > 
> >  Xun Liu
> >  Regards
> > 
> > > On Tue, 3 Sep 2019 at 12:01, Devaraj K  wrote:
> > >
> > > +1
> > >
> > > Thanks Wangda for the proposal.
> > > I would like to participate in this project, Please add me 
> > > also to
> > >> the
> > > project.
> > >
> > > Regards
> > > Devaraj K
> > >
> > > On Mon, Sep 2, 2019 at 8:50 PM zac yuan 
> > > 
> > >> wrote:
> > >
> > >> +1
> > >>
> > >> Submarine will be a complete solution for AI service development.
> > >> It
> >  can
> > >> take advantage of two best cluster systems: yarn and k8s, 
> > >> which
> > >> will
> >  help
> > >> more and more people get AI ability. To be a separate Apache
> > >> project,
> > > will
> > >> accelerate the procedure of development apparently.
> > >>
> > >> 

Re: [ANNOUNCE] Eric Badger is now a committer!

[Brahma Reddy Battula]

Congratulations Eric!!!


On Tue, Mar 5, 2019 at 10:50 PM, Eric Payne 
wrote:

> It is my pleasure to announce that Eric Badger has accepted an invitation
> to become a Hadoop Core committer.
>
> Congratulations, Eric! This is well-deserved!
>
> -Eric Payne
>
-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Brahma Reddy Battula]

Sunil thanks for driving this.

+1 ,(binding)

--Built from Source.
--Installed pseudo cluster.
--Ran basic shell commands
--Ran sample jobs pi,wordcount.



On Tue, Jan 8, 2019 at 5:12 PM Sunil G  wrote:

> Hi folks,
>
>
> Thanks to all of you who helped in this release [1] and for helping to vote
> for RC0. I have created second release candidate (RC1) for Apache Hadoop
> 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>
>
> RC tag in git is release-3.2.0-RC1.
>
>
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>
>
> This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>


-- 
--Brahma Reddy Battula

RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Brahma Reddy Battula]

My late +1. Really it's useful feature.. Great work.

-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common ; hdfs-dev 

Cc: mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent 
Reads from Standby

Obviously +1 from me.

With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.

Thanks,
--Konst

On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch 
> HDFS-12943 for Consistent Reads from Standby Node. The feature is 
> intended to scale read RPC workloads. On large clusters reads comprise 
> 95% of all RPCs to the NameNode. We should be able to accommodate 
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple 
> ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent 
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including 
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873), 
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>

Re: [DISCUSS] Move to gitbox

[Brahma Reddy Battula]

+1



On Sat, Dec 8, 2018 at 1:26 PM, Akira Ajisaka  wrote:

> Hi all,
>
> Apache Hadoop git repository is in git-wip-us server and it will be
> decommissioned.
> If there are no objection, I'll file a JIRA ticket with INFRA to
> migrate to https://gitbox.apache.org/ and update documentation.
>
> According to ASF infra team, the timeframe is as follows:
>
> > - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> relocation
> > - January 9th -> February 6th: Mandated (coordinated) relocation
> > - February 7th: All remaining repositories are mass migrated.
> > This timeline may change to accommodate various scenarios.
>
> If we got consensus by January 9th, I can file a ticket with INFRA and
> migrate it.
> Even if we cannot got consensus, the repository will be migrated by
> February 7th.
>
> Regards,
> Akira
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
> --



--Brahma Reddy Battula

RE: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

[Brahma Reddy Battula]

Akira, Thanks for driving this release.

+1 (binding)

-- Built from source
-- Installed HA cluster
--Verified basic operations
--Ran Sample Jobs
--Browsed the UI


-Brahma Reddy Battula

-Original Message-
From: Akira Ajisaka [mailto:aajis...@apache.org] 
Sent: Wednesday, November 14, 2018 6:33 AM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

Hi folks,

I have put together a release candidate (RC0) for Hadoop 2.9.2. It includes 204 
bug fixes and improvements since 2.9.1. [1]

The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/
Git signed tag is release-2.9.2-RC0 and the checksum is
826afbeae31ca687bc2f8471dc841b66ed2c6704
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1166/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Please try the release and vote. The vote will run for 5 days.

[1] https://s.apache.org/2.9.2-fixed-jiras

Thanks,
Akira

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Brahma Reddy Battula]

+1

It’s better to new version link in old version.


Brahma Reddy Battula

On Fri, Aug 31, 2018 at 9:59 PM, Sangjin Lee  wrote:

> +1. Thanks for the work, Marton!
>
> On Fri, Aug 31, 2018 at 8:37 AM Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> > Is there no way to host the new site and the old site concurrently? And
> > link back & forth?
> >
> > +Vinod
> >
> >
> > > On Aug 31, 2018, at 1:07 AM, Elek, Marton  wrote:
> > >
> > > Bumping this thread at last time.
> > >
> > > I have the following proposal:
> > >
> > > 1. I will request a new git repository hadoop-site.git and import the
> > new site to there (which has exactly the same content as the existing
> site).
> > >
> > > 2. I will ask infra to use the new repository as the source of
> > hadoop.apache.org
> > >
> > > 3. I will sync manually all of the changes in the next two months back
> > to the svn site from the git (release announcements, new committers)
> > >
> > > IN CASE OF ANY PROBLEM we can switch back to the svn without any
> problem.
> > >
> > > If no-one objects within three days, I'll assume lazy consensus and
> > start with this plan. Please comment if you have objections.
> > >
> > > Again: it allows immediate fallback at any time as svn repo will be
> kept
> > as is (+ I will keep it up-to-date in the next 2 months)
> > >
> > > Thanks,
> > > Marton
> > >
> > >
> > > On 06/21/2018 09:00 PM, Elek, Marton wrote:
> > >> Thank you very much to bump up this thread.
> > >> About [2]: (Just for the clarification) the content of the proposed
> > website is exactly the same as the old one.
> > >> About [1]. I believe that the "mvn site" is perfect for the
> > documentation but for website creation there are more simple and powerful
> > tools.
> > >> Hugo has more simple compared to jekyll. Just one binary, without
> > dependencies, works everywhere (mac, linux, windows)
> > >> Hugo has much more powerful compared to "mvn site". Easier to
> > create/use more modern layout/theme, and easier to handle the content
> (for
> > example new release announcements could be generated as part of the
> release
> > process)
> > >> I think it's very low risk to try out a new approach for the site (and
> > easy to rollback in case of problems)
> > >> Marton
> > >> ps: I just updated the patch/preview site with the recent releases:
> > >> ***
> > >> * http://hadoop.anzix.net *
> > >> ***
> > >> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> > >>> Got pinged about this offline.
> > >>>
> > >>> Thanks for keeping at it, Marton!
> > >>>
> > >>> I think there are two road-blocks here
> > >>>   (1) Is the mechanism using which the website is built good enough -
> > mvn-site / hugo etc?
> > >>>   (2) Is the new website good enough?
> > >>>
> > >>> For (1), I just think we need more committer attention and get
> > feedback rapidly and get it in.
> > >>>
> > >>> For (2), how about we do it in a different way in the interest of
> > progress?
> > >>>   - We create a hadoop.apache.org/new-site/ where this new site
> goes.
> > >>>   - We then modify the existing web-site to say that there is a new
> > site/experience that folks can click on a link and navigate to
> > >>>   - As this new website matures and gets feedback & fixes, we finally
> > pull the plug at a later point of time when we think we are good to go.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> +Vinod
> > >>>
> > >>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I would like to bump this thread up.
> > >>>>
> > >>>> TLDR; There is a proposed version of a new hadoop site which is
> > available from here: https://elek.github.io/hadoop-site-proposal/ and
> > https://issues.apache.org/jira/browse/HADOOP-14163
> > >>>>
> > >>>> Please let me know what you think about it.
> > >>>>
> > >>>>
> > >>>> Longer version:
> > >>>&

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Brahma Reddy Battula]

+1 ,(binding).

Wangda Tan thanks for driving this.


--Built with source.
--Installed the HA cluster
--Ran basic hdfs operations through shell/API
--Ran Sample jobs like pi, wordcount
--Browsed UI.

On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Brahma Reddy Battula]

Thanks yongjun zhang for driving this release.

+1 (binding).


---Built from the source
---Installed HA cluster
---Execute the basic shell commands
---Browsed the UI's
---Ran sample jobs like pi,wordcount




From: Yongjun Zhang 
Sent: Friday, June 8, 2018 1:04 PM
To: Allen Wittenauer
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

BTW, thanks Allen and Steve for discussing and suggestion about the site
build problem I hit earlier, I did the following step

mvn install -DskipTests

before doing the steps Nanda listed helped to solve the problems.

--Yongjun




On Thu, Jun 7, 2018 at 6:15 PM, Yongjun Zhang  wrote:

> Thank you all very much for the testing, feedback and discussion!
>
> I was able to build outside docker, by following the steps Nanda
> described, I saw the same problem; then I tried 3.0.2 released a while
> back, it has the same issue.
>
> As Allen pointed out, it seems the steps to build site are not correct. I
> have not figured out the correct steps yet.
>
> At this point, I think this issue should not block the 3.0.3 issue. While
> at the same time we need to figure out the right steps to build the site.
> Would you please let me know if you think differently?
>
> We only have the site build issue reported so far. And we don't have
> enough PMC votes yet. So need some more PMCs to help.
>
> Thanks again, and best regards,
>
> --Yongjun
>
>
> On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer  > wrote:
>
>> > On Jun 7, 2018, at 11:47 AM, Steve Loughran 
>> wrote:
>> >
>> > Actually, Yongjun has been really good at helping me get set up for a
>> 2.7.7 release, including "things you need to do to get GPG working in the
>> docker image”
>>
>> *shrugs* I use a different release script after some changes
>> broke the in-tree version for building on OS X and I couldn’t get the fixes
>> committed upstream.  So not sure what the problems are that you are hitting.
>>
>> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu <
>> nvadiv...@hortonworks.com> wrote:
>> >
>> > It will be helpful if we can get the correct steps, and also update the
>> wiki.
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>> Release+Validation
>>
>> Yup. Looking forward to seeing it.
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>
>

RE: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

[Brahma Reddy Battula]

Thanks Junping for driving this release.


+1  (binding)


-- Build successfully from the source code
-- Start HA cluster
-- Verified basic shell operations
-- Ran pi,wordcount
-- Browsed the NN and RM UI

  


-Brahma Reddy Battula

-Original Message-
From: 俊平堵 [mailto:junping...@apache.org] 
Sent: 09 May 2018 01:41
To: Hadoop Common <common-...@hadoop.apache.org>; Hdfs-dev 
<hdfs-...@hadoop.apache.org>; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.4. 
This is our next maint release to follow up 2.8.3. It includes 77 important 
fixes and improvements.

The RC artifacts are available at:
http://home.apache.org/~junping_du/hadoop-2.8.4-RC0

The RC tag in git is: release-2.8.4-RC0

The maven artifacts are available via repository.apache.org< 
http://repository.apache.org> at:
https://repository.apache.org/content/repositories/orgapachehadoop-1118

Please try the release and vote; the vote will run for the usual 5 working 
days, ending on 5/14/2018 PST time.

Thanks,

Junping

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

RE: Apache Hadoop 3.1.1 release plan

[Brahma Reddy Battula]

Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better addition 
to 3.1 line release for improving quality.

Looks only following two are pending which are in review state. Hope you are 
monitoring these two.

https://issues.apache.org/jira/browse/YARN-8265
https://issues.apache.org/jira/browse/YARN-8236



Note : https://issues.apache.org/jira/browse/YARN-8247==> committed branch-3.1


-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com] 
Sent: 19 April 2018 17:49
To: Hadoop Common ; 
mapreduce-...@hadoop.apache.org; Hdfs-dev ; 
yarn-dev@hadoop.apache.org
Subject: Apache Hadoop 3.1.1 release plan

Hi, All

We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the quality 
of the release, we plan to release 3.1.1 at May 06. The focus of 3.1.1 will be 
fixing blockers / critical bugs and other enhancements. So far there are 100 
JIRAs [1] have fix version marked to 3.1.1.

We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce") AND 
fixVersion = 3.1.1

RE: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Brahma Reddy Battula]

Konstantin thanks for driving this.

+1 (binding)


--Built from the source
--Installed HA cluster
-Verified the basic shell commands
-Ran sample jobs like pi,wordcount
-Browsed the UI's



-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: 10 April 2018 07:14
To: Hadoop Common ; hdfs-dev 
; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

Hi everybody,

This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.5 
was released on December 14, 2017.
Release 2.7.6 includes critical bug fixes and optimizations. See more details 
in Release Note:
http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days 
ending 04/16/2018.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Brahma Reddy Battula]

Wangda thanks for driving this.

+1(binding)

--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.


On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

RE: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Brahma Reddy Battula]

Thanks Lei Xu for driving this.

+1(binding)


--Built from the source
--Installed the HA Cluster
--Verified hdfs operations
--Ran sample jobs
--Verified the UI's



--Brahma Reddy Battula


-Original Message-
From: Lei Xu [mailto:l...@apache.org] 
Sent: 18 March 2018 12:11
To: Hadoop Common <common-...@hadoop.apache.org>; Hdfs-dev 
<hdfs-...@hadoop.apache.org>; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

Hi, all

I've created release candidate RC-1 for Apache Hadoop 3.0.1

Apache Hadoop 3.0.1 will be the first bug fix release for Apache Hadoop 3.0 
release. It includes 49 bug fixes and security fixes, which include 12 blockers 
and 17 are critical.

Please note:
* HDFS-12990. Change default NameNode RPC port back to 8020. It makes 
incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache Hadoop 
3.0.0 will be deprecated due to this change.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/

The git tag is release-3.0.1-RC1, and the latest commit is
496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1081/

Please try the release and vote; the vote will run for the usual 5 days, ending 
on 3/22/2017 6pm PST time.

Thanks!

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

RE: Can't pull the trunk code in windows- >File name too long

[Brahma Reddy Battula]

Hi,

Able to clone/pull after setting the following.

git config --system core.longpaths true


--Brahma Reddy Battula

-Original Message-
From: Vinayakumar B [mailto:vinayakum...@apache.org] 
Sent: 13 March 2018 14:30
To: Brahma Reddy Battula <brahmareddy.batt...@huawei.com>
Cc: Hadoop Common <common-...@hadoop.apache.org>; yarn-dev@hadoop.apache.org
Subject: Re: Can't pull the trunk code in windows- >File name too long

this is happening  in Windows, Please somebody from yarn look into this.

-Vinay

On 13 Mar 2018 11:30 am, "Brahma Reddy Battula" < 
brahmareddy.batt...@huawei.com> wrote:

Hi

Not able to pull/clone the trunk code. Does it possible make following shorten.


fatal: cannot create directory at 'hadoop-yarn-project/hadoop-
yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-
yarn-server-timelineservice-hbase-server/hadoop-yarn-
server-timelineservice-hbase-server-1/src/main/java/org/
apache/hadoop/yarn/server/timelineservice': Filename too long



--Brahma Reddy Battula

Can't pull the trunk code in windows- >File name too long

[Brahma Reddy Battula]

Hi

Not able to pull/clone the trunk code. Does it possible make following shorten.


fatal: cannot create directory at 
'hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1/src/main/java/org/apache/hadoop/yarn/server/timelineservice':
 Filename too long



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Brahma Reddy Battula]

Hi Lei Xu,

Thanks for driving this.

+1 binding

-Installed 3Node HA cluster
-Verified basic shell commands
-Ran word-count,pi jobs



Regards
Brahma Reddy Battula

On Fri, Feb 16, 2018 at 5:06 AM, Lei Xu <l...@apache.org> wrote:

> Hi, all
>
> I've created release candidate 0 for Apache Hadoop 3.0.1
>
> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> Hadoop 3.0 release. It includes 49 bug fixes, which include 10
> blockers and 8 are critical.
>
> Please note:
> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> Hadoop 3.0.0 will be deprecated due to this change.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC0/
>
> The git tag is release-3.0.1-RC0, and the latest commit is
> 494d075055b52b0cc922bc25237e231bb3771c90
>
> The maven artifacts are available:
> https://repository.apache.org/content/repositories/orgapachehadoop-1078/
>
> Please try the release and vote; the vote will run for the usual 5
> days, ending on 2/20/2017 6pm PST time.
>
> Thanks!
>
> --
> Lei (Eddy) Xu
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>


-- 



--Brahma Reddy Battula

Re: Apache Hadoop 3.0.1 Release plan

[Brahma Reddy Battula]

Hi Eddy,

HDFS-12990 got committed to 3.0.1,can we have RC for 3.0.1 (only  YARN-5742
blocker is open )  ?


On Sat, Feb 3, 2018 at 12:40 AM, Chris Douglas  wrote:

> On Fri, Feb 2, 2018 at 10:22 AM, Arpit Agarwal 
> wrote:
> > Do you plan to roll an RC with an uncommitted fix? That isn't the right
> approach.
>
> The fix will be committed to the release branch. We'll vote on the
> release, and if it receives a majority of +1 votes then it becomes
> 3.0.1. That's how the PMC decides how to move forward. In this case,
> that will also resolve whether or not it can be committed to trunk.
>
> If this logic is unpersuasive, then we can require a 2/3 majority to
> replace the codebase. Either way, the PMC will vote to define the
> consensus view when it is not emergent.
>
> > This issue has good visibility and enough discussion.
>
> Yes, it has. We always prefer consensus to voting, but when discussion
> reveals that complete consensus is impossible, we still need a way
> forward. This is rare, and usually reserved for significant changes
> (like merging YARN). Frankly, it's embarrassing to resort to it here,
> but here we are.
>
> > If there is a binding veto in effect then the change must be abandoned.
> Else you should be able to proceed with committing. However, 3.0.0 must be
> called out as an abandoned release if we commit it.
>
> This is not accurate. A binding veto from any committer halts
> progress, but the PMC sets the direction of the project. That includes
> making decisions that are not universally accepted. -C
>
> > On 2/1/18, 3:01 PM, "Lei Xu"  wrote:
> >
> > Sounds good to me, ATM.
> >
> > On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers 
> wrote:
> > > Hey Anu,
> > >
> > > My feeling on HDFS-12990 is that we've discussed it quite a bit
> already and
> > > it doesn't seem at this point like either side is going to budge.
> I'm
> > > certainly happy to have a phone call about it, but I don't expect
> that we'd
> > > make much progress.
> > >
> > > My suggestion is that we simply include the patch posted to
> HDFS-12990 in
> > > the 3.0.1 RC and call this issue out clearly in the subsequent
> VOTE thread
> > > for the 3.0.1 release. Eddy, are you up for that?
> > >
> > > Best,
> > > Aaron
> > >
> > > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu  wrote:
> > >>
> > >> +Xiao
> > >>
> > >> My understanding is that we will have this for 3.0.1.   Xiao,
> could
> > >> you give your inputs here?
> > >>
> > >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer <
> aengin...@hortonworks.com>
> > >> wrote:
> > >> > Hi Eddy,
> > >> >
> > >> > Thanks for driving this release. Just a quick question, do we
> have time
> > >> > to close this issue?
> > >> > https://issues.apache.org/jira/browse/HDFS-12990
> > >> >
> > >> > or are we abandoning it? I believe that this is the last window
> for us
> > >> > to fix this issue.
> > >> >
> > >> > Should we have a call and get this resolved one way or another?
> > >> >
> > >> > Thanks
> > >> > Anu
> > >> >
> > >> > On 2/1/18, 10:51 AM, "Lei Xu"  wrote:
> > >> >
> > >> > Hi, All
> > >> >
> > >> > I just cut branch-3.0.1 from branch-3.0.  Please make sure
> all
> > >> > patches
> > >> > targeted to 3.0.1 being checked in both branch-3.0 and
> branch-3.0.1.
> > >> >
> > >> > Thanks!
> > >> > Eddy
> > >> >
> > >> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu 
> wrote:
> > >> > > Hi, All
> > >> > >
> > >> > > We have released Apache Hadoop 3.0.0 in December [1]. To
> further
> > >> > > improve the quality of release, we plan to cut
> branch-3.0.1 branch
> > >> > > tomorrow for the preparation of Apache Hadoop 3.0.1
> release. The
> > >> > focus
> > >> > > of 3.0.1 will be fixing blockers (3), critical bugs (1)
> and bug
> > >> > fixes
> > >> > > [2].  No new features and improvement should be included.
> > >> > >
> > >> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote
> for RC on
> > >> > Feb
> > >> > > 1st, targeting for Feb 9th release.
> > >> > >
> > >> > > Please feel free to share your insights.
> > >> > >
> > >> > > [1]
> > >> > https://www.mail-archive.com/general@hadoop.apache.org/
> msg07757.html
> > >> > > [2] https://issues.apache.org/
> jira/issues/?filter=12342842
> > >> > >
> > >> > > Best,
> > >> > > --
> > >> > > Lei (Eddy) Xu
> > >> > > Software Engineer, Cloudera
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Lei (Eddy) Xu
> > >> > Software Engineer, Cloudera
> > >> >
> > >> >
> >   

RE: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Brahma Reddy Battula]

+1 (non-binding),Thanks Andrew wang for driving this.

---Built from the source
--Installed 3 Node HA cluster
--Verified Basic shell commands 
--Ran Sample jobs like pi,wordcount
--Verified the UI's



--Brahma Reddy Battula


-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: 09 December 2017 02:01
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 3.0.0 RC1

Hi all,

Let me start, as always, by thanking the efforts of all the contributors who 
contributed to this release, especially those who jumped on the issues found in 
RC0.

I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302 fixed 
JIRAs since the previous 3.0.0-beta1 release.

You can find the artifacts here:

http://home.apache.org/~wang/3.0.0-RC1/

I've done the traditional testing of building from the source tarball and 
running a Pi job on a single node cluster. I also verified that the shaded jars 
are not empty.

Found one issue that create-release (probably due to the mvn deploy change) 
didn't sign the artifacts, but I fixed that by calling mvn one more time.
Available here:

https://repository.apache.org/content/repositories/orgapachehadoop-1075/

This release will run the standard 5 days, closing on Dec 13th at 12:31pm 
Pacific. My +1 to start.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Brahma Reddy Battula]

+1 (non-binding), thanks Junping for driving this.


--Built from the source
--Installaed 3 Node HA cluster
--Verified Basic shell Commands
--Browsed the HDFS/YARN web UI
--Ran sample pi,wordcount jobs

--Brahma Reddy Battula


On Tue, Dec 5, 2017 at 3:28 PM, Junping Du <j...@hortonworks.com> wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79
> important fixes and improvements.
>
>   The RC artifacts are available at: http://home.apache.org/~
> junping_du/hadoop-2.8.3-RC0
>
>   The RC tag in git is: release-2.8.3-RC0
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1072
>
>   Please try the release and vote; the vote will run for the usual 5
> working days, ending on 12/12/2017 PST time.
>
> Thanks,
>
> Junping
>



-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Brahma Reddy Battula]

 +1  (non-binding)

 --Built from the source
 --Installed 3 Node HA Cluster
 --Ran basic shell commands
 --Verified append/snapshot/truncate
 --Ran sample jobs like pi,wordcount

 Thanks again Konstantin for driving this.


--Brahma Reddy Battula

On Fri, Dec 8, 2017 at 8:52 AM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>



-- 



--Brahma Reddy Battula

RE: VOTE] Release Apache Hadoop 2.7.5 (RC0)

[Brahma Reddy Battula]

+1  (non-binding), thanks Konstantin for driving this.


--Built from the source
--Installed 3 Node HA Cluster
--Ran basic shell commands
--Verified append/snapshot/truncate
--Ran sample jobs like pi,wordcount


Looks follow commits are missed in changes.txt.

MAPREDUCE-6975
HADOOP-14919
HDFS-12596
YARN-7084
HADOOP-14881
HADOOP-14827
HDFS-12832


--Brahma Reddy Battula

-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: 02 December 2017 10:13
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: VOTE] Release Apache Hadoop 2.7.5 (RC0)

Hi everybody,

This is the next dot release of Apache Hadoop 2.7 line. The previous one
2.7.4 was release August 4, 2017.
Release 2.7.5 includes critical bug fixes and optimizations. See more details 
in Release Note:
http://home.apache.org/~shv/hadoop-2.7.5-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days 
ending 12/08/2017.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Brahma Reddy Battula]

bq. (a) The behavior of this command. Clearly, it will conflict with the
MapReduce JHS - only one of them can be started on the same node.

Yes,Since the PID  file will be same.

This problem will not present in branch-2.9 + as pid file will be suffix
with mapred(for JHS)
<https://github.com/apache/hadoop/blob/branch-2.9/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh#L93>and
yarn (for AHS)
<https://github.com/apache/hadoop/blob/branch-2.9/hadoop-yarn-project/hadoop-yarn/bin/yarn-daemon.sh#L95>..i.e
PID is exported on each daemon script.

May be,
1) Change the name
2) Create PID based on the CLASS Name, here applicationhistoryserver and
jobhistoryserver
3) Use same as branch-2.9..i.e suffixing with mapred or yarn


@allen, any thoughts on this..?




On Wed, Nov 22, 2017 at 2:46 AM, Vinod Kumar Vavilapalli <vino...@apache.org
> wrote:

> >> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
> even work. Not just deprecated in favor of timelineserver as was advertised.
> >
> >   This works for me in trunk and the bash code doesn’t appear to
> have changed in a very long time.  Probably something local to your
> install.  (I do notice that the deprecation message says “starting” which
> is awkward when the stop command is given though.)  Also: is the
> deprecation message even true at this point?
>
>
> Sorry, I mischaracterized the problem.
>
> The real issue is that I cannot use this command line when the MapReduce
> JobHistoryServer is already started on the same machine.
>
> ~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
> WARNING: Use of this script to start YARN daemons is deprecated.
> WARNING: Attempting to execute replacement "yarn --daemon start" instead.
> DEPRECATED: Use of this command to start the timeline server is deprecated.
> Instead use the timelineserver command for it.
> Starting the History Server anyway...
> historyserver is running as process 86156.  Stop it first.
>
> So, it looks like in shell-scripts, there can ever be only one daemon of a
> given name, irrespective of which daemon scripts are invoked.
>
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the
> MapReduce JHS - only one of them can be started on the same node.
>  (b) We need to figure out if this V1 TimelineService should even be
> support given ATSv2.
>
> @Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please
> comment on (b).
>
> Thanks
> +Vinod




-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Brahma Reddy Battula]

+1 (non-binding).

-Built from the source
-Installed the HA Cluster
-Ran basic shell commands
-Ran sample jobs like pi,Slive

Thanks Andrew for driving this.

On Wed, Nov 15, 2017 at 3:04 AM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>



-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Brahma Reddy Battula]

+1 ( non-binding)

-Built from the source
-Installed 3 Node HA cluster and a pseudo cluster
-Verified through hdfs shell commands
-Verified HDFS router federation
-Ran sample jobs like pi,Slive



--Brahma Reddy


On Tue, Nov 14, 2017 at 5:40 AM, Arun Suresh <asur...@apache.org> wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
> <https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9>*
>
> New RC is available at: *https://home.apache.org/~
> asuresh/hadoop-2.9.0-RC3/
> <https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/>*
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
> <https://www.google.com/url?q=https%3A%2F%2Frepository.
> apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D&
> sntz=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>*https://
> repository.apache.org/content/repositories/orgapachehadoop-1068/
> <https://repository.apache.org/content/repositories/orgapachehadoop-1068/
> >*
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru
>



-- 



--Brahma Reddy Battula

RE: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Brahma Reddy Battula]

Hi Junping,

Thanks for driving the 2.8.2 release.

+1 (non-binding)

--Built from source and installed 3 Node HA Cluster
--Ran basic hdfs/webhdfs commands
--Ran pi,wordcount jobs
--Verified the UI's




--Brahma Reddy Battula

-Original Message-
From: Junping Du [mailto:j...@hortonworks.com] 
Sent: 20 October 2017 06:13
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1<http://home.apache.org/~junping_du/hadoop-2.8.2-RC0>

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org<http://repository.apache.org/> at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064<https://repository.apache.org/content/repositories/orgapachehadoop-1062>

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-beta1 RC0

[Brahma Reddy Battula]

Thanks Andrew.

+1 (non binding)

--Built from source
--installed 3 node HA cluster
--Verified shell commands and UI
--Ran wordcount/pic jobs




On Fri, 29 Sep 2017 at 5:34 AM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi all,
>
> Let me start, as always, by thanking the many, many contributors who helped
> with this release! I've prepared an RC0 for 3.0.0-beta1:
>
> http://home.apache.org/~wang/3.0.0-beta1-RC0/
>
> This vote will run five days, ending on Nov 3rd at 5PM Pacific.
>
> beta1 contains 576 fixed JIRA issues comprising a number of bug fixes,
> improvements, and feature enhancements. Notable additions include the
> addition of YARN Timeline Service v2 alpha2, S3Guard, completion of the
> shaded client, and HDFS erasure coding pluggable policy support.
>
> I've done the traditional testing of running a Pi job on a pseudo cluster.
> My +1 to start.
>
> We're working internally on getting this run through our integration test
> rig. I'm hoping Vijay or Ray can ring in with a +1 once that's complete.
>
> Best,
> Andrew
>
-- 



--Brahma Reddy Battula

qbt is failiing///RE: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Brahma Reddy Battula]

qbt is failing from two days with following errors, any idea on this..?




cd 
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
/opt/maven/bin/mvn 
-Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-full-1 -Ptest-patch 
-Pparallel-tests -Pshelltest -Pnative -Drequire.snappy -Drequire.openssl 
-Drequire.fuse -Drequire.test.libhadoop -Pyarn-ui clean test -fae > 
/testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 2>&1
FATAL: command execution failed
Command close created at
at hudson.remoting.Command.(Command.java:60)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
at hudson.remoting.Channel.close(Channel.java:1281)
at hudson.remoting.Channel.close(Channel.java:1263)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
Caused: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
at hudson.remoting.Channel$1.handle(Channel.java:527)
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused: java.io.IOException: Backing channel 'H10' is disconnected.
at 
hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
at 
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy125.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
at hudson.model.Run.execute(Run.java:1735)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Build step 'Execute shell' marked build as failure
ERROR: Step ?Publish Checkstyle analysis results? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Step ?Publish FindBugs analysis results? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Step ?Archive the artifacts? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Step ?Publish JUnit test result report? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Build step failed with exception
java.lang.NullPointerException
at 
hudson.plugins.violations.ViolationsPublisher.perform(ViolationsPublisher.java:74)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)
at hudson.model.Run.execute(Run.java:1760)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Build step 'Report Violations' marked build as failure

--Brahma Reddy Battula

-Original Message-
From: Apache Jenkins Server [mailto:jenk...@builds.apache.org] 
Sent: 19 September 2017 15:07
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/527/

[Sep 18, 2017 10:07:12 AM] (kai.zheng) HDFS-12460. Make addErasureCodingPolicy 
an idempotent operation.
[Sep 18, 2017 3:16:09 PM] (jlowe) YARN-7192. Add a pluggable StateMachine 
Listener that is notified of NM [Sep 18, 2017 4:53:24 PM] (arp) HDFS-12470. 
DiskBalancer: Some tests create plan files under system [Sep 18, 2017 5:32:08 
PM] (rkanter) Revert "YARN-7162. Remove XML excludes file format (rkanter)" - 
wrong [Sep 18, 2017 5:40:06 PM] (rkanter) MAPREDUCE-6954. Disable erasure 
coding for files that are

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Brahma Reddy Battula]

Thanks Junping Du.

+1 (non-binding)

-Built from source
-Installed HA cluster
-Executed basic shell commands
-Ran sample jobs
-Verified HDFS/YARN UI.



--Brahma Reddy Battula


On Mon, 11 Sep 2017 at 5:30 AM, Junping Du <j...@hortonworks.com> wrote:

> Hi folks,
>  With fix of HADOOP-14842 get in, I've created our first release
> candidate (RC0) for Apache Hadoop 2.8.2.
>
>  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line
> and will be the latest stable/production release for Apache Hadoop - it
> includes 305 new fixed issues since 2.8.1 and 63 fixes are marked as
> blocker/critical issues.
>
>   More information about the 2.8.2 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at:
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>
>   The RC tag in git is: release-2.8.2-RC0, and the latest commit id
> is: e6597fe3000b06847d2bf55f2bab81770f4b2505
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 09/15/2017 5pm PST time.
>
> Thanks,
>
> Junping
>
> --



--Brahma Reddy Battula

Re: Apache Hadoop 2.8.2 Release Plan

[Brahma Reddy Battula]

Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
<https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21>2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du<mailto:j...@hortonworks.com>,Please correct me,if I am wrong.


--Brahma Reddy Battula

From: Junping Du <j...@hortonworks.com>
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA<https://s.apache.org/JM5x>
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du <j...@hortonworks.com>
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If no other concerns, I will do the branch update in next 30 minutes.


Thanks,


Junping


____________
From: Brahma Reddy Battula <brahmareddy.batt...@hotmail.com>
Sent: Sunday, July 23, 2017 1:50 AM
To: Junping Du; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan



Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.


From: Junping Du <j...@hortonworks.com>
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du <j...@hortonworks.com>
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hado

RE: Branch merges and 3.0.0-beta1 scope

[Brahma Reddy Battula]

IMHO, when we propose any feature branch merge, we might need to consider 
following aspects

1)Use cases
2) Pluggable ---> if it's not pluggable , give the reason
3) API Compatibility
4) Impact---> when it's enable
5) Performance
6) Stability
7) Test Sufficiency
8) Documentation

Above points helps to validate the feature quality at first glance and based on 
that further consideration to merge can be done.


--Brahma Reddy Battula

-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com] 
Sent: 22 August 2017 06:27
To: Vinod Kumar Vavilapalli
Cc: Steve Loughran; Andrew Wang; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: Branch merges and 3.0.0-beta1 scope

Andrew,

Thanks for your help to pushing this release.

Echoing what Vinod said, all contributors in these branches are putting months 
to years of time working on these features, we don't have to decide excluded 
features now since we have 25 days till 3.0-beta1 planned release time.

The best approach to stabilize feature is to let people try that, instead of 
waiting for feature becomes perfect. For features which can be turned off, I 
think we should consider to bring it in if it is end-to-end ready. I will try 
best to help merge efforts of YARN-3926 branch to trunk before Sep 15, and I'm 
OK with moving to the next release train if we fail to merge the feature before 
release date.

Thanks,
Wangda


On Mon, Aug 21, 2017 at 2:22 PM, Vinod Kumar Vavilapalli <vino...@apache.org
> wrote:

> Steve,
>
> You can be strict & ruthless about the timelines. Anything that 
> doesn’t get in by mid-September, as was originally planned, can move 
> to the next release - whether it is feature work on branches or feature work 
> on trunk.
>
> The problem I see here is that code & branches being worked on for a 
> year are now (apparently) close to being done and we are telling them 
> to hold for 7 more months - this is not a reasonable ask..
>
> If you are advocating for a 3.1 plan, I’m sure one of these branch 
> ‘owners’ can volunteer. But this is how you get competing releases and 
> split bandwidth.
>
> As for compatibility / testing etc, it seems like there is a belief 
> that the current ‘scoped’ features are all tested well in these areas 
> and so adding more is going to hurt the release. There is no way this 
> is the reality, trunk has so many features that have been landing for 
> years, the only way we can collectively attempt towards making this 
> stable is by getting as many parties together as possible, each 
> verifying stuff that they need. Not by excluding specific features.
>
> +Vinod
>
> > This is one of those curse-of-cadence things: The higher your 
> > release
> cadence, the less pressure to get "everything in". With a slower 
> cadence, more pressure to get stuff in, more pressure to hold up the 
> release, slows the cadence, gets even more stuff in, etc. etc.
> >
> > - Andrew has been working on the release for months, we all need to
> appreciate how much hard work that is and has been, especially for 
> what is going to be a major release.
> >
> > - We know that things will be unstable in 3.0; Andrew's concern is 
> > about
> making sure that the newest, unstablest (?) features can at least be 
> bypassed if there are problems. I we should also call out in the 
> release notes what we think are the unstable bits where people need to 
> use caution
> (example: S3Guard in "authoritative" mode)
> >
> > - Anything related to wire compatibility has been problematic in the
> past; I think it's essential that whatever packets get sent around are 
> going to be stable, so changes there need to be in, or at least the 
> payloads set up ready for the features. Same for new public APIs.
> >
> > - As fpr the rest, I don't know. I think being strict about it and
> ruthless in assessing the feature's stability & consequences of 
> postponing the feature until a Hadoop 3.1 release in Jan/Feb, with a 
> plan to ship then and follow up with a 3.2 in the summer.
> >
> > Then: start planning that 3.1 release. Maybe I should put my hand up 
> > as
> release manager for that one. Then everyone would realise how amenable 
> Andrew is being today.
> >
> >
> > One other thing: alongside the big branches, there's the eternal 
> > backlog
> of small patches. We should organise spending a few days updating, 
> reviewing & merging them in
> >
> > -Steve
> >
> >
> > 
> > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> >  hdfs-dev-unsubscr...@hadoop.apache.org>
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> <mailto:hdfs-dev-h...@hadoop.apache.org>
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

RE: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Brahma Reddy Battula]

Hi Konstantin,

Thanks a lot again for your efforts.

+1  (non-binding)

-Built from the source on Suse-Linux with jdk_ 1.8.0_40
-Installed the HA cluster
-Verified basic shell commands
-Ran sample jobs
-Did the regression on IBR Feature, Balancer/mover,fsck

Downloaded the latest tarball, it contains the native and nodemanger can start 
(NM will not start without native). And install pseudo cluster and did basic 
verification.

IMHO, we should include natives in the tarball for user convenience (who 
doesn't have build tool can use for quick regression) 

As we are separately giving hadoop-2.7.4-RC0-src.tar.gz, still we need to 
include /src in tarball..?

As Andrew mentioned, these two are regression to prior release.


--Brahma Reddy Battula

-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: 01 August 2017 09:57
To: Chris Douglas
Cc: Andrew Wang; Allen Wittenauer; common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

Uploaded new binaries hadoop-2.7.4-RC0.tar.gz, which adds lib/native/.
Same place: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

Thanks,
--Konstantin

On Mon, Jul 31, 2017 at 3:56 PM, Chris Douglas <cdoug...@apache.org> wrote:

> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko 
> <shv.had...@gmail.com> wrote:
> > For the packaging, here is the exact phrasing from the sited
> release-policy
> > document relevant to binaries:
> > "As a convenience to users that might not have the appropriate tools 
> > to build a compiled version of the source, binary/bytecode packages 
> > MAY be distributed alongside official Apache releases. In all such 
> > cases, the binary/bytecode package MUST have the same version number 
> > as the source release and MUST only add binary/bytecode files that 
> > are the result of compiling that version of the source code release and its 
> > dependencies."
> > I don't think my binary package violates any of these.
>
> +1 The PMC VOTE applies to source code, only. If someone wants to
> rebuild the binary tarball with native libs and replace this one, 
> that's fine.
>
> My reading of the above is that source code must be distributed with 
> binaries, not that we omit the source code from binary releases... -C
>
> > But I'll upload an additional tar.gz with native bits and no src, as 
> > you guys requested.
> > Will keep it as RC0 as there is no source code change and it comes 
> > from
> the
> > same build.
> > Hope this is satisfactory.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang 
> > <andrew.w...@cloudera.com>
> > wrote:
> >
> >> I agree with Brahma on the two issues flagged (having src in the 
> >> binary tarball, missing native libs). These are regressions from 
> >> prior
> releases.
> >>
> >> As an aside, "we release binaries as a convenience" doesn't relax 
> >> the quality bar. The binaries are linked on our website and 
> >> distributed
> through
> >> official Apache channels. They have to adhere to Apache release 
> >> requirements. And, most users consume our work via Maven 
> >> dependencies, which are binary artifacts.
> >>
> >> http://www.apache.org/legal/release-policy.html goes into this in 
> >> more detail. A release must minimally include source packages, and 
> >> can also include binary artifacts.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko < 
> >> shv.had...@gmail.com> wrote:
> >>
> >>> To avoid any confusion in this regard. I built RC0 manually in
> compliance
> >>> with Apache release policy
> >>> http://www.apache.org/legal/release-policy.html
> >>> I edited the HowToReleasePreDSBCR page to make sure people don't 
> >>> use Jenkins option for building.
> >>>
> >>> A side note. This particular build is broken anyways, so no 
> >>> worries
> there.
> >>> I think though it would be useful to have it working for testing 
> >>> and
> as a
> >>> packaging standard.
> >>>
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer < 
> >>> a...@effectivemachines.com
> >>> > wrote:
> >>>
> >>> >
> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> >>> shv.had...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> >>> >
> >>> > FYI:
> >>> >
> >>> > If you are using ASF Jenkins to create an ASF 
> >>> > release artifact, it's pretty much an automatic vote failure as 
> >>> > any such
> >>> release is
> >>> > in violation of ASF policy.
> >>> >
> >>> >
> >>>
> >>
> >>
>

RE: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Brahma Reddy Battula]

Thanks konstantin.

Native files (lib/native) is missed
And src folder exists in tar ball.

Please check the same.

Host1:/home/Brahma/Release/hadoop-2.7.4 # ll
total 144
-rw-r--r--  1 20415 messagebus 86424 Jul 30 02:54 LICENSE.txt
-rw-r--r--  1 20415 messagebus 14978 Jul 30 02:54 NOTICE.txt
-rw-r--r--  1 20415 messagebus  1366 Jul 30 02:54 README.txt
drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 bin
drwxr-xr-x  3 20415 messagebus  4096 Jul 30 02:54 etc
-rw-r--r--  1 20415 messagebus  1683 Jul 30 03:15 hadoop-client.list
drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 include
drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 libexec
drwxr-xr-x  2 20415 messagebus  4096 Jul 30 02:54 sbin
drwxr-xr-x  4 20415 messagebus  4096 Jul 30 02:54 share
drwxr-xr-x 19 20415 messagebus  4096 Jul 30 03:01 src



--Brahma Reddy Battula

-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: 30 July 2017 07:29
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

Hi everybody,

Here is the next release of Apache Hadoop 2.7 line. The previous stable release 
2.7.3 was available since 25 August, 2016.
Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are critical 
bug fixes and major optimizations. See more details in Release
Note:
http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days 
ending 08/04/2017.

Please note that my up to date public key are available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Please don't forget to refresh the page if you've been there recently.
There are other place on Apache sites, which may contain my outdated key.

Thanks,
--Konstantin

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Brahma Reddy Battula]

Just executed the "git log branch-2.8 ^branch-2.8.2" found two commits are 
missed (HDFS-8312 and HADOOP-13867 ) in branch-2.8.I just pushed this two 
commits.Hope we'll not miss any commits which present in only in branch-2.8.2.


From: Junping Du 
Sent: Saturday, July 22, 2017 5:57 AM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du 
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du  wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe 
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
>  wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
Hadoop 2.8 Release - Hadoop - Apache Software 

RE: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0

[Brahma Reddy Battula]

+1 (non-binding)

-Verified checksums and signatures
-Built from source and installed HA cluster
-Ran basic shell operations
-Ran sample jobs.
-Verified the HttpFSServerWebServer.





-Brahma Reddy Battula


-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: 30 June 2017 10:41
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0

Hi all,

As always, thanks to the many, many contributors who helped with this release! 
I've prepared an RC0 for 3.0.0-alpha4:

http://home.apache.org/~wang/3.0.0-alpha4-RC0/

The standard 5-day vote would run until midnight on Tuesday, July 4th.
Given that July 4th is a holiday in the US, I expect this vote might have to be 
extended, but I'd like to close the vote relatively soon after.

I've done my traditional testing of a pseudo-distributed cluster with a single 
task pi job, which was successful.

Normally my testing would end there, but I'm slightly more confident this time. 
At Cloudera, we've successfully packaged and deployed a snapshot from a few 
days ago, and run basic smoke tests. Some bugs found from this include 
HDFS-11956, which fixes backwards compat with Hadoop 2 clients, and the revert 
of HDFS-11696, which broke NN QJM HA setup.

Vijay is working on a test run with a fuller test suite (the results of which 
we can hopefully post soon).

My +1 to start,

Best,
Andrew

Re: About 2.7.4 Release

[Brahma Reddy Battula]

Hi Konstantin Shvachko


how about creating a wiki page for 2.7.4 release status like 2.8 and trunk in 
following link.??


https://cwiki.apache.org/confluence/display/HADOOP



From: Konstantin Shvachko <shv.had...@gmail.com>
Sent: Saturday, May 13, 2017 3:58 AM
To: Akira Ajisaka
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey Akira,
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 <https://issues.apache.org/jira/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that filter. So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> wrote:
>
>> Hi Konstantin,
>>
>> Thank you for volunteering as release manager!
>>
>> > Actually the original link works fine: https://s.apache.org/Dzg4
>> I couldn't see the link. Maybe is it private filter?
>>
>> Here is a link I generated: https://s.apache.org/ehKy
>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>
>> Thanks and Regards,
>> Akira
>>
>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>
>>> Hi Brahma Reddy Battula,
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>> Your link excludes closed and resolved issues, which needs backporting,
>>> and
>>> which we cannot reopen, as discussed in this thread earlier.
>>>
>>> Looked through the issues you proposed:
>>>
>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>> Seems like a new feature. It helps failover to standby node when primary
>>> is
>>> under heavy load, but it introduces new APIs, addresses, config
>>> parameters.
>>> And needs at least one follow up jira.
>>> Looks like a backward compatible change, though.
>>> Did you have a chance to run it in production?
>>>
>>> +1 on
>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
[HDFS-10987] Make Decommission less expensive when lot of 
...<https://issues.apache.org/jira/browse/HDFS-10987>
issues.apache.org
When user want to decommission a node which having 50M blocks ,it could hold 
the namesystem lock for long time.We've seen it is taking 36 sec. As we knew 
during this ...



>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
[HDFS-9902] Support different values of dfs.datanode.du 
...<https://issues.apache.org/jira/browse/HDFS-9902>
issues.apache.org
Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, 
but they share one configuration dfs.datanode.du.reserved. The DISK size may be 
several ...



>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
Trash does not descent into child directories to check for 
...<https://issues.apache.org/jira/browse/HDFS-8312>
issues.apache.org
HDFS trash does not descent into child directory to check if user has 
permission to delete files. For example: Run the following command to 
initialize directory ...



>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
Upgrade Jsch jar to latest version to fix vulnerability in 
...<https://issues.apache.org/jira/browse/HADOOP-14100>
issues.apache.org
Recently there was on vulnerability reported on jsch library. Its fixed in 
latest 0.1.54 version before CVE was made public. 
https://cve.mitre.org/cgi-bin/cvename.cgi ...



>>>
>>> Ad