[jira] [Created] (HADOOP-19148) Update solr from 8.11.2 to 8.11.3 to address CVE-2023-50298

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-19148:
-

 Summary: Update solr from 8.11.2 to 8.11.3 to address 
CVE-2023-50298
 Key: HADOOP-19148
 URL: https://issues.apache.org/jira/browse/HADOOP-19148
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Reporter: Brahma Reddy Battula


Update solr from 8.11.2 to 8.11.3 to address CVE-2023-50298



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Query : Hadoop Cluster OS upgrade

[Brahma Reddy Battula]

Hi All,


Does anybody tried out/share learnings ,using maintenance state or  upgrade
domains for big data cluster OS upgrades?



Regards,
Brahma

Re: [INFO] Hadoop-3.4 Release Update

[Brahma Reddy Battula]

Thanks for driving this.

One query:
Does all the jiras from 3.3 are part of the 3.4?




On Mon, 22 Jan 2024 at 11:50 AM, slfan1989  wrote:

> Thank you very much for specifying the frozen time for the
> Branch-3.4.0.  @Xiaoqiao
> He 
>
> If there is a need to backport any PRs to branch-3.4/branch-3.4.0, please
> feel free to contact me.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He  wrote:
>
> > Hi All,
> >
> > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
> > If there is any blocker/critical PR please backport to branch-3.4 and
> > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
> >
> > If any reported issues are not ready in time please wait for the next
> > release.
> >
> > Thanks all.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >
> >
> > On Mon, Jan 22, 2024 at 10:43 AM slfan1989  wrote:
> >
> >> Thank you for the feedback!
> >>
> >> I apologize for updating partial information in the Hadoop module.
> >>
> >> Today, I will complete the rollback of the modified JIRA and add back
> the
> >> 3.4.0 version.
> >>
> >> Best
> >>
> >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
> >> wrote:
> >>
> >>> Thanks all for your input. Connected to Shilun offline yesterday, and
> he
> >>> will
> >>> update or recover the fix version tag before 3.4.0-RC1.
> >>>
> >>> Best Regards,
> >>> - He Xiaoqiao
> >>>
> >>>
> >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki  >
> >>> wrote:
> >>>
> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> >>> still
> >>> > be
> >>> > > included in the Fix versions,
> >>> > > even if it was already released in 3.3.x. If I understand
> correctly,
> >>> this
> >>> > > is usually our practice.
> >>> >
> >>> > +1.
> >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
> >>> > maintaining both branch-3.4 and branch-3.3.
> >>> >
> >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma <
> tasan...@apache.org>
> >>> > wrote:
> >>> > >
> >>> > > Hi Shilun,
> >>> > >
> >>> > > Thank you for leading the 3.4.0 release.
> >>> > >
> >>> > > > 2. In JIRA issues where other release versions have been
> released,
> >>> > remove
> >>> > > > the 3.4.0 entry from the fix version.
> >>> > >
> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> >>> still
> >>> > be
> >>> > > included in the Fix versions,
> >>> > > even if it was already released in 3.3.x. If I understand
> correctly,
> >>> this
> >>> > > is usually our practice.
> >>> > >
> >>> > > Thanks,
> >>> > > - Takanobu
> >>> > >
> >>> > > 2024年1月21日(日) 13:56 slfan1989 :
> >>> > >
> >>> > > > Hi All,
> >>> > > >
> >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
> >>> > perform
> >>> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN,
> >>> and
> >>> > > > MAPREDUCE. Specifically, the tasks include:
> >>> > > >
> >>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as
> >>> > > > non-blocker, update the target version to 3.5.0.
> >>> > > >
> >>> > > > 2. In JIRA issues where other release versions have been
> released,
> >>> > remove
> >>> > > > the 3.4.0 entry from the fix version.
> >>> > > >
> >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version
> >>> set to
> >>> > > > 3.4.0, and status set to
> >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are
> >>> not
> >>> > set,
> >>> > > > complete the information.
> >>> > > >
> >>> > > > Best Regards,
> >>> > > > - Shilun Fan.
> >>> > > >
> >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He  >
> >>> > wrote:
> >>> > > >
> >>> > > > > Hi All,
> >>> > > > >
> >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
> >>> > created[1].
> >>> > > > >
> >>> > > > > Please set the proper fix version while committing jira, 3.5.0
> >>> should
> >>> > > > > be priority. If there is any blocker/critical please also
> >>> backport to
> >>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
> >>> > > > >
> >>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
> >>> > > > >
> >>> > > > > Thanks.
> >>> > > > >
> >>> > > > > Best Regards,
> >>> > > > > - He Xiaoqiao
> >>> > > > >
> >>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
> >>> > > > >
> >>> > > >
> >>> >
> >>> > -
> >>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
> >>> > For additional commands, e-mail: private-h...@hadoop.apache.org
> >>> >
> >>> >
> >>>
> >>
>

Re: Resources for understanding Hadoop

[Brahma Reddy Battula]

Please go through the following

https://github.com/apache/hadoop/blob/trunk/BUILDING.txt

and a specific command to generate the distribution which can be run after
your changes.
mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Hope this helps.




On Thu, Jun 23, 2022 at 9:41 PM Rahul Bhardwaj 
wrote:

> I am following this wiki
> <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html>
>  to
> build and run hadoop locally in pseudo-dsitributed mode. But I am unable to
> figure out how to build my changes and generate similar binaries so that I
> can test my changes locally. Is there some documentation on how to do this?
>
> On Mon, 13 Jun 2022 at 00:26, Brahma Reddy Battula 
> wrote:
>
>> Hi Rahul,
>>
>> Welcome to hadoop world.
>>
>> Apart from the gautham mentioned, you can check the following also.
>> https://livebook.manning.com/book/hadoop-in-action/part-1/
>>
>> Go through the following wiki for contributions
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
>>
>>
>> Please subscribe to the hadoop mailing list[1], and shoot your queries
>> there from next time.
>>
>>
>>
>>
>>
>> 1. https://hadoop.apache.org/mailing_lists.html
>>
>> On Sun, Jun 12, 2022 at 10:42 PM Gautham Banasandra 
>> wrote:
>>
>>> Hi Rahul,
>>>
>>> I was looking for something more detailed and low-level like how the code
>>> > for the various services in HDFS is organized, entrypoints etc.
>>>
>>> I found this book useful to get a good idea of Hadoop in general - Apache
>>> Hadoop™ YARN: Moving beyond MapReduce and Batch Processing with Apache
>>> Hadoop™ 2 [Book] (oreilly.com)
>>> <
>>> https://www.oreilly.com/library/view/apache-hadooptm-yarn/9780133441925/
>>> >.
>>>
>>> In my opinion, you get into Open Source contributions by just doing so.
>>> You
>>> don't have to know HDFS in detail to start contributing to it. Now that
>>> you've gone through the Hadoop documentation, try setting up Hadoop in
>>> pseudo-distributed mode. If you notice any glitch, try fixing it and send
>>> out a PR. You never know what issue you'll find. I ran into this when I
>>> tried compiling Hadoop on Windows - [HDFS-15385] Upgrade boost library to
>>> 1.72 - ASF JIRA (apache.org)
>>> <https://issues.apache.org/jira/browse/HDFS-15385> (And yes, this was my
>>> first PR to Hadoop). Then use Docker and set up the Hadoop cluster with
>>> multiple nodes. Once you're able to do this, try browsing
>>> issues.apache.org
>>> and you'll find tons of issues that you can work on. There's always so
>>> much
>>> work to do in Open Source and the thing that I like the most is that
>>> "there's no deadline on anything" :) So, you can really work on some
>>> awesome stuff, own it, perfect it and share it with the world.
>>>
>>> Best of luck.
>>>
>>> Thanks,
>>> --Gautham
>>>
>>> On Sun, 12 Jun 2022 at 16:34, Rahul Bhardwaj 
>>> wrote:
>>>
>>> > Hi all,
>>> > I am a newbie wanting to start contributing to the hadoop ecosystem. I
>>> want
>>> > to start by contributing to HDFS and was looking for resources to
>>> > understand the architecture and I just found this -
>>> >
>>> >
>>> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
>>> > which is a fairly high level documentation. I was looking for something
>>> > more detailed and low-level like how the code for the various services
>>> in
>>> > HDFS is organized, entrypoints etc. Can someone point me to such
>>> resources?
>>> > Also is there a slack workspace for such discussions? Not sure if this
>>> > mailing list is the right forum for such doubts.
>>> >
>>>
>>
>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>

-- 



--Brahma Reddy Battula

Re: Resources for understanding Hadoop

[Brahma Reddy Battula]

Hi Rahul,

Welcome to hadoop world.

Apart from the gautham mentioned, you can check the following also.
https://livebook.manning.com/book/hadoop-in-action/part-1/

Go through the following wiki for contributions
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute


Please subscribe to the hadoop mailing list[1], and shoot your queries
there from next time.





1. https://hadoop.apache.org/mailing_lists.html

On Sun, Jun 12, 2022 at 10:42 PM Gautham Banasandra 
wrote:

> Hi Rahul,
>
> I was looking for something more detailed and low-level like how the code
> > for the various services in HDFS is organized, entrypoints etc.
>
> I found this book useful to get a good idea of Hadoop in general - Apache
> Hadoop™ YARN: Moving beyond MapReduce and Batch Processing with Apache
> Hadoop™ 2 [Book] (oreilly.com)
> <https://www.oreilly.com/library/view/apache-hadooptm-yarn/9780133441925/
> >.
>
> In my opinion, you get into Open Source contributions by just doing so. You
> don't have to know HDFS in detail to start contributing to it. Now that
> you've gone through the Hadoop documentation, try setting up Hadoop in
> pseudo-distributed mode. If you notice any glitch, try fixing it and send
> out a PR. You never know what issue you'll find. I ran into this when I
> tried compiling Hadoop on Windows - [HDFS-15385] Upgrade boost library to
> 1.72 - ASF JIRA (apache.org)
> <https://issues.apache.org/jira/browse/HDFS-15385> (And yes, this was my
> first PR to Hadoop). Then use Docker and set up the Hadoop cluster with
> multiple nodes. Once you're able to do this, try browsing
> issues.apache.org
> and you'll find tons of issues that you can work on. There's always so much
> work to do in Open Source and the thing that I like the most is that
> "there's no deadline on anything" :) So, you can really work on some
> awesome stuff, own it, perfect it and share it with the world.
>
> Best of luck.
>
> Thanks,
> --Gautham
>
> On Sun, 12 Jun 2022 at 16:34, Rahul Bhardwaj 
> wrote:
>
> > Hi all,
> > I am a newbie wanting to start contributing to the hadoop ecosystem. I
> want
> > to start by contributing to HDFS and was looking for resources to
> > understand the architecture and I just found this -
> >
> >
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
> > which is a fairly high level documentation. I was looking for something
> > more detailed and low-level like how the code for the various services in
> > HDFS is organized, entrypoints etc. Can someone point me to such
> resources?
> > Also is there a slack workspace for such discussions? Not sure if this
> > mailing list is the right forum for such doubts.
> >
>


-- 



--Brahma Reddy Battula

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Sorry, I was looking for some CVE’s fixes.will make it progress on this…


On Tue, 11 Jan 2022 at 5:58 PM, Wei-Chiu Chuang  wrote:

> Is this still making progress?
>
> On Tue, Oct 5, 2021 at 8:45 PM Brahma Reddy Battula 
> wrote:
>
> > Hi Akira,
> >
> > Thanks for your email!!
> >
> > I am evaluating the CVE’s which needs to go for this release..
> >
> > Will update soon!!
> >
> >
> > On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka 
> wrote:
> >
> > > Hi Brahma,
> > >
> > > What is the release process going on? Is there any blocker for the RC?
> > >
> > > -Akira
> > >
> > > On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He 
> wrote:
> > >
> > > > Hi Brahma,
> > > >
> > > > The feature 'BPServiceActor processes commands from NameNode
> > > > asynchronously' has been ready for both branch-3.2 and branch-3.2.3.
> > > While
> > > > cherry-picking there is only minor conflict, So I checked in
> directly.
> > > BTW,
> > > > run some unit tests and build pseudo cluster to verify, it seems to
> > work
> > > > fine.
> > > > FYI.
> > > >
> > > > Regards,
> > > > - He Xiaoqiao
> > > >
> > > > On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Please go ahead. Let me know any help required on review.
> > > >>
> > > >> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He 
> > > wrote:
> > > >>
> > > >>> Hi Brahma,
> > > >>>
> > > >>> I plan to involve HDFS-14997 and related JIRAs if possible. I have
> > > >>> resolved the conflict and verified them locally.
> > > >>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
> > > >>> I would like to hear some more response that if we have enough time
> > to
> > > >>> wait for it to be ready.
> > > >>> Thanks.
> > > >>>
> > > >>> Best Regards,
> > > >>> - He Xiaoqiao
> > > >>>
> > > >>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He 
> > > wrote:
> > > >>>
> > > >>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3.
> FYI.
> > > >>>>
> > > >>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula <
> > > bra...@apache.org>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi All,
> > > >>>>>
> > > >>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly
> > > this
> > > >>>>> can
> > > >>>>> be done by this week,then I will try to create the RC next if
> there
> > > is
> > > >>>>> no
> > > >>>>> objection.
> > > >>>>>
> > > >>>>> https://issues.apache.org/jira/browse/HDFS-15160
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula <
> > > >>>>> bra...@apache.org>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>> > @Akira Ajisaka   and @Masatake Iwasaki
> > > >>>>> > 
> > > >>>>> > Looks all are build related issues when you try with bigtop. We
> > can
> > > >>>>> > discuss and prioritize this.. Will connect with you guys.
> > > >>>>> >
> > > >>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
> > > >>>>> > iwasak...@oss.nttdata.co.jp> wrote:
> > > >>>>> >
> > > >>>>> >> >> -
> > > >>>>> >>
> > > >>>>>
> > >
> >
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> > > >>>>> >> >
> > > >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> > > >>>>> >> > we do not see the

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Hi Akira,

Thanks for your email!!

I am evaluating the CVE’s which needs to go for this release..

Will update soon!!


On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka  wrote:

> Hi Brahma,
>
> What is the release process going on? Is there any blocker for the RC?
>
> -Akira
>
> On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He  wrote:
>
> > Hi Brahma,
> >
> > The feature 'BPServiceActor processes commands from NameNode
> > asynchronously' has been ready for both branch-3.2 and branch-3.2.3.
> While
> > cherry-picking there is only minor conflict, So I checked in directly.
> BTW,
> > run some unit tests and build pseudo cluster to verify, it seems to work
> > fine.
> > FYI.
> >
> > Regards,
> > - He Xiaoqiao
> >
> > On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula  >
> > wrote:
> >
> >> Please go ahead. Let me know any help required on review.
> >>
> >> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He 
> wrote:
> >>
> >>> Hi Brahma,
> >>>
> >>> I plan to involve HDFS-14997 and related JIRAs if possible. I have
> >>> resolved the conflict and verified them locally.
> >>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
> >>> I would like to hear some more response that if we have enough time to
> >>> wait for it to be ready.
> >>> Thanks.
> >>>
> >>> Best Regards,
> >>> - He Xiaoqiao
> >>>
> >>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He 
> wrote:
> >>>
> >>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI.
> >>>>
> >>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula <
> bra...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly
> this
> >>>>> can
> >>>>> be done by this week,then I will try to create the RC next if there
> is
> >>>>> no
> >>>>> objection.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/HDFS-15160
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula <
> >>>>> bra...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>> > @Akira Ajisaka   and @Masatake Iwasaki
> >>>>> > 
> >>>>> > Looks all are build related issues when you try with bigtop. We can
> >>>>> > discuss and prioritize this.. Will connect with you guys.
> >>>>> >
> >>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
> >>>>> > iwasak...@oss.nttdata.co.jp> wrote:
> >>>>> >
> >>>>> >> >> -
> >>>>> >>
> >>>>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >>>>> >> >
> >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> >>>>> >> > we do not see the issue usually since branch-3.2 uses
> >>>>> zooekeper-3.4.13,
> >>>>> >> > while it would be harmless to add the exclusion even for
> >>>>> >> zooekeeper-3.4.13.
> >>>>> >>
> >>>>> >> I filed HADOOP-17849 for this.
> >>>>> >>
> >>>>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote:
> >>>>> >> > Thanks for bringing this up, Akira. Let me explain some
> >>>>> background.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >> -
> >>>>> >>
> >>>>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >>>>> >> >
> >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> >>>>> >> > we do not see the issue usually since branch-3.2 uses
> >>>>> zooekeper-3.4.13,
> >>>>> >> > while it would be harmless to add the exclusion even for
> >>>>> >> zooekeeper-3.4.13

[jira] [Created] (HADOOP-17917) Backport HADOOP-15993 to branch-3.2 which Address CVE-2014-4611

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-17917:
-

 Summary: Backport HADOOP-15993 to branch-3.2 which Address 
CVE-2014-4611
 Key: HADOOP-17917
 URL: https://issues.apache.org/jira/browse/HADOOP-17917
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula


Now the version is 0.8.2.1 and it has net.jpountz.lz4:lz4:1.2.0 dependency, 
which is vulnerable. 
([https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-4611])



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Please go ahead. Let me know any help required on review.

On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He  wrote:

> Hi Brahma,
>
> I plan to involve HDFS-14997 and related JIRAs if possible. I have
> resolved the conflict and verified them locally.
> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
> I would like to hear some more response that if we have enough time to
> wait for it to be ready.
> Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He  wrote:
>
>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI.
>>
>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula 
>> wrote:
>>
>>> Hi All,
>>>
>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly this
>>> can
>>> be done by this week,then I will try to create the RC next if there is no
>>> objection.
>>>
>>> https://issues.apache.org/jira/browse/HDFS-15160
>>>
>>>
>>>
>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula 
>>> wrote:
>>>
>>> > @Akira Ajisaka   and @Masatake Iwasaki
>>> > 
>>> > Looks all are build related issues when you try with bigtop. We can
>>> > discuss and prioritize this.. Will connect with you guys.
>>> >
>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
>>> > iwasak...@oss.nttdata.co.jp> wrote:
>>> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>>> >> >
>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>>> >> > we do not see the issue usually since branch-3.2 uses
>>> zooekeper-3.4.13,
>>> >> > while it would be harmless to add the exclusion even for
>>> >> zooekeeper-3.4.13.
>>> >>
>>> >> I filed HADOOP-17849 for this.
>>> >>
>>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote:
>>> >> > Thanks for bringing this up, Akira. Let me explain some background.
>>> >> >
>>> >> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>>> >> >
>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>>> >> > we do not see the issue usually since branch-3.2 uses
>>> zooekeper-3.4.13,
>>> >> > while it would be harmless to add the exclusion even for
>>> >> zooekeeper-3.4.13.
>>> >> >
>>> >> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>>> >> >
>>> >> > These are relevant to directory structure used by Bigtop package.
>>> >> > If the fix does not break the tarball dist,
>>> >> > it would be nice to have these on Hadoop too.
>>> >> >
>>> >> >
>>> >> >> -
>>> >>
>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>>> >> >
>>> >> > This is for aarch64 and ppe64le lacking required phantomjs.
>>> >> > It is only acceptable for Bigtop not running tests of YARN-UI2 on
>>> >> packaging.
>>> >> > Hadoop needs the phantomjs for testing YARN-UI2.
>>> >> >
>>> >> >
>>> >> > On 2021/08/16 2:59, Brahma Reddy Battula wrote:
>>> >> >> Thanks, I'll check them out.
>>> >> >>
>>> >> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka <
>>> aajis...@apache.org>
>>> >> wrote:
>>> >> >>
>>> >> >>> Thanks Brahma for cutting branch-3.2.3.
>>> >> >>>
>>> >> >>> In Apache Bigtop, there are some patches applied

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Thanks all for attending the session, Hope we can have some more Good
sessions.. Please find the recordings.

https://drive.google.com/drive/u/1/folders/1mtIEUfnEUtil65U0nGGeJNAUYVsPpktt

On Wed, Sep 8, 2021 at 7:37 PM Brahma Reddy Battula 
wrote:

> Hi Sean,
>
> Pacific Time 9th, Sept 9:30 AM…..
>
> Please check the following link relevant time zone..
>
> Time zone :: https://t.co/JmTN6zK8Ga?amp=1
>
>
>
> On Wed, 8 Sep 2021 at 7:12 PM, Sean Busbey  wrote:
>
>> Hi Brahma!
>>
>> Thanks for organizing this. What’s the timezone for the 10p - midnight?
>> Pacific Time?
>>
>> > On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula 
>> wrote:
>> >
>> > Hi All,
>> >
>> > Updated the meeting to record the session.. Please use the following
>> link
>> > to attend the conference tomorrow.
>> >
>> >
>> > Uber's story on running Apache Hadoop deployment in Docker
>> > September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
>> > Google Meet joining info
>> > *Video call link: https://meet.google.com/few-wppc-xoa
>> > <https://meet.google.com/few-wppc-xoa>*
>> > Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
>> > More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915
>> >
>> > On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula <
>> bra...@apache.org>
>> > wrote:
>> >
>> >> Hi All,
>> >>
>> >> Happy to announce the following conference. Block your calendar and
>> make
>> >> yourself available.
>> >> Thanks Mithun and team for accepting.
>> >>
>> >>
>> >> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
>> >>
>> >> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
>> >>
>> >> *Meeting Link :   *Google Meet joining info
>> >>   Video call link:
>> >> https://meet.google.com/akk-qmzy-qsu
>> >>
>> >> *  Note: *Created Teams meeting also as a
>> backup
>> >> [2].
>> >>
>> >> *High level Agenda: *
>> >>
>> >>   - Importance of Hadoop Cluster Management (3-5 min)
>> >>   - Introduction about problem space
>> >>   - Challenges in this space and why it is important to solve them
>> >>   - Evolution of Uber Hadoop Cluster Management (10-15 min)
>> >>   -  How our strategy evolved over time since Hadoop was set up at Uber
>> >>   -  Key learnings from the evolution over the years
>> >>   - Our current approach today (15-20min)
>> >>   -  Key learnings from this major overhaul
>> >>   -  Current challenges that we are working on
>> >>   - Q (20 min)
>> >>
>> >>
>> >> Aiming for 20 min Q assuming that there will be more questions based
>> on
>> >> the blog that was published(1) + presentation at the session.
>> >>
>> >>
>> >>
>> >>
>> >>
>> ---
>> >> 1) https://eng.uber.com/hadoop-container-blog/
>> >> <
>> https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22
>> >
>> >> 2)
>> >>
>> >> *Join on your computer or mobile app*
>> >>
>> >> Click here to join the meeting
>> >> <
>> https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d
>> >
>> >>
>> >> *Join with a video conferencing device*
>> >>
>> >> 967904...@t.plcm.vc
>> >>
>> >> Video Conference ID: 117 464 626 6
>> >>
>> >> Alternate VTC instructions
>> >> <https://dialin.plcm.vc/teams/?key=967904974=1174646266>
>> >>
>> >> *Or call in (audio only)*
>> >>
>> >> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
>> >> Los Angeles
>> >>
>> >> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
>> >> (Toll-free)
>> >>
>> >> Phone Conference ID: 273 569 658#
>> >>
>> >> Find a local number
>> >> <
>> https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658
>> >
>> >> | Reset PIN <https://mysettings.lync.com/pstnconferencing>
>> >>
>> >>
>> >>
>> >> -- Brahma Reddy Battula
>> >>
>> >
>> >
>> > --
>> >
>> >
>> >
>> > --Brahma Reddy Battula
>>
>>
>> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Hi All,

Waiting for the following jira to commit to hadoop-3.2.3 , mostly this can
be done by this week,then I will try to create the RC next if there is no
objection.

https://issues.apache.org/jira/browse/HDFS-15160



On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula 
wrote:

> @Akira Ajisaka   and @Masatake Iwasaki
> 
> Looks all are build related issues when you try with bigtop. We can
> discuss and prioritize this.. Will connect with you guys.
>
> On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>> >
>> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
>> > while it would be harmless to add the exclusion even for
>> zooekeeper-3.4.13.
>>
>> I filed HADOOP-17849 for this.
>>
>> On 2021/08/16 12:02, Masatake Iwasaki wrote:
>> > Thanks for bringing this up, Akira. Let me explain some background.
>> >
>> >
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>> >
>> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
>> > while it would be harmless to add the exclusion even for
>> zooekeeper-3.4.13.
>> >
>> >
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>> >
>> > These are relevant to directory structure used by Bigtop package.
>> > If the fix does not break the tarball dist,
>> > it would be nice to have these on Hadoop too.
>> >
>> >
>> >> -
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>> >
>> > This is for aarch64 and ppe64le lacking required phantomjs.
>> > It is only acceptable for Bigtop not running tests of YARN-UI2 on
>> packaging.
>> > Hadoop needs the phantomjs for testing YARN-UI2.
>> >
>> >
>> > On 2021/08/16 2:59, Brahma Reddy Battula wrote:
>> >> Thanks, I'll check them out.
>> >>
>> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka 
>> wrote:
>> >>
>> >>> Thanks Brahma for cutting branch-3.2.3.
>> >>>
>> >>> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
>> >>>
>> >>>
>> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
>> >>>
>> >>> In these patches, how about backporting the following issues to
>> >>> branch-3.2 and branch-3.2.3?
>> >>> - HADOOP-14922
>> >>> - HADOOP-15939
>> >>> - HADOOP-17569
>> >>>
>> >>> In addition, there are some patches that don't have JIRA issue ID.
>> >>> Maybe we need to create JIRAs and fix those.
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>> >>> -
>> >>>
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>> >>>
>> >>> Thanks and regards,
>> >>> Akira
>> >>>
>> >>> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He 
>> >>> wrote:
>> >>>>
>> >>>> Thanks Brahma for initiating this and making hadoop-3.2.3 release
>> happen.
>> >>>>
>> >>>> I would li

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Hi Sean,

Pacific Time 9th, Sept 9:30 AM…..

Please check the following link relevant time zone..

Time zone :: https://t.co/JmTN6zK8Ga?amp=1



On Wed, 8 Sep 2021 at 7:12 PM, Sean Busbey  wrote:

> Hi Brahma!
>
> Thanks for organizing this. What’s the timezone for the 10p - midnight?
> Pacific Time?
>
> > On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula 
> wrote:
> >
> > Hi All,
> >
> > Updated the meeting to record the session.. Please use the following link
> > to attend the conference tomorrow.
> >
> >
> > Uber's story on running Apache Hadoop deployment in Docker
> > September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
> > Google Meet joining info
> > *Video call link: https://meet.google.com/few-wppc-xoa
> > <https://meet.google.com/few-wppc-xoa>*
> > Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
> > More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915
> >
> > On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula  >
> > wrote:
> >
> >> Hi All,
> >>
> >> Happy to announce the following conference. Block your calendar and make
> >> yourself available.
> >> Thanks Mithun and team for accepting.
> >>
> >>
> >> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
> >>
> >> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
> >>
> >> *Meeting Link :   *Google Meet joining info
> >>   Video call link:
> >> https://meet.google.com/akk-qmzy-qsu
> >>
> >> *  Note: *Created Teams meeting also as a backup
> >> [2].
> >>
> >> *High level Agenda: *
> >>
> >>   - Importance of Hadoop Cluster Management (3-5 min)
> >>   - Introduction about problem space
> >>   - Challenges in this space and why it is important to solve them
> >>   - Evolution of Uber Hadoop Cluster Management (10-15 min)
> >>   -  How our strategy evolved over time since Hadoop was set up at Uber
> >>   -  Key learnings from the evolution over the years
> >>   - Our current approach today (15-20min)
> >>   -  Key learnings from this major overhaul
> >>   -  Current challenges that we are working on
> >>   - Q (20 min)
> >>
> >>
> >> Aiming for 20 min Q assuming that there will be more questions based
> on
> >> the blog that was published(1) + presentation at the session.
> >>
> >>
> >>
> >>
> >>
> ---
> >> 1) https://eng.uber.com/hadoop-container-blog/
> >> <
> https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22
> >
> >> 2)
> >>
> >> *Join on your computer or mobile app*
> >>
> >> Click here to join the meeting
> >> <
> https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d
> >
> >>
> >> *Join with a video conferencing device*
> >>
> >> 967904...@t.plcm.vc
> >>
> >> Video Conference ID: 117 464 626 6
> >>
> >> Alternate VTC instructions
> >> <https://dialin.plcm.vc/teams/?key=967904974=1174646266>
> >>
> >> *Or call in (audio only)*
> >>
> >> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
> >> Los Angeles
> >>
> >> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
> >> (Toll-free)
> >>
> >> Phone Conference ID: 273 569 658#
> >>
> >> Find a local number
> >> <
> https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658
> >
> >> | Reset PIN <https://mysettings.lync.com/pstnconferencing>
> >>
> >>
> >>
> >> -- Brahma Reddy Battula
> >>
> >
> >
> > --
> >
> >
> >
> > --Brahma Reddy Battula
>
>
> --



--Brahma Reddy Battula

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Hi All,

 Updated the meeting to record the session.. Please use the following link
to attend the conference tomorrow.


Uber's story on running Apache Hadoop deployment in Docker
September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
Google Meet joining info
*Video call link: https://meet.google.com/few-wppc-xoa
<https://meet.google.com/few-wppc-xoa>*
Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915

On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula 
wrote:

> Hi All,
>
> Happy to announce the following conference. Block your calendar and make
> yourself available.
> Thanks Mithun and team for accepting.
>
>
> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
>
> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
>
> *Meeting Link :   *Google Meet joining info
>Video call link:
> https://meet.google.com/akk-qmzy-qsu
>
> *  Note: *Created Teams meeting also as a backup
> [2].
>
> *High level Agenda: *
>
>- Importance of Hadoop Cluster Management (3-5 min)
>- Introduction about problem space
>- Challenges in this space and why it is important to solve them
>- Evolution of Uber Hadoop Cluster Management (10-15 min)
>-  How our strategy evolved over time since Hadoop was set up at Uber
>-  Key learnings from the evolution over the years
>- Our current approach today (15-20min)
>-  Key learnings from this major overhaul
>-  Current challenges that we are working on
>- Q (20 min)
>
>
> Aiming for 20 min Q assuming that there will be more questions based on
> the blog that was published(1) + presentation at the session.
>
>
>
>
> ---
> 1) https://eng.uber.com/hadoop-container-blog/
> <https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22>
> 2)
>
> *Join on your computer or mobile app*
>
> Click here to join the meeting
> <https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d>
>
> *Join with a video conferencing device*
>
> 967904...@t.plcm.vc
>
> Video Conference ID: 117 464 626 6
>
> Alternate VTC instructions
> <https://dialin.plcm.vc/teams/?key=967904974=1174646266>
>
> *Or call in (audio only)*
>
> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
> Los Angeles
>
> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
> (Toll-free)
>
> Phone Conference ID: 273 569 658#
>
> Find a local number
> <https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658>
>  | Reset PIN <https://mysettings.lync.com/pstnconferencing>
>
>
>
> -- Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

[Conference] Uber's story on running Apache Hadoop deployment in Docker

[Brahma Reddy Battula]

Hi All,

Happy to announce the following conference. Block your calendar and make
yourself available.
Thanks Mithun and team for accepting.


*Topic:*Uber's story on running Apache Hadoop deployment in Docker

*Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*

*Meeting Link :   *Google Meet joining info
   Video call link:
https://meet.google.com/akk-qmzy-qsu

*  Note: *Created Teams meeting also as a backup
[2].

*High level Agenda: *

   - Importance of Hadoop Cluster Management (3-5 min)
   - Introduction about problem space
   - Challenges in this space and why it is important to solve them
   - Evolution of Uber Hadoop Cluster Management (10-15 min)
   -  How our strategy evolved over time since Hadoop was set up at Uber
   -  Key learnings from the evolution over the years
   - Our current approach today (15-20min)
   -  Key learnings from this major overhaul
   -  Current challenges that we are working on
   - Q (20 min)


Aiming for 20 min Q assuming that there will be more questions based on
the blog that was published(1) + presentation at the session.



---
1) https://eng.uber.com/hadoop-container-blog/
<https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/=D=calendar=2=AOvVaw3OutuZVQBQnkhUd4fxaM22>
2)

*Join on your computer or mobile app*

Click here to join the meeting
<https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d>

*Join with a video conferencing device*

967904...@t.plcm.vc

Video Conference ID: 117 464 626 6

Alternate VTC instructions
<https://dialin.plcm.vc/teams/?key=967904974=1174646266>

*Or call in (audio only)*

+1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States, Los
Angeles

(833) 827-4491,,273569658# <8338274491,,273569658#>   United States
(Toll-free)

Phone Conference ID: 273 569 658#

Find a local number
<https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658>
 | Reset PIN <https://mysettings.lync.com/pstnconferencing>



-- Brahma Reddy Battula

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

@Akira Ajisaka   and @Masatake Iwasaki

Looks all are build related issues when you try with bigtop. We can discuss
and prioritize this.. Will connect with you guys.

On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >
> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
> > while it would be harmless to add the exclusion even for
> zooekeeper-3.4.13.
>
> I filed HADOOP-17849 for this.
>
> On 2021/08/16 12:02, Masatake Iwasaki wrote:
> > Thanks for bringing this up, Akira. Let me explain some background.
> >
> >
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >
> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13,
> > while it would be harmless to add the exclusion even for
> zooekeeper-3.4.13.
> >
> >
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
> >
> > These are relevant to directory structure used by Bigtop package.
> > If the fix does not break the tarball dist,
> > it would be nice to have these on Hadoop too.
> >
> >
> >> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
> >
> > This is for aarch64 and ppe64le lacking required phantomjs.
> > It is only acceptable for Bigtop not running tests of YARN-UI2 on
> packaging.
> > Hadoop needs the phantomjs for testing YARN-UI2.
> >
> >
> > On 2021/08/16 2:59, Brahma Reddy Battula wrote:
> >> Thanks, I'll check them out.
> >>
> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka 
> wrote:
> >>
> >>> Thanks Brahma for cutting branch-3.2.3.
> >>>
> >>> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
> >>>
> >>>
> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
> >>>
> >>> In these patches, how about backporting the following issues to
> >>> branch-3.2 and branch-3.2.3?
> >>> - HADOOP-14922
> >>> - HADOOP-15939
> >>> - HADOOP-17569
> >>>
> >>> In addition, there are some patches that don't have JIRA issue ID.
> >>> Maybe we need to create JIRAs and fix those.
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
> >>> -
> >>>
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
> >>>
> >>> Thanks and regards,
> >>> Akira
> >>>
> >>> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He 
> >>> wrote:
> >>>>
> >>>> Thanks Brahma for initiating this and making hadoop-3.2.3 release
> happen.
> >>>>
> >>>> I would like to validate the HBase project (both the latest release
> and
> >>>> trunk branch).
> >>>> Chao Sun will validate the Spark Project (Got in touch with Chao
> >>> already).
> >>>> once RC is out.
> >>>>
> >>>> Thanks and Regards,
> >>>> - He Xiaoqiao
> >>>>
> >>>>
> >>>> On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula <
> bra...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>

Re: Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Thanks, I'll check them out.

On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka  wrote:

> Thanks Brahma for cutting branch-3.2.3.
>
> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
>
> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop
>
> In these patches, how about backporting the following issues to
> branch-3.2 and branch-3.2.3?
> - HADOOP-14922
> - HADOOP-15939
> - HADOOP-17569
>
> In addition, there are some patches that don't have JIRA issue ID.
> Maybe we need to create JIRAs and fix those.
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
> -
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff
>
> Thanks and regards,
> Akira
>
> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He 
> wrote:
> >
> > Thanks Brahma for initiating this and making hadoop-3.2.3 release happen.
> >
> > I would like to validate the HBase project (both the latest release and
> > trunk branch).
> > Chao Sun will validate the Spark Project (Got in touch with Chao
> already).
> > once RC is out.
> >
> > Thanks and Regards,
> > - He Xiaoqiao
> >
> >
> > On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula 
> > wrote:
> >
> > > Hi All,
> > >
> > > I cut branch-3.2.3 and it is ready for release. Please commit to
> > > branch-3.2.3 if any critical/blocker issues need to go.
> > >
> > > *This time I am thinking of having downstream projects and companies'
> > > voices,let's know how this can go. *
> > >
> > >- Planning to check with downstream projects like Spark,HBase and
> Hive
> > >if they can help on validation(Or running their UT on this branch)
> > >- Collecting info from Companies who are already deployed and using
> the
> > >branch-3.2
> > >
> > >
> > > so that we can make a more stable release on 3.2 (so that features
> released
> > > on this branch impact can be known) ,please let me know anybody from
> these
> > > communities who can help on this.
> > >
> > >
> > > Planning to create RC by this month end. Any suggestions are welcome.
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
>
-- 



--Brahma Reddy Battula

Hadoop-3.2.3 Release Update

[Brahma Reddy Battula]

Hi All,

I cut branch-3.2.3 and it is ready for release. Please commit to
branch-3.2.3 if any critical/blocker issues need to go.

*This time I am thinking of having downstream projects and companies'
voices,let's know how this can go. *

   - Planning to check with downstream projects like Spark,HBase and Hive
   if they can help on validation(Or running their UT on this branch)
   - Collecting info from Companies who are already deployed and using the
   branch-3.2


so that we can make a more stable release on 3.2 (so that features released
on this branch impact can be known) ,please let me know anybody from these
communities who can help on this.


Planning to create RC by this month end. Any suggestions are welcome.



--Brahma Reddy Battula

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Brahma Reddy Battula]

@Wei-Chiu Chuang  looks this is not concluded yet...
Can we move forward..?

On Thu, Jul 15, 2021 at 11:09 PM Brahma Reddy Battula 
wrote:

>
> I agree with Ahmed Hussein…Jira should not be used for number generation..
>
> We can always revisit the jira to see useful discussion at one place…
>
> @wei-chu, +1 on proposal for cleaning the PR’s..
>
>
> On Thu, 15 Jul 2021 at 9:15 PM, epa...@apache.org 
> wrote:
>
>>  > I usually use PR comments to discuss about the patch submitted.
>> My concern is that still leaves multiple places to look in order to get a
>> full picture of an issue.
>> -Eric
>>
>> On Wednesday, July 14, 2021, 7:07:30 PM CDT, Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp> wrote:
>>
>>  > - recently, JIRA became some sort of a "number generator" with
>> insufficient
>> > description/details as the
>> >developers and the reviewers spending more time discussing in the PR.
>>
>> JIRA issues contain useful information in the fields.
>> We are leveraging them in development and release process.
>>
>> * https://yetus.apache.org/documentation/0.13.0/releasedocmaker/
>> *
>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336122
>>
>> I usually use PR comments to discuss about the patch submitted.
>> JIRA comments are used for background or design discussion before and
>> after submitting PR.
>> There would be no problem having no comment in minor/trivial JIRA issues.
>>
>>
>> On 2021/07/14 23:50, Ahmed Hussein wrote:
>> > Do you consider migrating Jira issues to Github issues?
>> >
>> > I am a little bit concerned that there are some committers who still
>> prefer
>> > Jira-precommits over GitHub PR
>> > (P.S. I am not a committer).
>> >
>> > Their point is that Github-PR confuses them with discussions/comments
>> being
>> > in two places rather than one.
>> >
>> > Personally, I found several Github-PRs comments discussing the validity
>> of
>> > the feature/bug.
>> > As a result:
>> > - recently, JIRA became some sort of a "number generator" with
>> insufficient
>> > description/details as the
>> >developers and the reviewers spending more time discussing in the PR.
>> > - the relation between a single Jira and Github-PR is 1-to-M. In order
>> to
>> > find related discussions, the user may
>> >need to visit every PR (that may include closed ones)
>> >
>> >
>> >
>> > On Wed, Jul 14, 2021 at 8:46 AM Steve Loughran
>> 
>> > wrote:
>> >
>> >> not sure about stale PR closing; when you've a patch which is still
>> pending
>> >> review it's not that fun to have it closed.
>> >>
>> >> maybe better to have review sessions. I recall many, many years ago
>> >> attempts to try and catch up with all outstanding patch reviews.
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, 14 Jul 2021 at 03:00, Akira Ajisaka 
>> wrote:
>> >>
>> >>> Thank you Wei-Chiu for starting the discussion,
>> >>>
>> >>>> 3. JIRA security
>> >>> I'm +1 to use private JIRA issues to handle vulnerabilities.
>> >>>
>> >>>> 5. Doc update
>> >>> +1, I build the document daily and it helps me fixing documents:
>> >>> https://aajisaka.github.io/hadoop-document/ It's great if the latest
>> >>> document is built and published by the Apache Hadoop community.
>> >>>
>> >>> My idea related to GitHub PR:
>> >>> 1. Disable the precommit jobs for JIRA, always use GitHub PR. It saves
>> >>> costs to configure and debug the precommit jobs.
>> >>> https://issues.apache.org/jira/browse/HADOOP-17798
>> >>> 2. Improve the pull request template for the contributors
>> >>> https://issues.apache.org/jira/browse/HADOOP-17799
>> >>>
>> >>> Regards,
>> >>> Akira
>> >>>
>> >>> On Tue, Jul 13, 2021 at 12:35 PM Wei-Chiu Chuang 
>> >>> wrote:
>> >>>>
>> >>>> I work on multiple projects and learned a bunch from those
>> >> projects.There
>> >>>> are nice add-ons that help with productivity. There are things we can
>> >> do
>> >>> to
>> >>>> help us manage the project better.
>>

Re: [External Sender] Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

**Important update**  Hadoop-3.2.3 Release progress.


I cut branch-3.2.3 and it is ready for release. Please commit to
branch-3.2.3 if any critical/blocker issues need to go.
As previously communicated,Planning to create RC by this month end. Any
suggestions are welcome.






On Thu, Aug 5, 2021 at 10:15 PM Jason Wen  wrote:

> Hi Steve,
>
> Which aws-java-sdk-bundle version did you test and encounter the following
> test failures?
>
> Thanks,
> Jason
>
> On 8/4/21, 7:51 AM, "Steve Loughran"  wrote:
>
> it didn't take, so don't worry about it. Test failures
>
> java.lang.IllegalArgumentException: Input is expected to be encoded in
> multiple of 4 bytes but found: 75
> at com.amazonaws.util.Base64Codec.decode(Base64Codec.java:198)
> at com.amazonaws.util.Base64.decode(Base64.java:116)
> at
>
> com.amazonaws.services.s3.AmazonS3Client.populateSSE_C(AmazonS3Client.java:4620)
> at
>
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1806)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1603)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$13(S3AFileSystem.java:2828)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2826)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2801)
> at
>
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2146)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2079)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2304)
> at
>
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
> at
>
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)
>
> On Tue, 3 Aug 2021 at 19:41, Brahma Reddy Battula 
> wrote:
>
> > Hi Steve,
> >
> > Is there any jira for this ??
> >
> >
> >
> > On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran
>  > >
> > wrote:
> >
> > > I'm testing how well a backport of the latest AWS SDK goes; if all
> is
> > > docile there then I'll merge that. Due diligence more than anything
> > > elseone of the shaded netty JARs (never used by our code) has
> a CVE
> > in
> > > older versions
> > >
> > > On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka 
> wrote:
> > >
> > > > Hi Steven,
> > > >
> > > > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
> > > > opened a PR to backport YARN-8990:
> > > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_pull_3254=DwIBaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o=dXWDhhZW6SZQt-9oQONYtGPMFTrUL4225rMKK-494YA=tLml0Gisl-xPsd4B67BNBZtPsz1T0qBPppHYRzLLl_0=
> > > >
> > > > Thanks,
> > > > Akira
> > > >
> > > > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand <
> stevenjr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > I think it would be helpful if we could include YARN-8990 and
> > YARN-8992
> > > > in the 3.2.3 release. Both are important fixes which were
> included in
> > > > 3.2.0, but never made their way to branch-3.2, so were omitted
> from
> > both
> > > > 3.2.1 and 3.2.2.
> > > > >
> > > > > Best,
> > > > > Steve
> > > > >
> > > > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He <
> xq.he2...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> cc @dev mail-list.
> > > > >>
> > > > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He <
> xq.he2...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > Hi Brahma,
> > > > >> >
>     > > > >> > I just created

[jira] [Resolved] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2

[Brahma Reddy Battula (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HADOOP-17840.
---
Fix Version/s: 3.2.3
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to branch-3.2.3...[~bbeaudreault] thanks for your contribution.

> Backport HADOOP-17837 to branch-3.2
> ---
>
> Key: HADOOP-17840
> URL: https://issues.apache.org/jira/browse/HADOOP-17840
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.2.3
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect

[Brahma Reddy Battula (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HADOOP-17837.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

[~bbeaudreault] thanks raising the PR. Committed to trunk and branch-3.3.. As 
this only test assertion, ran locally and pushed.

> Make it easier to debug UnknownHostExceptions from NetUtils.connect
> ---
>
> Key: HADOOP-17837
> URL: https://issues.apache.org/jira/browse/HADOOP-17837
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Most UnknownHostExceptions thrown throughout hadoop include a useful message, 
> either the hostname that was not found or some other descriptor of the 
> problem. The UnknownHostException thrown from NetUtils.connect only includes 
> the [message of the underlying 
> UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592].
>  If you take a look at the source for UnresolvedAddressException, [it only 
> has a no-args 
> constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html]
>  (java11, but same is true in other versions). So it never has a message, 
> meaning the UnknownHostException message is empty.
> We should include the endpoint.toString() in the UnknownHostException thrown 
> by NetUtils.connect



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Hi Steve,

Is there any jira for this ??



On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran 
wrote:

> I'm testing how well a backport of the latest AWS SDK goes; if all is
> docile there then I'll merge that. Due diligence more than anything
> elseone of the shaded netty JARs (never used by our code) has a CVE in
> older versions
>
> On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka  wrote:
>
> > Hi Steven,
> >
> > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
> > opened a PR to backport YARN-8990:
> > https://github.com/apache/hadoop/pull/3254
> >
> > Thanks,
> > Akira
> >
> > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand 
> > wrote:
> > >
> > > I think it would be helpful if we could include YARN-8990 and YARN-8992
> > in the 3.2.3 release. Both are important fixes which were included in
> > 3.2.0, but never made their way to branch-3.2, so were omitted from both
> > 3.2.1 and 3.2.2.
> > >
> > > Best,
> > > Steve
> > >
> > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He 
> wrote:
> > >>
> > >> cc @dev mail-list.
> > >>
> > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He 
> > wrote:
> > >>
> > >> > Hi Brahma,
> > >> >
> > >> > I just created version 3.2.4, and changed all unresolved issues
> > (target
> > >> > version/s: 3.2.3) to 3.2.4 after checking both of them are not
> blocker
> > >> > issues. Dashboard[1] is clean now.
> > >> >
> > >> > Regards,
> > >> > - He Xiaoqiao
> > >> >
> > >> > [1]
> > >> >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> > >> >
> > >> > On Sun, Jul 25, 2021 at 7:45 PM Brahma Reddy Battula <
> > bra...@apache.org>
> > >> > wrote:
> > >> >
> > >> >> Hi Xiaoqiao,
> > >> >>
> > >> >> Thanks for creating the Dashboard, we need to change the filters
> and
> > >> >> target versions in the jira.
> > >> >>
> > >> >> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He 
> > wrote:
> > >> >>
> > >> >>> Thanks Brahma for volunteering and driving this release plan. I
> just
> > >> >>> created a dashboard for 3.2.3 release[1].
> > >> >>> I would like to support for this release line if need. (cc Brahma)
> > >> >>>
> > >> >>> Thanks. Regards,
> > >> >>> - He Xiaoqiao
> > >> >>>
> > >> >>> [1]
> > >> >>>
> > >> >>>
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> > >> >>>
> > >> >>>
> > >> >>> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka <
> aajis...@apache.org>
> > >> >>> wrote:
> > >> >>>
> > >> >>> > Hi Brahma,
> > >> >>> >
> > >> >>> > Thank you for volunteering!
> > >> >>> >
> > >> >>> > -Akira
> > >> >>> >
> > >> >>> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula <
> > >> >>> bra...@apache.org>
> > >> >>> > wrote:
> > >> >>> > >
> > >> >>> > > Hi Akira,
> > >> >>> > >
> > >> >>> > > Thanks for bringing this..
> > >> >>> > >
> > >> >>> > > I want to drive this if nobody already plan to do this..
> > >> >>> > >
> > >> >>> > >
> > >> >>> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka <
> > aajis...@apache.org>
> > >> >>> > wrote:
> > >> >>> > >
> > >> >>> > > > Hi all,
> > >> >>> > > >
> > >> >>> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
> > >> >>> > > > accumulated more than 230 commits [1]. Therefore I want to
> > start
> > >> >>> the
> > >> >>> > > > release work for 3.2.3.
> > >> &

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Update on this thread.

Will cut off the branch next week.
And planning to release it in the last week of next month.


Please try to mark jira's which are required for this release if anything
is missed or reply to this mail.






On Sun, Jul 25, 2021 at 5:14 PM Brahma Reddy Battula 
wrote:

> Hi Xiaoqiao,
>
> Thanks for creating the Dashboard, we need to change the filters and
> target versions in the jira.
>
> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He  wrote:
>
>> Thanks Brahma for volunteering and driving this release plan. I just
>> created a dashboard for 3.2.3 release[1].
>> I would like to support for this release line if need. (cc Brahma)
>>
>> Thanks. Regards,
>> - He Xiaoqiao
>>
>> [1]
>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
>>
>>
>> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka 
>> wrote:
>>
>> > Hi Brahma,
>> >
>> > Thank you for volunteering!
>> >
>> > -Akira
>> >
>> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula > >
>> > wrote:
>> > >
>> > > Hi Akira,
>> > >
>> > > Thanks for bringing this..
>> > >
>> > > I want to drive this if nobody already plan to do this..
>> > >
>> > >
>> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka 
>> > wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
>> > > > accumulated more than 230 commits [1]. Therefore I want to start the
>> > > > release work for 3.2.3.
>> > > >
>> > > > There is one blocker for 3.2.3 [2].
>> > > > - https://issues.apache.org/jira/browse/HDFS-12920
>> > > >
>> > > > Is there anyone who would volunteer to be the 3.2.3 release manager?
>> > > > Are there any other blockers? If any, please file an issue, raise
>> the
>> > > > blocker, and add the target version.
>> > > >
>> > > > [1]
>> > > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
>> > > > [2]
>> > > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > >
>> -
>> > > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> > > > For additional commands, e-mail:
>> mapreduce-dev-h...@hadoop.apache.org
>> > > >
>> > > > --
>> > >
>> > >
>> > >
>> > > --Brahma Reddy Battula
>> >
>> > -
>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Hi Xiaoqiao,

Thanks for creating the Dashboard, we need to change the filters and target
versions in the jira.

On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He  wrote:

> Thanks Brahma for volunteering and driving this release plan. I just
> created a dashboard for 3.2.3 release[1].
> I would like to support for this release line if need. (cc Brahma)
>
> Thanks. Regards,
> - He Xiaoqiao
>
> [1]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
>
>
> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka  wrote:
>
> > Hi Brahma,
> >
> > Thank you for volunteering!
> >
> > -Akira
> >
> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula 
> > wrote:
> > >
> > > Hi Akira,
> > >
> > > Thanks for bringing this..
> > >
> > > I want to drive this if nobody already plan to do this..
> > >
> > >
> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
> > > > accumulated more than 230 commits [1]. Therefore I want to start the
> > > > release work for 3.2.3.
> > > >
> > > > There is one blocker for 3.2.3 [2].
> > > > - https://issues.apache.org/jira/browse/HDFS-12920
> > > >
> > > > Is there anyone who would volunteer to be the 3.2.3 release manager?
> > > > Are there any other blockers? If any, please file an issue, raise the
> > > > blocker, and add the target version.
> > > >
> > > > [1]
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
> > > > [2]
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
> > > >
> > > > Regards,
> > > > Akira
> > > >
> > > > -----
> > > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail:
> mapreduce-dev-h...@hadoop.apache.org
> > > >
> > > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.2.3 release

[Brahma Reddy Battula]

Hi Akira,

Thanks for bringing this..

I want to drive this if nobody already plan to do this..


On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka  wrote:

> Hi all,
>
> Hadoop 3.2.2 was released half a year ago, and now, we have
> accumulated more than 230 commits [1]. Therefore I want to start the
> release work for 3.2.3.
>
> There is one blocker for 3.2.3 [2].
> - https://issues.apache.org/jira/browse/HDFS-12920
>
> Is there anyone who would volunteer to be the 3.2.3 release manager?
> Are there any other blockers? If any, please file an issue, raise the
> blocker, and add the target version.
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
> [2]
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
>
> Regards,
> Akira
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
> --



--Brahma Reddy Battula

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Brahma Reddy Battula]

;> can perform static analysis, catching potential issues. For example,
> >>> Ozone
> >>>> adds SonarQube to post-commit, and exports the report to SonarCloud.
> >>> Other
> >>>> add-ons are available to scan for docker images, vulnerabilities
> scans.
> >>>>
> >>>> 3. JIRA security
> >>>> It is possible to set up security level (public/private) in JIRA. This
> >>> can
> >>>> be used to track vulnerability issues and be made only visible to
> >>>> committers. Example: INFRA-15258
> >>>> <https://issues.apache.org/jira/browse/INFRA-15258>
> >>>>
> >>>> 4. New JIRA fields
> >>>> It's possible to add new fields. For example, we can add a "Reviewer"
> >>>> field, which could help improve the attention to issues.
> >>>>
> >>>> 5. Doc update
> >>>> It is possible to set up automation such that the doc on the Hadoop
> >>> website
> >>>> is refreshed for every commit, providing the latest doc to the public.
> >>>>
> >>>> 6. Webhook
> >>>> It's possible to set up webhook such that every commit in GitHub sends
> >> a
> >>>> notification to the ASF slack. It can be used for other kinds of
> >>>> automation. Sky's the limit.
> >>>>
> >>>> Thoughts? What else can do we?
> >>>
> >>> -
> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>
> >
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

-- 



--Brahma Reddy Battula

[jira] [Created] (HADOOP-17800) CLONE - Uber-JIRA: Hadoop should support IPv6

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-17800:
-

 Summary: CLONE - Uber-JIRA: Hadoop should support IPv6
 Key: HADOOP-17800
 URL: https://issues.apache.org/jira/browse/HADOOP-17800
 Project: Hadoop Common
  Issue Type: Improvement
  Components: net
Reporter: Brahma Reddy Battula
Assignee: Nate Edel


Hadoop currently treats IPv6 as unsupported.  Track related smaller issues to 
support IPv6.

(Current case here is mainly HBase on HDFS, so any suggestions about other test 
cases/workload are really appreciated.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Wei-Chiu,

Thanks for driving this..


Looks index.html is not updated and Changelog is broken.


On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.3.1.
>
> This is the first stable release of Apache Hadoop 3.3.x line. It contains
> 697 bug fixes, improvements and enhancements since 3.3.0.
>
> Users are encouraged to read the overview of major changes
> <https://hadoop.apache.org/docs/r3.3.1/index.html> since 3.3.0. For
> details
> of 697 bug fixes, improvements, and other enhancements since the previous
> 3.3.0 release, please check release notes
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html
> >
>  and changelog
> <
> http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html
> >
> detail
> the changes since 3.3.0.
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki,
> Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B,
> Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this
> release process.
>
> Best Regards,
> Wei-Chiu Chuang
>


-- 



--Brahma Reddy Battula

Re: [E] Re: Do I need a +1 to merge a backport PR?

[Brahma Reddy Battula]

Currently how the addendum will be added to PR??
 Won’t  be another PR?? Where build run again ??


On Thu, 3 Jun 2021 at 1:33 AM, Eric Badger 
wrote:

> I'm of a similar opinion to most here. If the backport is clean, I think
> it's ok to do it with just the +1 on the original patch. However, please
> please please build the code on the target branch before backporting
>
> Eric
>
> On Wed, Jun 2, 2021 at 2:46 PM Ayush Saxena  wrote:
>
> > For trivial changes, like changes in import or conflicts due to line
> > number or other trivial stuff, I don’t think that is required. Unless the
> > general  logic isn’t changing, we can go ahead, may be we can do a test
> run
> > before merging, to be on the safer side as and when required. :-)
> >
> > -Ayush
> >
> > > On 02-Jun-2021, at 10:13 AM, Wei-Chiu Chuang 
> wrote:
> > >
> > > I'm curious about the GitHub PR conventions we use today... say I want
> > to
> > > backport a commit from trunk to branch-3.3, and there's a small code
> > > conflict so I push a PR against branch-3.3 using GitHub to go through
> the
> > > precommit check.
> > >
> > > Do I need explicit approval from another committer to merge the
> backport
> > > PR? (As.a committer, I know I can merge at any time) or can I merge
> when
> > > the precommit comes back okay?
> >
> > ---------
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.1 release

[Brahma Reddy Battula]

Hi Bilwa,

I have commented on the jira's you mentioned. Based on the stability we can
plan this.But needs to be merged ASAP.



On Fri, Feb 19, 2021 at 5:20 PM bilwa st  wrote:

> Hi Brahma,
>
> Can we have below features in 3.3.1 release? We have been using these
> features for a long time. They are stable and tested in bigger clusters.
>
> 1. Container reuse - https://issues.apache.org/jira/browse/MAPREDUCE-6749
> 2. Speculative attempts should not run on the same node -
> https://issues.apache.org/jira/browse/MAPREDUCE-7169
>
> Thanks,
> Bilwa
>
> On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula 
> wrote:
>
>> Sorry for the late reply..
>>
>> I will come up with a plan.. Please let me know if anybody has some
>> features/improvements/bugs that need to be included.
>>
>> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan  wrote:
>>
>> > Hi Wei-Chiu,
>> >
>> > What will be the next steps here for 3.3.1 planning?
>> >
>> > Thanks
>> > Sunil
>> >
>> > On Mon, Feb 8, 2021 at 11:56 PM Stack  wrote:
>> >
>> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
>> > > > > >
>> > > wrote:
>> > >
>> > > >
>> > > > Regarding blockers : how about we have a little hackathon where we
>> > try
>> > > > and get things in. This means a promise of review time from the
>> people
>> > > with
>> > > > commit rights and other people who understand the code (Stack?)
>> > > >
>> > > >
>> > >
>> > > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing).
>> > > Thanks,
>> > > S
>> > >
>> > >
>> > >
>> > >
>> > > > -steve
>> > > >
>> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena 
>> wrote:
>> > > >
>> > > > > +1
>> > > > > Just to mention we would need to release hadoop-thirdparty too
>> > before.
>> > > > > Presently we are using the snapshot version of it.
>> > > > >
>> > > > > -Ayush
>> > > > >
>> > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang > >
>> > > > wrote:
>> > > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've
>> > > > > accumulated
>> > > > > > more than 400 changes in the branch-3.3. A number of
>> downstreamers
>> > > are
>> > > > > > eagerly waiting for 3.3.1 which addresses the guava version
>> > conflict
>> > > > > issue.
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
>> > > > > >
>> > > > > > We should start the release work for 3.3.1 before the diff
>> becomes
>> > > even
>> > > > > > larger.
>> > > > > >
>> > > > > > I believe there are  currently only two real blockers for a
>> 3.3.1
>> > > > (using
>> > > > > > this filter
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
>> > > > > > )
>> > > > > >
>> > > > > >
>> > > > > >   1. HDFS-15566 <
>> https://issues.apache.org/jira/browse/HDFS-15566>
>> > > > > >   2.
>> > > > > >  1. HADOOP-17112 <
>> > > > https://issues.apache.org/jira/browse/HADOOP-17112
>> > > > > >
>> > > > > > 2.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Is there anyone who would volunteer to be the 3.3.1 RM?
>> > > > > >
>> > > > > > Also, the HowToRelease wiki does not describe the ARM build
>> > process.
>> > > > > That's
>> > > > > > going to be important for future releases.
>> > > > >
>> > > > >
>> -
>> > > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>

-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.1 release

[Brahma Reddy Battula]

Sorry for the late reply..

I will come up with a plan.. Please let me know if anybody has some
features/improvements/bugs that need to be included.

On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan  wrote:

> Hi Wei-Chiu,
>
> What will be the next steps here for 3.3.1 planning?
>
> Thanks
> Sunil
>
> On Mon, Feb 8, 2021 at 11:56 PM Stack  wrote:
>
> > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
>  > >
> > wrote:
> >
> > >
> > > Regarding blockers : how about we have a little hackathon where we
> try
> > > and get things in. This means a promise of review time from the people
> > with
> > > commit rights and other people who understand the code (Stack?)
> > >
> > >
> >
> > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing).
> > Thanks,
> > S
> >
> >
> >
> >
> > > -steve
> > >
> > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena  wrote:
> > >
> > > > +1
> > > > Just to mention we would need to release hadoop-thirdparty too
> before.
> > > > Presently we are using the snapshot version of it.
> > > >
> > > > -Ayush
> > > >
> > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've
> > > > accumulated
> > > > > more than 400 changes in the branch-3.3. A number of downstreamers
> > are
> > > > > eagerly waiting for 3.3.1 which addresses the guava version
> conflict
> > > > issue.
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
> > > > >
> > > > > We should start the release work for 3.3.1 before the diff becomes
> > even
> > > > > larger.
> > > > >
> > > > > I believe there are  currently only two real blockers for a 3.3.1
> > > (using
> > > > > this filter
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
> > > > > )
> > > > >
> > > > >
> > > > >   1. HDFS-15566 <https://issues.apache.org/jira/browse/HDFS-15566>
> > > > >   2.
> > > > >  1. HADOOP-17112 <
> > > https://issues.apache.org/jira/browse/HADOOP-17112
> > > > >
> > > > > 2.
> > > > >
> > > > >
> > > > >
> > > > > Is there anyone who would volunteer to be the 3.3.1 RM?
> > > > >
> > > > > Also, the HowToRelease wiki does not describe the ARM build
> process.
> > > > That's
> > > > > going to be important for future releases.
> > > >
> > > > -
> > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > > >
> > > >
> > >
> >
>


-- 



--Brahma Reddy Battula

[jira] [Created] (HADOOP-17236) Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-17236:
-

 Summary: Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640
 Key: HADOOP-17236
 URL: https://issues.apache.org/jira/browse/HADOOP-17236
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Brahma Reddy Battula


Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Virtual MEETUP]: Migration to Hadoop 3

[Brahma Reddy Battula]

One more update from me.

We didn't face any issues with YARN, for HDFS you can have a look at the
following jira's.

https://issues.apache.org/jira/browse/HDFS-13596
https://issues.apache.org/jira/browse/HDFS-14396
https://issues.apache.org/jira/browse/HDFS-14509

Following jira is incompatible for ACL commands.Only hadoop-3 clients will
work against hadoop-3 server during the upgrade.

https://issues.apache.org/jira/browse/HDFS-6984



On Wed, Aug 26, 2020 at 11:06 PM Brahma Reddy Battula 
wrote:

>
> Hi Eric,
>
> check the following references for the same.
>
> 01/02/2020 Didi talked about their large scale HDFS cluster upgrade
> experience.
>
> Slides:
> https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy
>
> Recording:
> https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl
>
> Didi studied two upgrade approaches from the community documentation:
> express upgrade and rolling upgrade. Rolling upgrade was selected.
>
> Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1
>
> https://techblog.yahoo.co.jp/entry/20191206786320/
>
> On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org 
> wrote:
>
>> Hello. Just a reminder that today I would like to invite you all to
>> discuss your
>> experiences migrating from Hadoop 2 to Hadoop 3.
>>
>> -Eric
>>
>> On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org <
>> epa...@apache.org> wrote:
>>
>> Hello everyone!
>>
>> We are considering migrating to Hadoop 3, and we would be very interested
>> to
>> hear about your experiences. If you have migrated from Hadoop 2 to Hadoop
>> 3
>> and can provide insights, please kindly consider attending the following:
>>
>> Date: Wednesday, Aug 26, 2020
>> Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M. GMT
>> Location: Zoom: https://cloudera.zoom.us/j/880548968
>>
>> Hope to see you there!
>>
>> Thank you!
>> Eric Payne
>> @ Verizon Media
>>
>> -------------
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [Virtual MEETUP]: Migration to Hadoop 3

[Brahma Reddy Battula]

Hi Eric,

check the following references for the same.

01/02/2020 Didi talked about their large scale HDFS cluster upgrade
experience.

Slides:
https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy

Recording:
https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl

Didi studied two upgrade approaches from the community documentation:
express upgrade and rolling upgrade. Rolling upgrade was selected.

Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1

https://techblog.yahoo.co.jp/entry/20191206786320/

On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org  wrote:

> Hello. Just a reminder that today I would like to invite you all to
> discuss your
> experiences migrating from Hadoop 2 to Hadoop 3.
>
> -Eric
>
> On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org <
> epa...@apache.org> wrote:
>
> Hello everyone!
>
> We are considering migrating to Hadoop 3, and we would be very interested
> to
> hear about your experiences. If you have migrated from Hadoop 2 to Hadoop 3
> and can provide insights, please kindly consider attending the following:
>
> Date: Wednesday, Aug 26, 2020
> Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M. GMT
> Location: Zoom: https://cloudera.zoom.us/j/880548968
>
> Hope to see you there!
>
> Thank you!
> Eric Payne
> @ Verizon Media
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

-- 



--Brahma Reddy Battula

Re: Mandarin Hadoop online sync this week

[Brahma Reddy Battula]

HI,

what you are planning for this week?

On Tue, Aug 25, 2020 at 10:18 PM Wei-Chiu Chuang  wrote:

> Hello,
>
> There hasn't been a Mandarin online sync for quite some time. I'd like to
> call for one this week:
>
> Date/time:
>
> 8/27 Thursday Beijing Time 1PM
> 8/26 Wednesday US Pacific Time 10PM
>
> Link:
> https://cloudera.zoom.us/j/880548968
>
> Past sync summary:
>
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit
>


-- 



--Brahma Reddy Battula

[jira] [Created] (HADOOP-17225) Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: CVE-2019-10172

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-17225:
-

 Summary: Update jackson-mapper-asl-1.9.13 to atlassian version to 
mitigate: CVE-2019-10172
 Key: HADOOP-17225
 URL: https://issues.apache.org/jira/browse/HADOOP-17225
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula


Currently jersey depends on the jackson, and upgradation of jersey from 1.X to 
2.x looks complicated(see HADOOP-15984 and HADOOP-16485).

Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: 
CVE-2019-10172.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17221) Upgrade log4j-1.2.17 to atlassian ( To Adress: CVE-2019-17571)

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-17221:
-

 Summary: Upgrade log4j-1.2.17 to atlassian ( To Adress: 
CVE-2019-17571)
 Key: HADOOP-17221
 URL: https://issues.apache.org/jira/browse/HADOOP-17221
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Brahma Reddy Battula


Currentlly there are no active release under 1.X in log4j and log4j2 is 
incompatiable to upgrade (see HADOOP-16206 ) for more details.

But following CVE is reported on log4j 1.2.17..I think,we should consider to 
update to 
Atlassian([https://mvnrepository.com/artifact/log4j/log4j/1.2.17-atlassian-0.4])
 or redhat versions

[https://nvd.nist.gov/vuln/detail/CVE-2019-17571]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17220) Upgrade slf4j to 1.7.30 ( To Adress: CVE-2018-8088)

[Brahma Reddy Battula (Jira)]

Brahma Reddy Battula created HADOOP-17220:
-

 Summary: Upgrade slf4j to 1.7.30 ( To Adress: CVE-2018-8088)
 Key: HADOOP-17220
 URL: https://issues.apache.org/jira/browse/HADOOP-17220
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula


To address the following CVE, upgrade the slf4j to latest stable release 1.7.30.

[https://nvd.nist.gov/vuln/detail/CVE-2018-8088]
 

Note: We don't use EventData but should consider upgrading.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Progress

[Brahma Reddy Battula]

Looks still the problem with yetus isn't solved even after docker is
running and has the permissions.

As of now, I coped the old script from the build(
https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/8/console).

Currently it's running.
https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/8/console

On Sun, Jul 26, 2020 at 10:05 PM Akira Ajisaka  wrote:

> Hi Gavin,
>
> Thank you for moving the arm nodes. Would you install and start the Docker
> daemon on arm nodes?
>
> ```
> Got permission denied while trying to connect to the Docker daemon socket
> at unix:///var/run/docker.sock: Get
> http://%2Fvar%2Frun%2Fdocker.sock/v1.40/version:
> dial unix /var/run/docker.sock: connect: permission denied
>
> ```
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/3/console
>
> Thanks,
> Akira
>
> On Thu, Jul 23, 2020 at 11:47 AM Zhenyu Zheng 
> wrote:
>
> > Thanks for doing this, I was going to post in the JIRA about this
> yesterday
> > but was disturbed by something,
> > please help us migrate the ARM nodes.
> >
> > BR,
> >
> > On Thu, Jul 23, 2020 at 1:40 AM Gavin McDonald 
> > wrote:
> >
> > > Hi ,
> > >
> > > On Wed, Jul 22, 2020 at 7:00 PM Akira Ajisaka 
> > wrote:
> > >
> > > > +CC: common-dev@hadoop
> > > >
> > > > Hi Gavin,
> > > >
> > > > Thank you for the reminder.
> > > > I have one question. When ARM servers will be moved to ci-hadoop?
> > Hadoop
> > > > and HBase have daily jobs that run on ARM servers.
> > > >
> > > > I'll migrate the other Hadoop Jenkins jobs to ci-hadoop this weekend.
> > > >
> > >
> > > Thanks.
> > >
> > > I'll move 2 of the ARM nodes tomorrow
> > >
> > > HTH
> > >
> > >
> > > >
> > > > Regards,
> > > > Akira
> > > >
> > > > On Wed, Jul 22, 2020 at 8:39 PM Gavin McDonald  >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Seems there is still not much happening in the way of migrating to
> > > > > ci-hadoop
> > > > >
> > > > > Do you need any more information from me, and help needed?
> > > > >
> > > > > The cut off date is 15th August and builds.a.o will be turned off
> on
> > > that
> > > > > date.
> > > > >
> > > > > --
> > > > >
> > > > > *Gavin McDonald*
> > > > > Systems Administrator
> > > > > ASF Infrastructure Team
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > *Gavin McDonald*
> > > Systems Administrator
> > > ASF Infrastructure Team
> > >
> >
>


-- 



--Brahma Reddy Battula

[ANNOUNCE] Apache Hadoop 3.3.0 release

[Brahma Reddy Battula]

Greetings all,

It gives me great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.3.0.

Apache Hadoop 3.3.0 is the first release of Apache Hadoop 3.3 line for the
year 2020,
which includes 2148 fixes since the previous Hadoop 3.2.0 release.


Of these fixes:
   - 525 in Hadoop Common
   - 804 in HDFS
   - 763 in YARN
   - 56 in MapReduce

Apache Hadoop 3.3.0 contains a number of significant features and
enhancements.A few of them are noted as below.


- *Support ARM:* This is the first release to support ARM architectures

- *Upgrade protobuf from 2.5.0 : *Protobuf upgraded to 3.7.1 as
protobuf-2.5.0 reached EOL.

- *S3A Enhancements : *Lots of enhancements to the S3A code including
Delegation Token support, better handling of 404 caching, S3guard
performance, resilience improvements

- *ABFS Enhancements :* Address issues which surface in the field and tune
things which need tuning, add more tests where appropriate.Improve docs,
especially troubleshooting.

- *Tencent Cloud COS File System Implementation: *Tencent cloud is top
2 cloud vendors in China market and the object store COS is widely
used among China’s cloud users. COSN filesytem to support Tencent
cloud COS natively in Hadoop.

-  *Java11 Runtime Support* : Java 11 runtime support is completed.

- *HDFS RBF stabilizatio**n*: HDFS Router now supports security. Also
contains many bug fixes and improvements.

- *DNS Resolution to support Nameservices to IP Support:*DFS clients can
use a single domain name to discover servers (namenodes/routers/observers)
instead of explicitly listing out all hosts in the config

*- Scheduling of opportunistic containers :  S*cheduling of opportunistic
container through the central RM (YARN-5220), through distributed
scheduling (YARN-2877), as well as the scheduling of containers based on
actual node utilization (YARN-1011) and the container
promotion/demotion (YARN-5085).

*- Application Catalog for YARN applications: **Application catalog
system which provides an editorial and search interface for YARN
applications.*This improves the usability of YARN for managing the
life cycle of applications.

* For major changes included in Hadoop 3.3 line, please refer to Hadoop3.3.0
main page [1].


* For more details about fixes in 3.3.0 release, please read the CHANGELOG
[2] and RELEASENOTES [3].

The release news is posted on the Hadoop website too, you can go to the
downloads section directly [4].

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! This release is a direct result of your great
contributions.

Many thanks to Vinayakumar B who helped in this release process.

[1] https://hadoop.apache.org/docs <https://hadoop.apache.org/docs/r3.2.0/>
[2]
https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/CHANGELOG.3.3.0.html
[3]
https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/RELEASENOTES.3.3.0.html
[4] https://hadoop.apache.org/rele <https://hadoop.apache.org/releases.html>
ases.html





Regards,
Brahma Reddy Battula

Re: [RESULT][VOTE] Rlease Apache Hadoop-3.3.0

[Brahma Reddy Battula]

Hi Stephen,

thanks for bringing this to my attention.

Looks it's late..I pushed the release tag ( which can't be reverted) and
updated the release date in the jira.

Can we plan this next release near future..?


On Wed, Jul 15, 2020 at 5:25 PM Stephen O'Donnell
 wrote:

> Hi All,
>
> Sorry for being a bit late to this, but I wonder if we have a potential
> blocker to this release.
>
> In Cloudera we have recently encountered a serious dataloss issue in HDFS
> surrounding snapshots. To hit the dataloss issue, you must have HDFS-13101
> and HDFS-15012 on the build (which branch-3.3.0 does). To prevent it, you
> must also have HDFS-15313 and unfortunately, this was only committed to
> trunk, so we need to cherry-pick it down the active branches.
>
> With data loss being a serious issue, should we pull this Jira into
> branch-3.3.0 and cut a new release candidate?
>
> Thanks,
>
> Stephen.
>
> On Tue, Jul 14, 2020 at 1:22 PM Brahma Reddy Battula 
> wrote:
>
> > Hi All,
> >
> > With 8 binding and 11 non-binding +1s and no -1s the vote for Apache
> > hadoop-3.3.0 Release
> > passes.
> >
> > Thank you everybody for contributing to the release, testing, and voting.
> >
> > Special thanks whoever verified the ARM Binary as this is the first
> release
> > to support the ARM in hadoop.
> >
> >
> > Binding +1s
> >
> > =====
> > Akira Ajisaka
> > Vinayakumar B
> > Inigo Goiri
> > Surendra Singh Lilhore
> > Masatake Iwasaki
> > Rakesh Radhakrishnan
> > Eric Badger
> > Brahma Reddy Battula
> >
> > Non-binding +1s
> >
> > =
> > Zhenyu Zheng
> > Sheng Liu
> > Yikun Jiang
> > Tianhua huang
> > Ayush Saxena
> > Hemanth Boyina
> > Bilwa S T
> > Takanobu Asanuma
> > Xiaoqiao He
> > CR Hota
> > Gergely Pollak
> >
> > I'm going to work on staging the release.
> >
> >
> > The voting thread is:
> >
> >  https://s.apache.org/hadoop-3.3.0-Release-vote-thread
> >
> >
> >
> > --Brahma Reddy Battula
> >
>


-- 



--Brahma Reddy Battula

[RESULT][VOTE] Rlease Apache Hadoop-3.3.0

[Brahma Reddy Battula]

Hi All,

With 8 binding and 11 non-binding +1s and no -1s the vote for Apache
hadoop-3.3.0 Release
passes.

Thank you everybody for contributing to the release, testing, and voting.

Special thanks whoever verified the ARM Binary as this is the first release
to support the ARM in hadoop.


Binding +1s

=
Akira Ajisaka
Vinayakumar B
Inigo Goiri
Surendra Singh Lilhore
Masatake Iwasaki
Rakesh Radhakrishnan
Eric Badger
Brahma Reddy Battula

Non-binding +1s

=
Zhenyu Zheng
Sheng Liu
Yikun Jiang
Tianhua huang
Ayush Saxena
Hemanth Boyina
Bilwa S T
Takanobu Asanuma
Xiaoqiao He
CR Hota
Gergely Pollak

I'm going to work on staging the release.


The voting thread is:

 https://s.apache.org/hadoop-3.3.0-Release-vote-thread



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0

[Brahma Reddy Battula]

Hi Xiaoqiao,

Thanks for bringing this to my attention.

It's too rare to occur this scenario and a couple of releases went after
HDFS-4882(broken jira).

So,You think it's fine to include in the next release? Or if you still
insist we can go for another RC...


On Mon, Jul 13, 2020 at 12:29 PM Xiaoqiao He  wrote:

> Thanks Brahma Reddy Battula for your great work here.
> Stephen fixed lease leak in namenode, and it is ready now:
> https://issues.apache.org/jira/browse/HDFS-14498.
> I think this affects 3.3.0-RC0. Would you check this?
> Sorry for reporting it so late.
>
> On Mon, Jul 13, 2020 at 2:53 PM Takanobu Asanuma 
> wrote:
>
>> +1(non-binding)
>>- verified checksums
>>- succeeded in building the package with OpenJDK 8
>>- started HDFS cluster with kerberos with OpenJDK 11
>>- verified Web UIs (NN, DN, Router)
>>- Ran some operations of Router-based Federation with Security
>>- Ran some operations of Erasure Coding
>>
>> Thanks,
>> Takanobu
>>
>> 
>> From: Bilwa S T 
>> Sent: Monday, July 13, 2020 2:23
>> To: Surendra Singh Lilhore; hemanth boyina
>> Cc: Brahma Reddy Battula; mapreduce-dev; Hdfs-dev; Hadoop Common; yarn-dev
>> Subject: RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0
>>
>> +1(non-binding)
>>
>> 1. Deployed 3 node cluster
>> 2. Browsed through Web UI (RM, NM)
>> 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort)
>> 4. Verified basic yarn commands
>>
>> Thanks,
>> Bilwa
>>
>> -Original Message-----
>> From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com]
>> Sent: 12 July 2020 18:32
>> To: hemanth boyina 
>> Cc: Iñigo Goiri ; Vinayakumar B <
>> vinayakum...@apache.org>; Brahma Reddy Battula ;
>> mapreduce-dev ; Hdfs-dev <
>> hdfs-...@hadoop.apache.org>; Hadoop Common ;
>> yarn-dev 
>> Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
>>
>> +1(binding)
>>
>> Deployed HDFS and Yarn Cluster
>> > Verified basic shell commands
>> > Ran some jobs
>> > Verified UI
>>
>> -Surendra
>>
>> On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina <
>> hemanthboyina...@gmail.com>
>> wrote:
>>
>> > +1(non-binding)
>> > Deployed Cluster with Namenodes and Router *)verified shell commands
>> > *)Executed various jobs *)Browsed UI's
>> >
>> >
>> > Thanks,
>> > HemanthBoyina
>> >
>> >
>> > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri,  wrote:
>> >
>> > > +1 (Binding)
>> > >
>> > > Deployed a cluster on Azure VMs with:
>> > > * 3 VMs with HDFS Namenodes and Routers
>> > > * 2 VMs with YARN Resource Managers
>> > > * 5 VMs with HDFS Datanodes and Node Managers
>> > >
>> > > Tests:
>> > > * Executed Tergagen+Terasort+Teravalidate.
>> > > * Executed wordcount.
>> > > * Browsed through the Web UI.
>> > >
>> > >
>> > >
>> > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B
>> > > 
>> > > wrote:
>> > >
>> > > > +1 (Binding)
>> > > >
>> > > > -Verified all checksums and Signatures.
>> > > > -Verified site, Release notes and Change logs
>> > > >   + May be changelog and release notes could be grouped based on
>> > > > the project at second level for better look (this needs to be
>> > > > supported
>> > from
>> > > > yetus)
>> > > > -Tested in x86 local 3-node docker cluster.
>> > > >   + Built from source with OpenJdk 8 and Ubuntu 18.04
>> > > >   + Deployed 3 node docker cluster
>> > > >   + Ran various Jobs (wordcount, Terasort, Pi, etc)
>> > > >
>> > > > No Issues reported.
>> > > >
>> > > > -Vinay
>> > > >
>> > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu 
>> > > wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages
>> > > > >
>> > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04
>> > > > > ARM/aarch64, openjdk-11-jdk
>> > > > >
>> > > > > - checked some web UIs (NN, DN, RM, NM)
>> > > > >
>> > > > > - Executed a wo

[VOTE] Release Apache Hadoop 3.3.0 - RC0

[Brahma Reddy Battula]

Hi folks,

This is the first release candidate for the first release of Apache
Hadoop 3.3.0
line.

It contains *1644[1]* fixed jira issues since 3.2.1 which include a lot of
features and improvements(read the full set of release notes).

Below feature additions are the highlights of the release.

- ARM Support
- Enhancements and new features on S3a,S3Guard,ABFS
- Java 11 Runtime support and TLS 1.3.
- Support Tencent Cloud COS File System.
- Added security to HDFS Router.
- Support non-volatile storage class memory(SCM) in HDFS cache directives
- Support Interactive Docker Shell for running Containers.
- Scheduling of opportunistic containers
- A pluggable device plugin framework to ease vendor plugin development

*The RC0 artifacts are at*: http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/

*First release to include ARM binary, Have a check.*
*RC tag is *release-3.3.0-RC0.


*The maven artifacts are hosted here:*
https://repository.apache.org/content/repositories/orgapachehadoop-1271/

*My public key is available here:*
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM IST.


I have done a few testing with my pseudo cluster. My +1 to start.



Regards,
Brahma Reddy Battula


1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.3.0) AND
fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER BY
fixVersion ASC

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Yes, All the blockers are closed.will cut RC soon..

On Thu, Jun 18, 2020 at 7:49 PM Adam Antal 
wrote:

> YARN-10314 is also merged. I don't see any blockers at this point.
> (Actually I couldn't see any jiras
> <
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%203.3.0%20ORDER%20BY%20priority%20DESC
> >
> targeted for 3.3.0).
>
> In the community sync yesterday we wanted to discuss the 3.3.0 release, but
> nobody had information about it in the call. Could you share the latest on
> the upcoming 3.3.0 release?
>
> Thanks,
> Adam
>
> On Mon, Jun 15, 2020 at 9:17 AM Ayush Saxena  wrote:
>
> > YARN-10314 also seems to be a blocker.
> >
> > https://issues.apache.org/jira/browse/YARN-10314
> >
> > We should wait for that as well, should get concluded in a day or two.
> >
> > -Ayush
> >
> > > On 15-Jun-2020, at 7:21 AM, Sheng Liu  wrote:
> > >
> > > The  HADOOP-17046 <https://issues.apache.org/jira/browse/HADOOP-17046
> >
> > has
> > > been merged :)
> > >
> > > Brahma Reddy Battula  于2020年6月4日周四 下午10:43写道：
> > >
> > >> Following blocker is pending for 3.3.0 release which is ready for
> > review.
> > >> Mostly we'll have RC soon.
> > >> https://issues.apache.org/jira/browse/HADOOP-17046
> > >>
> > >> Protobuf dependency was unexpected .
> > >>
> > >>> On Mon, Jun 1, 2020 at 7:11 AM Sheng Liu 
> > wrote:
> > >>>
> > >>> Hi folks,
> > >>>
> > >>> It looks like the 3.3.0 branch has been created for quite a while.
> Not
> > >> sure
> > >>> if there is remain block issue that need to be addressed before
> Hadoop
> > >>> 3.3.0 release publishing, maybe we can bring up to here and move the
> > >>> release forward ?
> > >>>
> > >>> Thank.
> > >>>
> > >>> Brahma Reddy Battula  于2020年3月25日周三 上午1:55写道：
> > >>>
> > >>>> thanks to all.
> > >>>>
> > >>>> will make this as optional..will update the wiki accordingly.
> > >>>>
> > >>>> On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B <
> > >> vinayakum...@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>>> Making ARM artifact optional, makes the release process simpler for
> > >> RM
> > >>>> and
> > >>>>> unblocks release process (if there is unavailability of ARM
> > >> resources).
> > >>>>>
> > >>>>> Still there are possible options to collaborate with RM ( as brahma
> > >>>>> mentioned earlier) and provide ARM artifact may be before or after
> > >>> vote.
> > >>>>> If feasible RM can decide to add ARM artifact by collaborating with
> > >>>> @Brahma
> > >>>>> Reddy Battula  or me to get the ARM artifact.
> > >>>>>
> > >>>>> -Vinay
> > >>>>>
> > >>>>> On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
> > >>>>>  wrote:
> > >>>>>
> > >>>>>> Thanks for the clarification Brahma. Can you update the proposal
> to
> > >>>> state
> > >>>>>> that it is optional (it may help to put the proposal on cwiki)?
> > >>>>>>
> > >>>>>> Also if we go ahead then the RM documentation should be clear this
> > >> is
> > >>>> an
> > >>>>>> optional step.
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula <
> > >>>> bra...@apache.org>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>> Sure, we can't make mandatory while voting and we can upload to
> > >>>>> downloads
> > >>>>>>> once release vote is passed.
> > >>>>>>>
> > >>>>>>> On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> > >>>>>>>  wrote:
> > >>>>>>>
> > >>>>>>>>> Sorry,didn't get you...do you mean, once release v

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Following blocker is pending for 3.3.0 release which is ready for review.
Mostly we'll have RC soon.
https://issues.apache.org/jira/browse/HADOOP-17046

Protobuf dependency was unexpected .

On Mon, Jun 1, 2020 at 7:11 AM Sheng Liu  wrote:

> Hi folks,
>
> It looks like the 3.3.0 branch has been created for quite a while. Not sure
> if there is remain block issue that need to be addressed before Hadoop
> 3.3.0 release publishing, maybe we can bring up to here and move the
> release forward ?
>
> Thank.
>
> Brahma Reddy Battula  于2020年3月25日周三 上午1:55写道：
>
> > thanks to all.
> >
> > will make this as optional..will update the wiki accordingly.
> >
> > On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B 
> > wrote:
> >
> > > Making ARM artifact optional, makes the release process simpler for RM
> > and
> > > unblocks release process (if there is unavailability of ARM resources).
> > >
> > > Still there are possible options to collaborate with RM ( as brahma
> > > mentioned earlier) and provide ARM artifact may be before or after
> vote.
> > > If feasible RM can decide to add ARM artifact by collaborating with
> > @Brahma
> > > Reddy Battula  or me to get the ARM artifact.
> > >
> > > -Vinay
> > >
> > > On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
> > >  wrote:
> > >
> > > > Thanks for the clarification Brahma. Can you update the proposal to
> > state
> > > > that it is optional (it may help to put the proposal on cwiki)?
> > > >
> > > > Also if we go ahead then the RM documentation should be clear this is
> > an
> > > > optional step.
> > > >
> > > >
> > > > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula <
> > bra...@apache.org>
> > > > wrote:
> > > > >
> > > > > Sure, we can't make mandatory while voting and we can upload to
> > > downloads
> > > > > once release vote is passed.
> > > > >
> > > > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> > > > >  wrote:
> > > > >
> > > > >>> Sorry,didn't get you...do you mean, once release voting is
> > > > >>> processed and upload by RM..?
> > > > >>
> > > > >> Yes, that is what I meant. I don’t want us to make more mandatory
> > work
> > > > for
> > > > >> the release manager because the job is hard enough already.
> > > > >>
> > > > >>
> > > > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula <
> > > bra...@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>> Sorry,didn't get you...do you mean, once release voting is
> > processed
> > > > and
> > > > >>> upload by RM..?
> > > > >>>
> > > > >>> FYI. There is docker image for ARM also which support all scripts
> > > > >>> (createrelease, start-build-env.sh, etc ).
> > > > >>>
> > > > >>> https://issues.apache.org/jira/browse/HADOOP-16797
> > > > >>>
> > > > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> > > > >>>  wrote:
> > > > >>>
> > > > >>>> Can ARM binaries be provided after the fact? We cannot increase
> > the
> > > > RM’s
> > > > >>>> burden by asking them to generate an extra set of binaries.
> > > > >>>>
> > > > >>>>
> > > > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> > > > bra...@apache.org>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>> + Dev mailing list.
> > > > >>>>>
> > > > >>>>> -- Forwarded message -
> > > > >>>>> From: Brahma Reddy Battula 
> > > > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
> > > > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > > > >>>>> To: junping_du 
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> thanks junping for your reply.
> > > > >>>>>
> > > > >>>>> bq.  I think mos

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Brahma Reddy Battula]

gt; > > incompatible types: com.google.protobuf.ServiceException cannot be
> > > > > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
> > > > > 17:01:19 2020/04/16 00:01:19 INFO: [INFO] 2 errors
> > > > >
> > > > >
> > > > > There's another error where Ozone uses the Hadoop RPC framework
> which
> > > > uses
> > > > > the hadoop.thirdparty protobuf.
> > > > >
> > > > > [ERROR] Failed to execute goal
> > > > > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > > > > (default-testCompile) on project hadoop-hdds-container-service:
> > > > Compilation
> > > > > failure
> > > > > [ERROR]
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/weichiu/sandbox/ozone/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java:[103,41]
> > > > > incompatible types: com.google.protobuf.BlockingService cannot be
> > > > converted
> > > > > to org.apache.hadoop.thirdparty.protobuf.BlockingService
> > > > >
> > > > > BlockingService scmDatanodeService =
> > > > > StorageContainerDatanodeProtocolService.
> > > > > newReflectiveBlockingService(
> > > > >     new
> > StorageContainerDatanodeProtocolServerSideTranslatorPB(
> > > > > server,
> Mockito.mock(ProtocolMessageMetrics.class)));
> > > > >
> > > > >
> > > > >
> > > > > Ratis probably breaks as well since it depends on the Hadoop RPC
> > > > framework
> > > > > too.
> > > > >
> > > > > On Tue, Apr 28, 2020 at 10:58 PM Vinayakumar B <
> > > vinayakum...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > hi Wei-Chiu,
> > > > > >
> > > > > > Can you elaborate on what failures you are facing related to
> > > relocated
> > > > > > protobuf classes.. ?
> > > > > >
> > > > > > IFAIK, if the issue with location of protobuf classes, still old
> > jar
> > > > > > protobuf-2.5.0.jar will be available in classpath. So downstream
> > > > > depending
> > > > > > on 2.5.0 version of protobuf still be able to access them.
> > > > > >
> > > > > > -vinay
> > > > > >
> > > > > > On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, <
> > weic...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> I'm sorry for coming to this late. I missed this message. It
> > should
> > > > have
> > > > > >> been a DISCUSS thread rather than NOTICE.
> > > > > >>
> > > > > >> Looks like this is inevitable. But we should make the downstream
> > > > > >> developers aware & make the update easier. As long as it is
> stated
> > > > > clearly
> > > > > >> how to update the code to support Hadoop 3.3, I am okay with
> that.
> > > > > >>
> > > > > >> Here's what I suggest:
> > > > > >> (1) label the jira incompatible (just updated the jira) and
> > updated
> > > > the
> > > > > >> release note to tell app developer how to update.
> > > > > >> (2) declare ProtobufHelper a public API HADOOP-17019
> > > > > >> <https://issues.apache.org/jira/browse/HADOOP-17019>
> > > > > >>
> > > > > >> Tez doesn't use the removed Token API, but there's code that
> > breaks
> > > > with
> > > > > >> the relocated protobuf class. The ProtobufHelper API will make
> > this
> > > > > >> transition much easier.
> > > > > >>
> > > > > >> Other downstreamers that break with the relocated protobuf
> > include:
> > > > > Ozone
> > > > > >> and HBase. but neither of them use the removed Token API.
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Jan 8, 2020 at 4:40 AM Vinayakumar B <
> > > vinayakum...@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi All,
> > > > > >>>
> > > > > >>>This mail is to notify about the Removal of following public
> > > APIs
> > > > > from
> > > > > >>> Hadoop Common.
> > > > > >>>
> > > > > >>>  ClassName: org.apache.hadoop.security.token.Token
> > > > > >>>  APIs:
> > > > > >>>  public Token(TokenProto tokenPB);
> > > > > >>>  public TokenProto toTokenProto();
> > > > > >>>
> > > > > >>>Reason: These APIs are having Generated protobuf classes in
> > the
> > > > > >>> signature. Right now due to protobuf upgrade in trunk (soon to
> be
> > > > 3.3.0
> > > > > >>> release) these APIs are breaking the downstream builds, even
> > though
> > > > > >>> downstreams dont use these APIs (just Loading Token class).
> > > > Downstreams
> > > > > >>> are
> > > > > >>> still referencing having older version (2.5.0) of protobuf,
> hence
> > > > build
> > > > > >>> is
> > > > > >>> being broken.
> > > > > >>>
> > > > > >>> These APIs were added for the internal
> purpose(HADOOP-12563),
> > > to
> > > > > >>> support serializing tokens using protobuf in UGI Credentials.
> > > > > >>> Same purpose can be achieved using the Helper classes without
> > > > > introducing
> > > > > >>> protobuf classes in API signatures.
> > > > > >>>
> > > > > >>> Token.java is marked as Evolving, so I believe APIs can be
> > changed
> > > > > >>> whenever
> > > > > >>> absolute necessary.
> > > > > >>>
> > > > > >>> Jira https://issues.apache.org/jira/browse/HADOOP-16621
> has
> > > been
> > > > > >>> reported to solve downstream build failure.
> > > > > >>>
> > > > > >>> So since this API was added for internal purpose easy approach
> to
> > > > solve
> > > > > >>> this is to remove APIs and use helper classes. Otherwise, as
> > > > mentioned
> > > > > in
> > > > > >>> HADOOP-16621, workaround will add unnecessary codes to be
> > > maintained.
> > > > > >>>
> > > > > >>> If anyone using these APIs outside hadoop project accidentally,
> > > > please
> > > > > >>> reply to this mail immediately.
> > > > > >>>
> > > > > >>> If no objection by next week, will go ahead with removal of
> above
> > > > said
> > > > > >>> APIs
> > > > > >>> in HADOOP-16621.
> > > > > >>>
> > > > > >>> -Vinay
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] making Ozone a separate Apache project

[Brahma Reddy Battula]

+1


On Sat, 16 May 2020 at 12:09 AM, Subru Krishnan  wrote:

> +1.
>
> Thanks,
> Subru
>
> On Thu, May 14, 2020 at 10:00 PM Akira Ajisaka 
> wrote:
>
> > +1
> >
> > -Akira
> >
> > On Wed, May 13, 2020 at 4:53 PM Elek, Marton  wrote:
> >
> > >
> > >
> > > I would like to start a discussion to make a separate Apache project
> for
> > > Ozone
> > >
> > >
> > >
> > > ### HISTORY [1]
> > >
> > >   * Apache Hadoop Ozone development started on a feature branch of
> > > Hadoop repository (HDFS-7240)
> > >
> > >   * In the October of 2017 a discussion has been started to merge it to
> > > the Hadoop main branch
> > >
> > >   * After a long discussion it's merged to Hadoop trunk at the March of
> > > 2018
> > >
> > >   * During the discussion of the merge, it was suggested multiple times
> > > to create a separated project for the Ozone. But at that time:
> > >  1). Ozone was tightly integrated with Hadoop/HDFS
> > >  2). There was an active plan to use Block layer of Ozone (HDDS or
> > > HDSL at that time) as the block level of HDFS
> > >  3). The community of Ozone was a subset of the HDFS community
> > >
> > >   * The first beta release of Ozone was just released. Seems to be a
> > > good time before the first GA to make a decision about the future.
> > >
> > >
> > >
> > > ### WHAT HAS BEEN CHANGED
> > >
> > >   During the last years Ozone became more and more independent both at
> > > the community and code side. The separation has been suggested again
> and
> > > again (for example by Owen [2] and Vinod [3])
> > >
> > >
> > >
> > >   From COMMUNITY point of view:
> > >
> > >
> > >* Fortunately more and more new contributors are helping Ozone.
> > > Originally the Ozone community was a subset of HDFS project. But now a
> > > bigger and bigger part of the community is related to Ozone only.
> > >
> > >* It seems to be easier to _build_ the community as a separated
> > project.
> > >
> > >* A new, younger project might have different practices
> > > (communication, commiter criteria, development style) compared to old,
> > > mature project
> > >
> > >* It's easier to communicate (and improve) these standards in a
> > > separated projects with clean boundaries
> > >
> > >* Separated project/brand can help to increase the adoption rate and
> > > attract more individual contributor (AFAIK it has been seen in
> Submarine
> > > after a similar move)
> > >
> > >   * Contribution process can be communicated more easily, we can make
> > > first time contribution more easy
> > >
> > >
> > >
> > >   From CODE point of view Ozone became more and more independent:
> > >
> > >
> > >   * Ozone has different release cycle
> > >
> > >   * Code is already separated from Hadoop code base
> > > (apache/hadoop-ozone.git)
> > >
> > >   * It has separated CI (github actions)
> > >
> > >   * Ozone uses different (more strict) coding style (zero toleration of
> > > unit test / checkstyle errors)
> > >
> > >   * The code itself became more and more independent from Hadoop on
> > > Maven level. Originally it was compiled together with the in-tree
> latest
> > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > > Configuration...)
> > >
> > >   * It starts to use multiple version of Hadoop (on client side)
> > >
> > >   * Volume of resolved issues are already very high on Ozone side
> (Ozone
> > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > > together in the last 2-3 months)
> > >
> > >
> > > Summary: Before the first Ozone GA release, It seems to be a good time
> > > to discuss the long-term future of Ozone. Managing it as a separated
> TLP
> > > project seems to have more benefits.
> > >
> > >
> > > Please let me know what your opinion is...
> > >
> > > Thanks a lot,
> > > Marton
> > >
> > >
> > >
> > >
> > >
> > > [1]: For more details, see:
> > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> > >
> > > [2]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> > >
> > > [3]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>
-- 



--Brahma Reddy Battula

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Thanks Akira and Surendra for pointing that out.

Looks I missed after creating the branch..



On Thu, 30 Apr 2020 at 10:16 AM, Akira Ajisaka  wrote:

> Hi Surendra,
>
> Updated the version to 3.3.1-SNAPSHOT in branch-3.3.
>
> Thanks,
> Akira
>
> On Wed, Apr 29, 2020 at 4:22 PM Surendra Singh Lilhore <
> surendralilh...@gmail.com> wrote:
>
>> Hi Brahma,
>>
>> Why the branch-3.3 & branch-3.3.0 pom version is same ?.
>>
>> In branch-3.3 pom version should be 3.3.1.
>>
>> Please correct me if I am wrong.
>>
>> -Surendra
>>
>>
>> On Sat, 25 Apr, 2020, 9:33 am Mingliang Liu,  wrote:
>>
>> > Brahma,
>> >
>> > What about https://issues.apache.org/jira/browse/HADOOP-17007?
>> >
>> > Thanks,
>> >
>> > On Fri, Apr 24, 2020 at 11:07 AM Brahma Reddy Battula <
>> bra...@apache.org>
>> > wrote:
>> >
>> > > Ok. Done. Branch created.
>> > >
>> > > Following blockers are pending, will closely track this.
>> > >
>> > > https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under
>> > discussion
>> > > )
>> > > https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available)
>> > > https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available)
>> > > https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available)
>> > >
>> > >
>> > > On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang
>> > >  wrote:
>> > >
>> > > > +1 we should have the branch ASAP.
>> > > >
>> > > > On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka > >
>> > > > wrote:
>> > > >
>> > > > > > Since blockers are not closed, I didn't cut the branch because
>> > > > > multiple branches might confuse or sombody might miss to commit.
>> > > > >
>> > > > > The current situation is already confusing. The 3.3.1 version
>> already
>> > > > > exists in JIRA, so some committers wrongly commit non-critical
>> issues
>> > > to
>> > > > > branch-3.3 and set the fix version to 3.3.1.
>> > > > > I think now we should cut branch-3.3.0 and freeze source code
>> except
>> > > the
>> > > > > blockers.
>> > > > >
>> > > > > -Akira
>> > > > >
>> > > > > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula <
>> > > bra...@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > >> Sure, I will do that.
>> > > > >>
>> > > > >> Since blockers are not closed, I didn't cut the branch because
>> > > > >> multiple branches might confuse or sombody might miss to
>> > commit.Shall
>> > > I
>> > > > >> wait till this weekend to create..?
>> > > > >>
>> > > > >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka <
>> aajis...@apache.org
>> > >
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Hi Brahma,
>> > > > >>>
>> > > > >>> Thank you for preparing the release.
>> > > > >>> Could you cut branch-3.3.0? I would like to backport some fixes
>> for
>> > > > >>> 3.3.1 and not for 3.3.0.
>> > > > >>>
>> > > > >>> Thanks and regards,
>> > > > >>> Akira
>> > > > >>>
>> > > > >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula <
>> > > > bra...@apache.org>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Hi All,
>> > > > >>>>
>> > > > >>>> we are down to two blockers issues now (YARN-10194 and
>> YARN-9848)
>> > > > which
>> > > > >>>> are in patch available state.Hopefully we can out the RC soon.
>> > > > >>>>
>> > > > >>>> thanks to @Prabhu Joseph 
>> > > > ,@masakate,@akira
>> > > > >>>> and @Wei-Chiu Chuang   and others for
>> > helping
>> > > > >>>> resloving the blockers.
>> > > > >>>>
>> >

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Ok. Done. Branch created.

Following blockers are pending, will closely track this.

https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under discussion )
https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available)
https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available)
https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available)


On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang
 wrote:

> +1 we should have the branch ASAP.
>
> On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka 
> wrote:
>
> > > Since blockers are not closed, I didn't cut the branch because
> > multiple branches might confuse or sombody might miss to commit.
> >
> > The current situation is already confusing. The 3.3.1 version already
> > exists in JIRA, so some committers wrongly commit non-critical issues to
> > branch-3.3 and set the fix version to 3.3.1.
> > I think now we should cut branch-3.3.0 and freeze source code except the
> > blockers.
> >
> > -Akira
> >
> > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula 
> > wrote:
> >
> >> Sure, I will do that.
> >>
> >> Since blockers are not closed, I didn't cut the branch because
> >> multiple branches might confuse or sombody might miss to commit.Shall I
> >> wait till this weekend to create..?
> >>
> >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka 
> >> wrote:
> >>
> >>> Hi Brahma,
> >>>
> >>> Thank you for preparing the release.
> >>> Could you cut branch-3.3.0? I would like to backport some fixes for
> >>> 3.3.1 and not for 3.3.0.
> >>>
> >>> Thanks and regards,
> >>> Akira
> >>>
> >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula <
> bra...@apache.org>
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> we are down to two blockers issues now (YARN-10194 and YARN-9848)
> which
> >>>> are in patch available state.Hopefully we can out the RC soon.
> >>>>
> >>>> thanks to @Prabhu Joseph 
> ,@masakate,@akira
> >>>> and @Wei-Chiu Chuang   and others for helping
> >>>> resloving the blockers.
> >>>>
> >>>>
> >>>>
> >>>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula <
> >>>> bra...@apache.org> wrote:
> >>>>
> >>>>>
> >>>>> @Prabhu Joseph 
> >>>>> >>> Have committed the YARN blocker YARN-10219 to trunk and
> >>>>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras -
> >>>>> YARN-10233 and HADOOP-16982
> >>>>> which i will help to review and commit. Thanks.
> >>>>>
> >>>>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as
> a
> >>>>> blockers. (without YARN-10233 we have given so many releases,it's
> not newly
> >>>>> introduced.).. Thanks
> >>>>>
> >>>>> @Vinod Kumar Vavilapalli  ,@adam Antal,
> >>>>>
> >>>>> I noted YARN-9848 as a blocker as you mentioned above.
> >>>>>
> >>>>> @All,
> >>>>>
> >>>>> Currently following four blockers are pending for 3.3.0 RC.
> >>>>>
> >>>>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
> >>>>> vino...@apache.org> wrote:
> >>>>>
> >>>>>> Looks like a really bad bug to me.
> >>>>>>
> >>>>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should
> >>>>>> also revert it in a 3.2 maintenance release too.
> >>>>>>
> >>>>>> Thanks
> >>>>>> +Vinod
> >>>>>>
> >>>>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal  .INVALID>
> >>>>>> wrote:
> >>>>>> >
> >>>>>> > Hi everyone,
> >>>>>> >
> >>>>>> > Sorry for coming a bit late with this, but there's also one jira
> >>>>>> that can
> >>>>>> > have potential impact on clusters and we should talk about it.
> >>>>>> >
> >>>&g

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Sure, I will do that.

Since blockers are not closed, I didn't cut the branch because
multiple branches might confuse or sombody might miss to commit.Shall I
wait till this weekend to create..?

On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka  wrote:

> Hi Brahma,
>
> Thank you for preparing the release.
> Could you cut branch-3.3.0? I would like to backport some fixes for 3.3.1
> and not for 3.3.0.
>
> Thanks and regards,
> Akira
>
> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula 
> wrote:
>
>> Hi All,
>>
>> we are down to two blockers issues now (YARN-10194 and YARN-9848) which
>> are in patch available state.Hopefully we can out the RC soon.
>>
>> thanks to @Prabhu Joseph  ,@masakate,@akira
>> and @Wei-Chiu Chuang   and others for helping
>> resloving the blockers.
>>
>>
>>
>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula 
>> wrote:
>>
>>>
>>> @Prabhu Joseph 
>>> >>> Have committed the YARN blocker YARN-10219 to trunk and
>>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras -
>>> YARN-10233 and HADOOP-16982
>>> which i will help to review and commit. Thanks.
>>>
>>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
>>> blockers. (without YARN-10233 we have given so many releases,it's not newly
>>> introduced.).. Thanks
>>>
>>> @Vinod Kumar Vavilapalli  ,@adam Antal,
>>>
>>> I noted YARN-9848 as a blocker as you mentioned above.
>>>
>>> @All,
>>>
>>> Currently following four blockers are pending for 3.3.0 RC.
>>>
>>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
>>>
>>>
>>>
>>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
>>> vino...@apache.org> wrote:
>>>
>>>> Looks like a really bad bug to me.
>>>>
>>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should
>>>> also revert it in a 3.2 maintenance release too.
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
>>>> wrote:
>>>> >
>>>> > Hi everyone,
>>>> >
>>>> > Sorry for coming a bit late with this, but there's also one jira that
>>>> can
>>>> > have potential impact on clusters and we should talk about it.
>>>> >
>>>> > Steven Rand found this problem earlier and commented to
>>>> > https://issues.apache.org/jira/browse/YARN-4946.
>>>> > The bug has impact on the RM state store: the RM does not delete apps
>>>> - see
>>>> > more details in his comment here:
>>>> >
>>>> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
>>>> > .
>>>> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
>>>> with
>>>> > the revert task).
>>>> >
>>>> > It might not be an actual blocker, but since there wasn't any
>>>> consensus
>>>> > about a follow up action, I thought we should decide how to proceed
>>>> before
>>>> > release 3.3.0.
>>>> >
>>>> > Regards,
>>>> > Adam
>>>> >
>>>> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
>>>> prabhujose.ga...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Thanks Brahma for the update.
>>>> >>
>>>> >> Have committed the YARN blocker YARN-10219 to trunk and
>>>> cherry-picked to
>>>> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
>>>> >> HADOOP-16982
>>>> >> which i will help to review and commit. Thanks.
>>>> >>
>>>> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
>>>> >>
>>>> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>>> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
>>>> ORDER
>>>> >> BY priority DESC
>>>> >>
>>>> >>
>>>> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
>>>> bra...@apache.org>
>>>> >> wrote:
>>>>

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Hi All,

we are down to two blockers issues now (YARN-10194 and YARN-9848) which are
in patch available state.Hopefully we can out the RC soon.

thanks to @Prabhu Joseph 
,@masakate,@akira and @Wei-Chiu
Chuang   and others for helping resloving the
blockers.



On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula 
wrote:

>
> @Prabhu Joseph 
> >>> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked
> to branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
> HADOOP-16982
> which i will help to review and commit. Thanks.
>
> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
> blockers. (without YARN-10233 we have given so many releases,it's not newly
> introduced.).. Thanks
>
> @Vinod Kumar Vavilapalli  ,@adam Antal,
>
> I noted YARN-9848 as a blocker as you mentioned above.
>
> @All,
>
> Currently following four blockers are pending for 3.3.0 RC.
>
> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
>
>
>
> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> Looks like a really bad bug to me.
>>
>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should also
>> revert it in a 3.2 maintenance release too.
>>
>> Thanks
>> +Vinod
>>
>> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > Sorry for coming a bit late with this, but there's also one jira that
>> can
>> > have potential impact on clusters and we should talk about it.
>> >
>> > Steven Rand found this problem earlier and commented to
>> > https://issues.apache.org/jira/browse/YARN-4946.
>> > The bug has impact on the RM state store: the RM does not delete apps -
>> see
>> > more details in his comment here:
>> >
>> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
>> > .
>> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
>> with
>> > the revert task).
>> >
>> > It might not be an actual blocker, but since there wasn't any consensus
>> > about a follow up action, I thought we should decide how to proceed
>> before
>> > release 3.3.0.
>> >
>> > Regards,
>> > Adam
>> >
>> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
>> prabhujose.ga...@gmail.com>
>> > wrote:
>> >
>> >> Thanks Brahma for the update.
>> >>
>> >> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked
>> to
>> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
>> >> HADOOP-16982
>> >> which i will help to review and commit. Thanks.
>> >>
>> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
>> >>
>> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
>> ORDER
>> >> BY priority DESC
>> >>
>> >>
>> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
>> bra...@apache.org>
>> >> wrote:
>> >>
>> >>> *Pending for 3.3.0 Release:*
>> >>>
>> >>> One Blocker(HADOOP-16963) confirmation and following jira's are open
>> as
>> >>> these needs to merged to other branches(I am tracking the same,
>> Ideally
>> >>> this can be closed and can raise seperate jira's to track).
>> >>>
>> >>>
>> >>> 1–4 of 4Refresh results
>> >>> <
>> >>>
>> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
>> >>>>
>> >>> Columns
>> >>> Patch InfoKeyTSummaryAssigneeReporterP
>> StatusResolutionUpdatedDueCreated
>> >>> <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353
>> >>> <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task]
>> >>> <https://issues.apache.org/jira/browse/HDFS-14353>
>> >>>
>> >>> HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure
>> >>> Coding:
>> >>> metrics xmitsInProgress become to neg

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

@Prabhu Joseph 
>>> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked
to branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
HADOOP-16982
which i will help to review and commit. Thanks.

Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
blockers. (without YARN-10233 we have given so many releases,it's not newly
introduced.).. Thanks

@Vinod Kumar Vavilapalli  ,@adam Antal,

I noted YARN-9848 as a blocker as you mentioned above.

@All,

Currently following four blockers are pending for 3.3.0 RC.

HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.



On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli 
wrote:

> Looks like a really bad bug to me.
>
> +1 for revert and +1 for making that a 3.3.0 blocker. I think should also
> revert it in a 3.2 maintenance release too.
>
> Thanks
> +Vinod
>
> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
> wrote:
> >
> > Hi everyone,
> >
> > Sorry for coming a bit late with this, but there's also one jira that can
> > have potential impact on clusters and we should talk about it.
> >
> > Steven Rand found this problem earlier and commented to
> > https://issues.apache.org/jira/browse/YARN-4946.
> > The bug has impact on the RM state store: the RM does not delete apps -
> see
> > more details in his comment here:
> >
> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
> > .
> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
> with
> > the revert task).
> >
> > It might not be an actual blocker, but since there wasn't any consensus
> > about a follow up action, I thought we should decide how to proceed
> before
> > release 3.3.0.
> >
> > Regards,
> > Adam
> >
> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
> prabhujose.ga...@gmail.com>
> > wrote:
> >
> >> Thanks Brahma for the update.
> >>
> >> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked to
> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
> >> HADOOP-16982
> >> which i will help to review and commit. Thanks.
> >>
> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
> >>
> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
> ORDER
> >> BY priority DESC
> >>
> >>
> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
> bra...@apache.org>
> >> wrote:
> >>
> >>> *Pending for 3.3.0 Release:*
> >>>
> >>> One Blocker(HADOOP-16963) confirmation and following jira's are open as
> >>> these needs to merged to other branches(I am tracking the same, Ideally
> >>> this can be closed and can raise seperate jira's to track).
> >>>
> >>>
> >>> 1–4 of 4Refresh results
> >>> <
> >>>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
> >>>>
> >>> Columns
> >>> Patch InfoKeyTSummaryAssigneeReporterP
> StatusResolutionUpdatedDueCreated
> >>> <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353
> >>> <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task]
> >>> <https://issues.apache.org/jira/browse/HDFS-14353>
> >>>
> >>> HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure
> >>> Coding:
> >>> metrics xmitsInProgress become to negative.
> >>> <https://issues.apache.org/jira/browse/HDFS-14353>
> >>> maobaolong
> >>> <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> >>> maobaolong
> >>> <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> >>> [image:
> >>> Major] REOPENED *Unresolved* 11/Apr/20   11/Mar/19 Actions
> >>> <
> >>>
> https://issues.apache.org/jira/rest/api/1.0/issues/13220750/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> >>>>
> >>> <https://issues.apache.org/jira/browse/HDFS-14788> HDFS-14788
> >>> <https://issues.apache.org/jira/browse/HDFS-14

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Thanks Hexiaoqiao for taking care of this jira.Hope other's can do the
needful.

On Sun, Apr 12, 2020 at 11:25 AM Xiaoqiao He  wrote:

> Hi Brahma,
>
> Thanks for your great works here.
> About HDFS-12862 <https://issues.apache.org/jira/browse/HDFS-12862>, I try
> to raise another JIRA (HDFS-15272) to track backport branch-3.1, close this
> one to decouple release progress.
>
> Thanks,
> Hexiaoqiao
>
> On Sun, Apr 12, 2020 at 2:49 AM Brahma Reddy Battula 
> wrote:
>
> > *Pending for 3.3.0 Release:*
> >
> > One Blocker(HADOOP-16963) confirmation and following jira's are open as
> > these needs to merged to other branches(I am tracking the same, Ideally
> > this can be closed and can raise seperate jira's to track).
> >
> >
> > 1–4 of 4Refresh results
> > <
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
> > >
> > Columns
> > Patch InfoKeyTSummaryAssigneeReporterP StatusResolutionUpdatedDueCreated
> > <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353
> > <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task]
> > <https://issues.apache.org/jira/browse/HDFS-14353>
> >
> > HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure
> > Coding:
> > metrics xmitsInProgress become to negative.
> > <https://issues.apache.org/jira/browse/HDFS-14353>
> > maobaolong
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> > maobaolong
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong>
> > [image:
> > Major] REOPENED *Unresolved* 11/Apr/20   11/Mar/19 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13220750/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> > <https://issues.apache.org/jira/browse/HDFS-14788> HDFS-14788
> > <https://issues.apache.org/jira/browse/HDFS-14788> [image: Improvement]
> > <https://issues.apache.org/jira/browse/HDFS-14788>
> >
> > Use dynamic regex filter to ignore copy of source files in Distcp
> > <https://issues.apache.org/jira/browse/HDFS-14788>
> > Mukund Thakur
> > <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mukund-thakur
> > >
> > Mukund
> > Thakur
> > <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mukund-thakur
> > >
> > [image:
> > Major] OPEN *Unresolved* 11/Apr/20   28/Aug/19 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13253437/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> > <https://issues.apache.org/jira/browse/HDFS-12862> HDFS-12862
> > <https://issues.apache.org/jira/browse/HDFS-12862> [image: Bug]
> > <https://issues.apache.org/jira/browse/HDFS-12862>
> >
> > CacheDirective becomes invalid when NN restart or failover
> > <https://issues.apache.org/jira/browse/HDFS-12862>
> > Wang XL
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Wang%20XL>
> > Wang
> > XL <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Wang%20XL>
> > [image:
> > Major] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/17 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13121225/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> > <https://issues.apache.org/jira/browse/HDFS-14476> HDFS-14476
> > <https://issues.apache.org/jira/browse/HDFS-14476> [image: Improvement]
> > <https://issues.apache.org/jira/browse/HDFS-14476>
> >
> > lock too long when fix inconsistent blocks between disk and in-memory
> > <https://issues.apache.org/jira/browse/HDFS-14476>
> > Sean Chow
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=seanlook>
> > Sean
> > Chow <
> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=seanlook
> > >
> > [image:
> > Major] PATCH AVAILABLE *Unresolved* 12/Apr/20   07/May/19 Actions
> > <
> >
> https://issues.apache.org/jira/rest/api/1.0/issues/13231926/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
> > >
> >
> >
> > 1–5 of 5Refresh results
&

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

ache.org/jira/secure/ViewProfile.jspa?name=gandras> Szilard
Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
[image:
Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
<https://issues.apache.org/jira/rest/api/1.0/issues/13271201/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin>
<https://issues.apache.org/jira/browse/YARN-10002> YARN-10002
<https://issues.apache.org/jira/browse/YARN-10002> [image: Improvement]
<https://issues.apache.org/jira/browse/YARN-10002>

Code cleanup and improvements in ConfigurationStoreBaseTest
<https://issues.apache.org/jira/browse/YARN-10002>
Benjamin Teke
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke> Szilard
Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
[image:
Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
<https://issues.apache.org/jira/rest/api/1.0/issues/13271215/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin>
<https://issues.apache.org/jira/browse/YARN-9998> YARN-9998
<https://issues.apache.org/jira/browse/YARN-9998> [image: Improvement]
<https://issues.apache.org/jira/browse/YARN-9998>

Code cleanup in LeveldbConfigurationStore
<https://issues.apache.org/jira/browse/YARN-9998>
Benjamin Teke
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke> Szilard
Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
[image:
Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
<https://issues.apache.org/jira/rest/api/1.0/issues/13271202/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin>
<https://issues.apache.org/jira/browse/YARN-9354> YARN-9354
<https://issues.apache.org/jira/browse/YARN-9354> [image: Improvement]
<https://issues.apache.org/jira/browse/YARN-9354>

Resources sh <https://issues.apache.org/jira/browse/YARN-9354>

On Thu, Apr 9, 2020 at 11:41 PM Brahma Reddy Battula 
wrote:

> thanks a lot Akira and Prabhu for ping.
>
> All the blockers/critical issues are fixed. Recently two are reported
> again like HADOOP-16955 and HADOOP-16963 waiting for confirmation.
>
> @prabhu, Those 156 jira's are non-blocker issues which needs to moved to
> 3.4.0. and filter for blocker/critical is following.
>
> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
> BY priority DESC
>
>
> On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph 
> wrote:
>
>> Hi Brahma,
>>
>>  Please let me know if i can be of any help in review and commit /
>> clear the pending issues in YARN and MapReduce targeted for 3.3.0.
>> I see there are 156 YARN + MapReduce Unresolved Jiras which are either
>> with
>> target or fix version set to 3.3.0.
>>
>> project in (YARN, MAPREDUCE) AND resolution = Unresolved AND (cf[12310320]
>> = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC
>>
>> Thanks,
>> Prabhu Joseph
>>
>> On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka  wrote:
>>
>> > Hi Brahma,
>> >
>> > How is this issue going?
>> > If there are some blockers, I can help you.
>> >
>> > -Akira
>> >
>> > On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula > >
>> > wrote:
>> >
>> > > Hi All,
>> > >
>> > > branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.
>> > >
>> > > while committing jira's please set the proper fix version and if
>> there is
>> > > any blocker/critical please commit to branch-3.3 or let me know.
>> > >
>> > > Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
>> > > communication in another thread[2]).
>> > >
>> > >
>> > > 1.https://github.com/apache/hadoop/commits/branch-3.3
>> > > 2.
>> https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2020-3
>> > >
>> > >
>> > >
>> > >
>> > > --Brahma Reddy Battula
>> > >
>> >
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

thanks a lot Akira and Prabhu for ping.

All the blockers/critical issues are fixed. Recently two are reported again
like HADOOP-16955 and HADOOP-16963 waiting for confirmation.

@prabhu, Those 156 jira's are non-blocker issues which needs to moved to
3.4.0. and filter for blocker/critical is following.

project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
BY priority DESC


On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph 
wrote:

> Hi Brahma,
>
>  Please let me know if i can be of any help in review and commit /
> clear the pending issues in YARN and MapReduce targeted for 3.3.0.
> I see there are 156 YARN + MapReduce Unresolved Jiras which are either with
> target or fix version set to 3.3.0.
>
> project in (YARN, MAPREDUCE) AND resolution = Unresolved AND (cf[12310320]
> = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC
>
> Thanks,
> Prabhu Joseph
>
> On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka  wrote:
>
> > Hi Brahma,
> >
> > How is this issue going?
> > If there are some blockers, I can help you.
> >
> > -Akira
> >
> > On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula 
> > wrote:
> >
> > > Hi All,
> > >
> > > branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.
> > >
> > > while committing jira's please set the proper fix version and if there
> is
> > > any blocker/critical please commit to branch-3.3 or let me know.
> > >
> > > Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
> > > communication in another thread[2]).
> > >
> > >
> > > 1.https://github.com/apache/hadoop/commits/branch-3.3
> > > 2.
> https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2020-3
> > >
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>


-- 



--Brahma Reddy Battula

Re: JENKINS is not posting build results in jiras

[Brahma Reddy Battula]

Thanks Akira!!

On Wed, Mar 25, 2020 at 8:22 AM Sunil Govindan  wrote:

> Thanks Akira for helping.
>
> Best Regards,
> Sunil
>
> On Wed, Mar 25, 2020 at 6:41 AM Akira Ajisaka  wrote:
>
> > Sorry it was the root cause. Reverted.
> > Now the precommit job can add comments to GitHub.
> > https://github.com/apache/hadoop/pull/1820#issuecomment-603545202
> >
> > Akira
> >
> > On Tue, Mar 24, 2020 at 8:53 PM Akira Ajisaka 
> wrote:
> >
> > > A few days ago I upgraded the build image and maybe it is the root
> cause.
> > > Created a pull request to revert the change to see if it is really the
> > > cause: https://github.com/apache/hadoop/pull/1911
> > >
> > > Thanks,
> > > Akira
> > >
> > > On Tue, Mar 24, 2020 at 8:21 PM Akira Ajisaka 
> > wrote:
> > >
> > >> Thanks Sunil for the report.
> > >>
> > >> The precommit jobs are failing to add comments to GitHub also.
> > >> I'm looking into the cause, and any help is appreciated.
> > >>
> > >> Thanks,
> > >> Akira
> > >>
> > >>
> > >> On Mon, Mar 23, 2020 at 7:39 PM Sunil Govindan 
> > wrote:
> > >>
> > >>> Hello
> > >>>
> > >>> From last few days, jenkins results are not getting published in
> jiras.
> > >>>
> > >>> For eg,
> > https://builds.apache.org/job/PreCommit-YARN-Build/25735/console
> > >>> has
> > >>> results. but I cannot see the same in YARN-9879. Similarly in the
> cases
> > >>> of YARN-10198 etc.
> > >>>
> > >>> I could see a message in console o/p as "Adding comment to JIRA". But
> > its
> > >>> not happening.
> > >>>
> > >>> Any ideas?
> > >>>
> > >>> Thanks
> > >>> Sunil
> > >>>
> > >>
> >
>


-- 



--Brahma Reddy Battula

[Hadoop-3.3 Release update]- branch-3.3 has created

[Brahma Reddy Battula]

Hi All,

branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.

while committing jira's please set the proper fix version and if there is
any blocker/critical please commit to branch-3.3 or let me know.

Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
communication in another thread[2]).


1.https://github.com/apache/hadoop/commits/branch-3.3
2.https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2020-3




--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

*Important update: *branch-3.3 has created. and now trunk is to target to
3.4.0 ( As per above plan). ( will send in another mail with subject and
post in slack group also.)

Please merge any critical or blocker issues to branch-3.3 also.

Thanks.

On Tue, Mar 24, 2020 at 11:34 PM Brahma Reddy Battula 
wrote:

>
> Critical and blockers issues came down[1].
>
> Planning the cut branch-3.3 this weekend and start release vote on next
> weekend.
>
>
>
> 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
> BY priority DESC
>
>
>
> On Thu, Feb 27, 2020 at 12:42 AM Brahma Reddy Battula 
> wrote:
>
>> Hi All,
>>
>> Inline with the original 3.3.0 communication proposal dated 8th Jan 2020,
>> I would like to provide more updates[1].
>>
>> We are approaching previously proposed code freeze date (March 10,2020).
>> So I would like to cut 3.3 branch on *10th March* and point existing *trunk
>> to 3.4* if there are no issues.
>>
>> *Current Release Plan:*
>>
>> *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's
>> almost done)
>> *Code freeze Date *: blockers/critical only, no improvements and
>> blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues
>> are there which are very old and I am tracking same.)
>> *Release Date*: March 15, 2020.
>>
>> *Please let me know if I missed anything.*
>>
>> 1.
>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
>> 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
>> BY priority DESC
>>
>>
>> On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula <
>> brbapa...@gmail.com> wrote:
>>
>>>
>>> Wiki was updated for 3.3
>>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
>>> .
>>>
>>>
>>> >I'll move out anything that isn't needed.
>>>
>>> thanks steve.
>>>
>>> > We need to fix the shaded protobuf in
>>> > Token issue to even get spark to compile.
>>>
>>> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621
>>>
>>> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran
>>>  wrote:
>>>
>>>> >
>>>> > 2. Features close to finish:
>>>> >
>>>> >
>>>> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
>>>> owner
>>>> > :  Steve Loughran)
>>>> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
>>>> > fixes. ( owner :  Steve Loughran)
>>>> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features.
>>>> (
>>>> > owner :  Steve Loughran)
>>>> >
>>>> > I'll move out anything that isn't needed.
>>>>
>>>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing
>>>> and
>>>> I've got some provisional tuning to do. That is -if things didn't work
>>>> in
>>>> the test/production deployments, I'd know about the regressions (e.g.
>>>> HADOOP-16751).
>>>>
>>>> This is S3A and ABFS code -no idea about the rest, and inevitably the
>>>> big
>>>> JAR changes will have surprises. We need to fix the shaded protobuf in
>>>> Token issue to even get spark to compile.
>>>>
>>>> -Steve
>>>>
>>>> >
>>>> >
>>>>
>>>
>>>
>>> --
>>> --Brahma Reddy Battula
>>>
>>
>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Critical and blockers issues came down[1].

Planning the cut branch-3.3 this weekend and start release vote on next
weekend.



1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
BY priority DESC



On Thu, Feb 27, 2020 at 12:42 AM Brahma Reddy Battula 
wrote:

> Hi All,
>
> Inline with the original 3.3.0 communication proposal dated 8th Jan 2020,
> I would like to provide more updates[1].
>
> We are approaching previously proposed code freeze date (March 10,2020).
> So I would like to cut 3.3 branch on *10th March* and point existing *trunk
> to 3.4* if there are no issues.
>
> *Current Release Plan:*
>
> *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's
> almost done)
> *Code freeze Date *: blockers/critical only, no improvements and
> blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues
> are there which are very old and I am tracking same.)
> *Release Date*: March 15, 2020.
>
> *Please let me know if I missed anything.*
>
> 1.
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
> 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
> BY priority DESC
>
>
> On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula 
> wrote:
>
>>
>> Wiki was updated for 3.3
>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0.
>>
>>
>> >I'll move out anything that isn't needed.
>>
>> thanks steve.
>>
>> > We need to fix the shaded protobuf in
>> > Token issue to even get spark to compile.
>>
>> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621
>>
>> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran 
>> wrote:
>>
>>> >
>>> > 2. Features close to finish:
>>> >
>>> >
>>> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
>>> owner
>>> > :  Steve Loughran)
>>> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
>>> > fixes. ( owner :  Steve Loughran)
>>> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
>>> > owner :  Steve Loughran)
>>> >
>>> > I'll move out anything that isn't needed.
>>>
>>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing
>>> and
>>> I've got some provisional tuning to do. That is -if things didn't work in
>>> the test/production deployments, I'd know about the regressions (e.g.
>>> HADOOP-16751).
>>>
>>> This is S3A and ABFS code -no idea about the rest, and inevitably the big
>>> JAR changes will have surprises. We need to fix the shaded protobuf in
>>> Token issue to even get spark to compile.
>>>
>>> -Steve
>>>
>>> >
>>> >
>>>
>>
>>
>> --
>> --Brahma Reddy Battula
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

thanks to all.

will make this as optional..will update the wiki accordingly.

On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B 
wrote:

> Making ARM artifact optional, makes the release process simpler for RM  and
> unblocks release process (if there is unavailability of ARM resources).
>
> Still there are possible options to collaborate with RM ( as brahma
> mentioned earlier) and provide ARM artifact may be before or after vote.
> If feasible RM can decide to add ARM artifact by collaborating with @Brahma
> Reddy Battula  or me to get the ARM artifact.
>
> -Vinay
>
> On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
>  wrote:
>
> > Thanks for the clarification Brahma. Can you update the proposal to state
> > that it is optional (it may help to put the proposal on cwiki)?
> >
> > Also if we go ahead then the RM documentation should be clear this is an
> > optional step.
> >
> >
> > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula 
> > wrote:
> > >
> > > Sure, we can't make mandatory while voting and we can upload to
> downloads
> > > once release vote is passed.
> > >
> > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> > >  wrote:
> > >
> > >>> Sorry,didn't get you...do you mean, once release voting is
> > >>> processed and upload by RM..?
> > >>
> > >> Yes, that is what I meant. I don’t want us to make more mandatory work
> > for
> > >> the release manager because the job is hard enough already.
> > >>
> > >>
> > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula <
> bra...@apache.org>
> > >> wrote:
> > >>>
> > >>> Sorry,didn't get you...do you mean, once release voting is processed
> > and
> > >>> upload by RM..?
> > >>>
> > >>> FYI. There is docker image for ARM also which support all scripts
> > >>> (createrelease, start-build-env.sh, etc ).
> > >>>
> > >>> https://issues.apache.org/jira/browse/HADOOP-16797
> > >>>
> > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> > >>>  wrote:
> > >>>
> > >>>> Can ARM binaries be provided after the fact? We cannot increase the
> > RM’s
> > >>>> burden by asking them to generate an extra set of binaries.
> > >>>>
> > >>>>
> > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> > bra...@apache.org>
> > >>>> wrote:
> > >>>>>
> > >>>>> + Dev mailing list.
> > >>>>>
> > >>>>> -- Forwarded message -
> > >>>>> From: Brahma Reddy Battula 
> > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
> > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > >>>>> To: junping_du 
> > >>>>>
> > >>>>>
> > >>>>> thanks junping for your reply.
> > >>>>>
> > >>>>> bq.  I think most of us in Hadoop community doesn't want to
> have
> > >>>> biased
> > >>>>> on ARM or any other platforms.
> > >>>>>
> > >>>>> Yes, release voting will be based on the source code.AFAIK,Binary
> we
> > >> are
> > >>>>> providing for user to easy to download and verify.
> > >>>>>
> > >>>>> bq. The only thing I try to understand is how much complexity
> get
> > >>>>> involved for our RM work. Does that potentially become a blocker
> for
> > >>>> future
> > >>>>> releases? And how we can get rid of this risk.
> > >>>>>
> > >>>>> As I mentioned earlier, RM need to access the ARM machine(it will
> be
> > >>>>> donated and current qbt also using one ARM machine) and build tar
> > using
> > >>>> the
> > >>>>> keys. As it can be common machine, RM can delete his keys once
> > release
> > >>>>> approved.
> > >>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM
> > >> machine)
> > >>>>>
> > >>>>> bq.   If you can list the concrete work that RM need to do
> extra
> > >> for
> > >>>>> ARM release, that would hel

Re: [RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Brahma Reddy Battula]

My late +1.. I missed this..


On Thu, 19 Mar 2020 at 12:43 AM, Vinayakumar B 
wrote:

> Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s
>
> Thanks everyone for verification.
> I will push out bits soon.
>
> -Vinay
>
>
> On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore <
> surendralilh...@gmail.com> wrote:
>
> > +1
> >
> > -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0
> > -Verified artifacts available in repo.
> >
> >
> > Thanks
> > Surendra
> >
> >
> > On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka 
> wrote:
> >
> >> +1
> >>
> >> - Verified signatures and checksums
> >> - Built jars and docs from source
> >> - Built hadoop trunk with hadoop-thirdparty 1.0.0
> >> - Checked rat files and documents
> >> - Checked LICENSE and NOTICE files
> >>
> >> Thanks,
> >> Akira
> >>
> >> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B 
> >> wrote:
> >>
> >> > Hi folks,
> >> >
> >> > Thanks to everyone's help on this release.
> >> >
> >> > I have re-created a release candidate (RC1) for Apache Hadoop
> Thirdparty
> >> > 1.0.0.
> >> >
> >> > RC Release artifacts are available at :
> >> >
> >> >
> >>
> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
> >> >
> >> > Maven artifacts are available in staging repo:
> >> >
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
> >> >
> >> > The RC tag in git is here:
> >> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
> >> >
> >> > And my public key is at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm
> >> IST.*
> >> >
> >> > For the testing, I have verified Hadoop trunk compilation with
> >> >"-DdistMgmtSnapshotsUrl=
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
> >> >  -Dhadoop-thirdparty-protobuf.version=1.0.0"
> >> >
> >> > My +1 to start.
> >> >
> >> > -Vinay
> >> >
> >>
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Sure, I will update in cwiki,Once it's concluded here..Thanks a lot arpit...

On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
 wrote:

> Thanks for the clarification Brahma. Can you update the proposal to state
> that it is optional (it may help to put the proposal on cwiki)?
>
> Also if we go ahead then the RM documentation should be clear this is an
> optional step.
>
>
> > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula 
> wrote:
> >
> > Sure, we can't make mandatory while voting and we can upload to downloads
> > once release vote is passed.
> >
> > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> >  wrote:
> >
> >>> Sorry,didn't get you...do you mean, once release voting is
> >>> processed and upload by RM..?
> >>
> >> Yes, that is what I meant. I don’t want us to make more mandatory work
> for
> >> the release manager because the job is hard enough already.
> >>
> >>
> >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
> >> wrote:
> >>>
> >>> Sorry,didn't get you...do you mean, once release voting is processed
> and
> >>> upload by RM..?
> >>>
> >>> FYI. There is docker image for ARM also which support all scripts
> >>> (createrelease, start-build-env.sh, etc ).
> >>>
> >>> https://issues.apache.org/jira/browse/HADOOP-16797
> >>>
> >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> >>>  wrote:
> >>>
> >>>> Can ARM binaries be provided after the fact? We cannot increase the
> RM’s
> >>>> burden by asking them to generate an extra set of binaries.
> >>>>
> >>>>
> >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> bra...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>> + Dev mailing list.
> >>>>>
> >>>>> -- Forwarded message -
> >>>>> From: Brahma Reddy Battula 
> >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
> >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> >>>>> To: junping_du 
> >>>>>
> >>>>>
> >>>>> thanks junping for your reply.
> >>>>>
> >>>>> bq.  I think most of us in Hadoop community doesn't want to have
> >>>> biased
> >>>>> on ARM or any other platforms.
> >>>>>
> >>>>> Yes, release voting will be based on the source code.AFAIK,Binary we
> >> are
> >>>>> providing for user to easy to download and verify.
> >>>>>
> >>>>> bq. The only thing I try to understand is how much complexity get
> >>>>> involved for our RM work. Does that potentially become a blocker for
> >>>> future
> >>>>> releases? And how we can get rid of this risk.
> >>>>>
> >>>>> As I mentioned earlier, RM need to access the ARM machine(it will be
> >>>>> donated and current qbt also using one ARM machine) and build tar
> using
> >>>> the
> >>>>> keys. As it can be common machine, RM can delete his keys once
> release
> >>>>> approved.
> >>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM
> >> machine)
> >>>>>
> >>>>> bq.   If you can list the concrete work that RM need to do extra
> >> for
> >>>>> ARM release, that would help us to better understand.
> >>>>>
> >>>>> I can write and update for future reference.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >>>>>
> >>>>>> Hi Brahma,
> >>>>>>   I think most of us in Hadoop community doesn't want to have biased
> >>>> on
> >>>>>> ARM or any other platforms.
> >>>>>>   The only thing I try to understand is how much complexity get
> >>>>>> involved for our RM work. Does that potentially become a blocker for
> >>>> future
> >>>>>> releases? And how we can get rid of this risk.
> >>>>>>If you can li

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Sure, we can't make mandatory while voting and we can upload to downloads
once release vote is passed.

On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
 wrote:

> > Sorry,didn't get you...do you mean, once release voting is
> > processed and upload by RM..?
>
> Yes, that is what I meant. I don’t want us to make more mandatory work for
> the release manager because the job is hard enough already.
>
>
> > On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
> wrote:
> >
> > Sorry,didn't get you...do you mean, once release voting is processed and
> > upload by RM..?
> >
> > FYI. There is docker image for ARM also which support all scripts
> > (createrelease, start-build-env.sh, etc ).
> >
> > https://issues.apache.org/jira/browse/HADOOP-16797
> >
> > On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> >  wrote:
> >
> >> Can ARM binaries be provided after the fact? We cannot increase the RM’s
> >> burden by asking them to generate an extra set of binaries.
> >>
> >>
> >>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula 
> >> wrote:
> >>>
> >>> + Dev mailing list.
> >>>
> >>> -- Forwarded message -
> >>> From: Brahma Reddy Battula 
> >>> Date: Tue, Mar 17, 2020 at 10:31 PM
> >>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> >>> To: junping_du 
> >>>
> >>>
> >>> thanks junping for your reply.
> >>>
> >>> bq.  I think most of us in Hadoop community doesn't want to have
> >> biased
> >>> on ARM or any other platforms.
> >>>
> >>> Yes, release voting will be based on the source code.AFAIK,Binary we
> are
> >>> providing for user to easy to download and verify.
> >>>
> >>> bq. The only thing I try to understand is how much complexity get
> >>> involved for our RM work. Does that potentially become a blocker for
> >> future
> >>> releases? And how we can get rid of this risk.
> >>>
> >>> As I mentioned earlier, RM need to access the ARM machine(it will be
> >>> donated and current qbt also using one ARM machine) and build tar using
> >> the
> >>> keys. As it can be common machine, RM can delete his keys once release
> >>> approved.
> >>> Can be sorted out as I mentioned earlier.(For accessing the ARM
> machine)
> >>>
> >>> bq.   If you can list the concrete work that RM need to do extra
> for
> >>> ARM release, that would help us to better understand.
> >>>
> >>> I can write and update for future reference.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >>>
> >>>> Hi Brahma,
> >>>>    I think most of us in Hadoop community doesn't want to have biased
> >> on
> >>>> ARM or any other platforms.
> >>>>The only thing I try to understand is how much complexity get
> >>>> involved for our RM work. Does that potentially become a blocker for
> >> future
> >>>> releases? And how we can get rid of this risk.
> >>>> If you can list the concrete work that RM need to do extra for ARM
> >>>> release, that would help us to better understand.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Junping
> >>>>
> >>>> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >>>>
> >>>>> If you can provide ARM release for future releases, I'm fine with
> that.
> >>>>>
> >>>>> Thanks,
> >>>>> Akira
> >>>>>
> >>>>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
> >> bra...@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> thanks Akira.
> >>>>>>
> >>>>>> Currently only problem is dedicated ARM for future RM.This i want to
> >>>>> sort
> >>>>>> out like below,if you've some other,please let me know.
> >>>>>>
> >>>>>> i) Single machine and share cred to future RM ( as we can delete
> keys
> >>>>> once
> >>>>>> release is over).
> >>>>>> ii) Creating the 

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Sorry,didn't get you...do you mean, once release voting is processed and
upload by RM..?

FYI. There is docker image for ARM also which support all scripts
(createrelease, start-build-env.sh, etc ).

https://issues.apache.org/jira/browse/HADOOP-16797

On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
 wrote:

> Can ARM binaries be provided after the fact? We cannot increase the RM’s
> burden by asking them to generate an extra set of binaries.
>
>
> > On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula 
> wrote:
> >
> > + Dev mailing list.
> >
> > -- Forwarded message -
> > From: Brahma Reddy Battula 
> > Date: Tue, Mar 17, 2020 at 10:31 PM
> > Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > To: junping_du 
> >
> >
> > thanks junping for your reply.
> >
> > bq.  I think most of us in Hadoop community doesn't want to have
> biased
> > on ARM or any other platforms.
> >
> > Yes, release voting will be based on the source code.AFAIK,Binary we are
> > providing for user to easy to download and verify.
> >
> > bq. The only thing I try to understand is how much complexity get
> > involved for our RM work. Does that potentially become a blocker for
> future
> > releases? And how we can get rid of this risk.
> >
> > As I mentioned earlier, RM need to access the ARM machine(it will be
> > donated and current qbt also using one ARM machine) and build tar using
> the
> > keys. As it can be common machine, RM can delete his keys once release
> > approved.
> > Can be sorted out as I mentioned earlier.(For accessing the ARM machine)
> >
> > bq.   If you can list the concrete work that RM need to do extra for
> > ARM release, that would help us to better understand.
> >
> > I can write and update for future reference.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >
> >> Hi Brahma,
> >> I think most of us in Hadoop community doesn't want to have biased
> on
> >> ARM or any other platforms.
> >> The only thing I try to understand is how much complexity get
> >> involved for our RM work. Does that potentially become a blocker for
> future
> >> releases? And how we can get rid of this risk.
> >>  If you can list the concrete work that RM need to do extra for ARM
> >> release, that would help us to better understand.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >>
> >>> If you can provide ARM release for future releases, I'm fine with that.
> >>>
> >>> Thanks,
> >>> Akira
> >>>
> >>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
> bra...@apache.org>
> >>> wrote:
> >>>
> >>>> thanks Akira.
> >>>>
> >>>> Currently only problem is dedicated ARM for future RM.This i want to
> >>> sort
> >>>> out like below,if you've some other,please let me know.
> >>>>
> >>>> i) Single machine and share cred to future RM ( as we can delete keys
> >>> once
> >>>> release is over).
> >>>> ii) Creating the jenkins project ( may be we need to discuss in the
> >>>> board..)
> >>>> iii) I can provide ARM release for future releases.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
> >>> wrote:
> >>>>
> >>>>> Hi Brahma,
> >>>>>
> >>>>> I think we cannot do any of your proposed actions.
> >>>>>
> >>>>>
> >>>>
> >>>
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> >>>>>> Strictly speaking, releases must be verified on hardware owned and
> >>>>> controlled by the committer. That means hardware the committer has
> >>>> physical
> >>>>> possession and control of and exclusively full
> >>> administrative/superuser
> >>>>> access to. That's because only such hardware is qualified to hold a
> >>> PGP
> >>>>> private key, and the release should be verified on the machine the
> >>>> private
> >>>>> key

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Thanks Masatake!!

I was aware of this thread which you given for reference as I am the source
to discuss this(as I verified binary and given some comments). Please check
following for same.

https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2017-7


AFAIK, that discussion whether we should vote ton he binary or not.Even
Andrew discussed with legal team [1] and finally it was concluded that vote
should only on source I think.

1. https://issues.apache.org/jira/browse/LEGAL-323


On Tue, Mar 17, 2020 at 11:23 AM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> This thread seems to be relevant.
>
> https://lists.apache.org/thread.html/0d2a1b39f7e890c4f40be5fd92f107fbf048b936005901b7b53dd0f1%40%3Ccommon-dev.hadoop.apache.org%3E
>
>  > Convenience binary artifacts are not official release artifacts and thus
>  > are not voted on. However, since they are distributed by Apache, they
> are
>  > still subject to the same distribution requirements as official release
>  > artifacts. This means they need to have a LICENSE and NOTICE file,
> follow
>  > ASF licensing rules, etc. The PMC needs to ensure that binary artifacts
>  > meet these requirements.
>  >
>  > However, being a "convenience" artifact doesn't mean it isn't important.
>  > The appropriate level of quality for binary artifacts is left up to the
>  > project. An OpenOffice person mentioned the quality of their binary
>  > artifacts is super important since very few of their users will compile
>  > their own office suite.
>  >
>  > I don't know if we've discussed the topic of binary artifact quality in
>  > Hadoop. My stance is that if we're going to publish something, it
> should be
>  > good, or we shouldn't publish it at all. I think we do want to publish
>  > binary tarballs (it's the easiest way for new users to get started with
>  > Hadoop), so it's fair to consider them when evaluating a release.
>
> Just providing build machine to RM would not be enough if
> PMC need to ensure that binary artifiacts meet these requirements.
>
> Thanks,
> Masatake Iwasaki
>
> On 3/17/20 14:11, 俊平堵 wrote:
> > Hi Brahma,
> >   I think most of us in Hadoop community doesn't want to have biased
> on
> > ARM or any other platforms.
> >   The only thing I try to understand is how much complexity get
> involved
> > for our RM work. Does that potentially become a blocker for future
> > releases? And how we can get rid of this risk.
> >If you can list the concrete work that RM need to do extra for ARM
> > release, that would help us to better understand.
> >
> > Thanks,
> >
> > Junping
> >
> > Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >
> >> If you can provide ARM release for future releases, I'm fine with that.
> >>
> >> Thanks,
> >> Akira
> >>
> >> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula  >
> >> wrote:
> >>
> >>> thanks Akira.
> >>>
> >>> Currently only problem is dedicated ARM for future RM.This i want to
> sort
> >>> out like below,if you've some other,please let me know.
> >>>
> >>> i) Single machine and share cred to future RM ( as we can delete keys
> >> once
> >>> release is over).
> >>> ii) Creating the jenkins project ( may be we need to discuss in the
> >>> board..)
> >>> iii) I can provide ARM release for future releases.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
> >> wrote:
> >>>> Hi Brahma,
> >>>>
> >>>> I think we cannot do any of your proposed actions.
> >>>>
> >>>>
> >>
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> >>>>> Strictly speaking, releases must be verified on hardware owned and
> >>>> controlled by the committer. That means hardware the committer has
> >>> physical
> >>>> possession and control of and exclusively full
> administrative/superuser
> >>>> access to. That's because only such hardware is qualified to hold a
> PGP
> >>>> private key, and the release should be verified on the machine the
> >>> private
> >>>> key lives on or on a machine as trusted as that.
> >>>>
> >>>> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> >>>>> Private keys MUST NOT be stored on an

Fwd: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

+ Dev mailing list.

-- Forwarded message -
From: Brahma Reddy Battula 
Date: Tue, Mar 17, 2020 at 10:31 PM
Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
To: junping_du 


thanks junping for your reply.

bq.  I think most of us in Hadoop community doesn't want to have biased
on ARM or any other platforms.

Yes, release voting will be based on the source code.AFAIK,Binary we are
providing for user to easy to download and verify.

bq. The only thing I try to understand is how much complexity get
involved for our RM work. Does that potentially become a blocker for future
releases? And how we can get rid of this risk.

As I mentioned earlier, RM need to access the ARM machine(it will be
donated and current qbt also using one ARM machine) and build tar using the
keys. As it can be common machine, RM can delete his keys once release
approved.
Can be sorted out as I mentioned earlier.(For accessing the ARM machine)

bq.   If you can list the concrete work that RM need to do extra for
ARM release, that would help us to better understand.

I can write and update for future reference.









On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:

> Hi Brahma,
>  I think most of us in Hadoop community doesn't want to have biased on
> ARM or any other platforms.
>  The only thing I try to understand is how much complexity get
> involved for our RM work. Does that potentially become a blocker for future
> releases? And how we can get rid of this risk.
>   If you can list the concrete work that RM need to do extra for ARM
> release, that would help us to better understand.
>
> Thanks,
>
> Junping
>
> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
>
>> If you can provide ARM release for future releases, I'm fine with that.
>>
>> Thanks,
>> Akira
>>
>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula 
>> wrote:
>>
>> > thanks Akira.
>> >
>> > Currently only problem is dedicated ARM for future RM.This i want to
>> sort
>> > out like below,if you've some other,please let me know.
>> >
>> > i) Single machine and share cred to future RM ( as we can delete keys
>> once
>> > release is over).
>> > ii) Creating the jenkins project ( may be we need to discuss in the
>> > board..)
>> > iii) I can provide ARM release for future releases.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
>> wrote:
>> >
>> > > Hi Brahma,
>> > >
>> > > I think we cannot do any of your proposed actions.
>> > >
>> > >
>> >
>> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
>> > > > Strictly speaking, releases must be verified on hardware owned and
>> > > controlled by the committer. That means hardware the committer has
>> > physical
>> > > possession and control of and exclusively full
>> administrative/superuser
>> > > access to. That's because only such hardware is qualified to hold a
>> PGP
>> > > private key, and the release should be verified on the machine the
>> > private
>> > > key lives on or on a machine as trusted as that.
>> > >
>> > > https://www.apache.org/dev/release-distribution.html#sigs-and-sums
>> > > > Private keys MUST NOT be stored on any ASF machine. Likewise,
>> > signatures
>> > > for releases MUST NOT be created on ASF machines.
>> > >
>> > > We need to have dedicated physical ARM machines for each release
>> manager,
>> > > and now it is not feasible.
>> > > If you provide an unofficial ARM binary release in some repository,
>> > that's
>> > > okay.
>> > >
>> > > -Akira
>> > >
>> > > On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula <
>> bra...@apache.org>
>> > > wrote:
>> > >
>> > >> Hello folks,
>> > >>
>> > >> As currently trunk will support ARM based compilation and qbt(1) is
>> > >> running
>> > >> from several months with quite stable, hence planning to propose ARM
>> > >> binary
>> > >> this time.
>> > >>
>> > >> ( Note : As we'll know voting will be based on the source,so this
>> will
>> > not
>> > >> issue.)
>> > >>
>> > >> *Proposed Change:*
>> > >> Currently in downloads we are keeping only x86 binary(2),Can we keep
>> ARM
>> > >> binary also.?
>> > >>
>> > >> *Actions:*
>> > >> a) *Dedicated* *Machine*:
>> > >>i) Dedicated ARM machine will be donated which I confirmed
>> > >>ii) Or can use jenkins ARM machine itself which is currently
>> used
>> > >> for ARM
>> > >> b) *Automate Release:* How about having one release project in
>> > jenkins..?
>> > >> So that future RM's just trigger the jenkin project.
>> > >>
>> > >> Please let me know your thoughts on this.
>> > >>
>> > >>
>> > >> 1.
>> > >>
>> > >>
>> >
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> > >> 2.https://hadoop.apache.org/releases.html
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --Brahma Reddy Battula
>> > >>
>> > >
>> >
>> > --
>> >
>> >
>> >
>> > --Brahma Reddy Battula
>> >
>>
>

-- 



--Brahma Reddy Battula


-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

thanks Akira.

Currently only problem is dedicated ARM for future RM.This i want to sort
out like below,if you've some other,please let me know.

i) Single machine and share cred to future RM ( as we can delete keys once
release is over).
ii) Creating the jenkins project ( may be we need to discuss in the board..)
iii) I can provide ARM release for future releases.







On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka  wrote:

> Hi Brahma,
>
> I think we cannot do any of your proposed actions.
>
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> > Strictly speaking, releases must be verified on hardware owned and
> controlled by the committer. That means hardware the committer has physical
> possession and control of and exclusively full administrative/superuser
> access to. That's because only such hardware is qualified to hold a PGP
> private key, and the release should be verified on the machine the private
> key lives on or on a machine as trusted as that.
>
> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> > Private keys MUST NOT be stored on any ASF machine. Likewise, signatures
> for releases MUST NOT be created on ASF machines.
>
> We need to have dedicated physical ARM machines for each release manager,
> and now it is not feasible.
> If you provide an unofficial ARM binary release in some repository, that's
> okay.
>
> -Akira
>
> On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula 
> wrote:
>
>> Hello folks,
>>
>> As currently trunk will support ARM based compilation and qbt(1) is
>> running
>> from several months with quite stable, hence planning to propose ARM
>> binary
>> this time.
>>
>> ( Note : As we'll know voting will be based on the source,so this will not
>> issue.)
>>
>> *Proposed Change:*
>> Currently in downloads we are keeping only x86 binary(2),Can we keep ARM
>> binary also.?
>>
>> *Actions:*
>> a) *Dedicated* *Machine*:
>>i) Dedicated ARM machine will be donated which I confirmed
>>ii) Or can use jenkins ARM machine itself which is currently used
>> for ARM
>> b) *Automate Release:* How about having one release project in jenkins..?
>> So that future RM's just trigger the jenkin project.
>>
>> Please let me know your thoughts on this.
>>
>>
>> 1.
>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> 2.https://hadoop.apache.org/releases.html
>>
>>
>>
>>
>>
>>
>> --Brahma Reddy Battula
>>
>

-- 



--Brahma Reddy Battula

[DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Brahma Reddy Battula]

Hello folks,

As currently trunk will support ARM based compilation and qbt(1) is running
from several months with quite stable, hence planning to propose ARM binary
this time.

( Note : As we'll know voting will be based on the source,so this will not
issue.)

*Proposed Change:*
Currently in downloads we are keeping only x86 binary(2),Can we keep ARM
binary also.?

*Actions:*
a) *Dedicated* *Machine*:
   i) Dedicated ARM machine will be donated which I confirmed
   ii) Or can use jenkins ARM machine itself which is currently used
for ARM
b) *Automate Release:* How about having one release project in jenkins..?
So that future RM's just trigger the jenkin project.

Please let me know your thoughts on this.


1.
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
2.https://hadoop.apache.org/releases.html






--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Hi All,

Inline with the original 3.3.0 communication proposal dated 8th Jan 2020, I
would like to provide more updates[1].

We are approaching previously proposed code freeze date (March 10,2020). So
I would like to cut 3.3 branch on *10th March* and point existing *trunk to
3.4* if there are no issues.

*Current Release Plan:*

*Feature freeze Date *: all features to merge by Feb 28, 2020.( it's almost
done)
*Code freeze Date *: blockers/critical only, no improvements and
blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues
are there which are very old and I am tracking same.)
*Release Date*: March 15, 2020.

*Please let me know if I missed anything.*

1. https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0
2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER
BY priority DESC


On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula 
wrote:

>
> Wiki was updated for 3.3
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0.
>
>
> >I'll move out anything that isn't needed.
>
> thanks steve.
>
> > We need to fix the shaded protobuf in
> > Token issue to even get spark to compile.
>
> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621
>
> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran 
> wrote:
>
>> >
>> > 2. Features close to finish:
>> >
>> >
>> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
>> owner
>> > :  Steve Loughran)
>> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
>> > fixes. ( owner :  Steve Loughran)
>> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
>> > owner :  Steve Loughran)
>> >
>> > I'll move out anything that isn't needed.
>>
>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing and
>> I've got some provisional tuning to do. That is -if things didn't work in
>> the test/production deployments, I'd know about the regressions (e.g.
>> HADOOP-16751).
>>
>> This is S3A and ABFS code -no idea about the rest, and inevitably the big
>> JAR changes will have surprises. We need to fix the shaded protobuf in
>> Token issue to even get spark to compile.
>>
>> -Steve
>>
>> >
>> >
>>
>
>
> --
> --Brahma Reddy Battula
>


-- 



--Brahma Reddy Battula

Re: Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Wiki was updated for 3.3
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0.


>I'll move out anything that isn't needed.

thanks steve.

> We need to fix the shaded protobuf in
> Token issue to even get spark to compile.

Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621

On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran 
wrote:

> >
> > 2. Features close to finish:
> >
> >
> >   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. (
> owner
> > :  Steve Loughran)
> >   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
> > fixes. ( owner :  Steve Loughran)
> >   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
> > owner :  Steve Loughran)
> >
> > I'll move out anything that isn't needed.
>
> FWIW, most of these are in CDP 1.x, so there's been reasonable testing and
> I've got some provisional tuning to do. That is -if things didn't work in
> the test/production deployments, I'd know about the regressions (e.g.
> HADOOP-16751).
>
> This is S3A and ABFS code -no idea about the rest, and inevitably the big
> JAR changes will have surprises. We need to fix the shaded protobuf in
> Token issue to even get spark to compile.
>
> -Steve
>
> >
> >
>


-- 
--Brahma Reddy Battula

Hadoop 3.3 Release Plan Proposal

[Brahma Reddy Battula]

Hi All,



To continue a faster cadence of releases to accommodate more features,we could 
plan a Hadoop 3.3 release around March Mid.



To start the process sooner, and to establish a timeline, I propose to target 
Hadoop 3.3.0 release by March Mid 2020. (About 2 months from now).



I would also would like to take this opportunity to come up with a detailed 
plan.



Feature Freeze Date : All features should be merged by Feb 28, 2020.



Code Freeze Date : blockers/critical only, no improvements and non 
blocker/critical bug-fixes March 10, 2020.



Release Date: March 15, 2020



I have tried to come up with a list of features on my radar which could be 
candidates



1. Merged & Completed features:

  *   HDFS-13891 HDFS RBF stabilization phase 1. (Owner: Brahma)
  *   HDFS-12345: Scale testing HDFS NameNode with real metadata and workloads 
(Dynamometer) (owner: Erik Krogen)
  *   HDFS-13762: Support non-volatile storage class memory(SCM) in HDFS cache 
directives ( owner: Feilong He)
  *   HADOOP-16095 : Support impersonation for AuthenticationFilter (owner:  
Eric Yang)
  *   YARN-7129: Application Catalog for YARN applications (Owner: Eric Yang)
  *   YARN-5542: Scheduling of opportunistic containers (owner: Konstantinos 
Karanasos)
  *   YARN-9473: Support Vector Engine ( a new accelerator hardware) based on 
pluggable device framework. (owner :Peter Bacsko)
  *   YARN-9264: [Umbrella] Follow-up on IntelOpenCL FPGA plugin. (owner: Peter 
Bacsko)
  *   YARN-9145: [Umbrella] Dynamically add or remove auxiliary services



2. Features close to finish:



  *   HADOOP-13363 Upgrade protobuf from 2.5.0 to something newer (Owner: Vinay)
  *YARN-1011: Schedule containers based on utilization of currently 
allocated containers
  *   (Owner: Haibo Chen)
  *YARN-9698: [Umbrella] Tools to help migration from Fair Scheduler to 
Capacity Scheduler. (Owner: Weiwei Yang)
  *YARN-9050: [Umbrella] Usability improvements for scheduler activities. 
(Owner: Tao Yang)
  *YARN-8851:[Umbrella] A pluggable device plugin framework to ease vendor 
plugin development (owner: Zhankun Tang)
  *   YARN-9014: runC container runtime (owner : Eric Badger)
  *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( owner :  
Steve Loughran)
  *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes. ( 
owner :  Steve Loughran)
  *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. ( owner : 
 Steve Loughran)
  *HADOOP-15338: Support Java 11 LTS in Hadoop (owner: Akira Ajisaka)



3. Summary of Issues Status



There are 1781 issues are fixed in 3.3.0(1) which very big number.



13 Blocker and critical issues are open(2),I will followup owners to get status 
on each of them to get in by code Freeze date.



Please let me know if I missed any features targeted to 3.3 per this timeline. 
I would like to volunteer myself as release manager of 3.3.0 release.



Please let me know if you have any suggestions.

Reference:


   1) project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND 
fixVersion = 3.3.0

   2) project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, 
Critical) AND resolution = Unresolved AND "TargetVersion/s" = 3.3.0 
ORDER BY priority DESC



Note:

i) added the owners based on the jira assignee and reporter.. Please correct me
ii) will update cwiki


Regards,
Brahma Reddy Battula

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Brahma Reddy Battula]

gt; Thoughts?
> > > >> > > >
> > > >> > > > .. Owen
> > > >> > > >
> > > >> > > > On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B <
> > > >> vinayakum...@apache.org
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > >> Hi All,
> > > >> > > >>
> > > >> > > >>I wanted to discuss about the separate repo for thirdparty
> > > >> > > dependencies
> > > >> > > >> which we need to shaded and include in Hadoop component's
> jars.
> > > >> > > >>
> > > >> > > >>Apologies for the big text ahead, but this needs clear
> > > >> > explanation!!
> > > >> > > >>
> > > >> > > >>Right now most needed such dependency is protobuf.
> Protobuf
> > > >> > > dependency
> > > >> > > >> was not upgraded from 2.5.0 onwards with the fear that
> > downstream
> > > >> > > builds,
> > > >> > > >> which depends on transitive dependency protobuf coming from
> > > >> hadoop's
> > > >> > > jars,
> > > >> > > >> may fail with the upgrade. Apparently protobuf does not
> > guarantee
> > > >> > source
> > > >> > > >> compatibility, though it guarantees wire compatibility
> between
> > > >> > versions.
> > > >> > > >> Because of this behavior, version upgrade may cause breakage
> in
> > > >> known
> > > >> > > and
> > > >> > > >> unknown (private?) downstreams.
> > > >> > > >>
> > > >> > > >>So to tackle this, we came up the following proposal in
> > > >> > HADOOP-13363.
> > > >> > > >>
> > > >> > > >>Luckily, As far as I know, no APIs, either public to user
> or
> > > >> > between
> > > >> > > >> Hadoop processes, is not directly using protobuf classes in
> > > >> > signatures.
> > > >> > > >> (If
> > > >> > > >> any exist, please let us know).
> > > >> > > >>
> > > >> > > >>Proposal:
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>1. Create a artifact(s) which contains shaded
> dependencies.
> > > All
> > > >> > such
> > > >> > > >> shading/relocation will be with known prefix
> > > >> > > >> **org.apache.hadoop.thirdparty.**.
> > > >> > > >>2. Right now protobuf jar (ex:
> > > >> > > o.a.h.thirdparty:hadoop-shaded-protobuf)
> > > >> > > >> to start with, all **com.google.protobuf** classes will be
> > > >> relocated
> > > >> > as
> > > >> > > >> **org.apache.hadoop.thirdparty.com.google.protobuf**.
> > > >> > > >>3. Hadoop modules, which needs protobuf as dependency,
> will
> > > add
> > > >> > this
> > > >> > > >> shaded artifact as dependency (ex:
> > > >> > > >> o.a.h.thirdparty:hadoop-shaded-protobuf).
> > > >> > > >>4. All previous usages of "com.google.protobuf" will be
> > > >> relocated
> > > >> > to
> > > >> > > >> "org.apache.hadoop.thirdparty.com.google.protobuf" in the
> code
> > > and
> > > >> > will
> > > >> > > be
> > > >> > > >> committed. Please note, this replacement is One-Time directly
> > in
> > > >> > source
> > > >> > > >> code, NOT during compile and package.
> > > >> > > >>5. Once all usages of "com.google.protobuf" is relocated,
> > then
> > > >> > hadoop
> > > >> > > >> dont care about which version of original  "protobuf-java" is
> > in
> > > >> > > >> dependency.
> > > >> > > >>6. Just keep "protobuf-java:2.5.0" in dependency tree not
> to
> > > >> break
> > > >> > > the
> > > >> > > >> downstreams. But hadoop will be originally using the latest
> > > >> protobuf
> > > >> > > >> present in "o.a.h.thirdparty:hadoop-shaded-protobuf".
> > > >> > > >>
> > > >> > > >>7. Coming back to separate repo, Following are most
> > > appropriate
> > > >> > > reasons
> > > >> > > >> of keeping shaded dependency artifact in separate repo
> instead
> > of
> > > >> > > >> submodule.
> > > >> > > >>
> > > >> > > >>  7a. These artifacts need not be built all the time. It
> > needs
> > > >> to
> > > >> > be
> > > >> > > >> built only when there is a change in the dependency version
> or
> > > the
> > > >> > build
> > > >> > > >> process.
> > > >> > > >>  7b. If added as "submodule in Hadoop repo",
> > > >> > > maven-shade-plugin:shade
> > > >> > > >> will execute only in package phase. That means, "mvn compile"
> > or
> > > >> "mvn
> > > >> > > >> test-compile" will not be failed as this artifact will not
> have
> > > >> > > relocated
> > > >> > > >> classes, instead it will have original classes, resulting in
> > > >> > compilation
> > > >> > > >> failure. Workaround, build thirdparty submodule first and
> > exclude
> > > >> > > >> "thirdparty" submodule in other executions. This will be a
> > > complex
> > > >> > > process
> > > >> > > >> compared to keeping in a separate repo.
> > > >> > > >>
> > > >> > > >>  7c. Separate repo, will be a subproject of Hadoop, using
> > the
> > > >> > same
> > > >> > > >> HADOOP jira project, with different versioning prefixed with
> > > >> > > "thirdparty-"
> > > >> > > >> (ex: thirdparty-1.0.0).
> > > >> > > >>  7d. Separate will have same release process as Hadoop.
> > > >> > > >>
> > > >> > > >>HADOOP-13363 (
> > > >> https://issues.apache.org/jira/browse/HADOOP-13363)
> > > >> > > is
> > > >> > > >> an
> > > >> > > >> umbrella jira tracking the changes to protobuf upgrade.
> > > >> > > >>
> > > >> > > >>PR (https://github.com/apache/hadoop-thirdparty/pull/1)
> has
> > > >> been
> > > >> > > >> raised
> > > >> > > >> for separate repo creation in (HADOOP-16595 (
> > > >> > > >> https://issues.apache.org/jira/browse/HADOOP-16595)
> > > >> > > >>
> > > >> > > >>Please provide your inputs for the proposal and review the
> > PR
> > > >> to
> > > >> > > >> proceed with the proposal.
> > > >> > > >>
> > > >> > > >>
> > > >> > > >-Thanks,
> > > >> > > >>Vinay
> > > >> > > >>
> > > >> > > >> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli <
> > > >> > > >> vino...@apache.org>
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> > Moving the thread to the dev lists.
> > > >> > > >> >
> > > >> > > >> > Thanks
> > > >> > > >> > +Vinod
> > > >> > > >> >
> > > >> > > >> > > On Sep 23, 2019, at 11:43 PM, Vinayakumar B <
> > > >> > > vinayakum...@apache.org>
> > > >> > > >> > wrote:
> > > >> > > >> > >
> > > >> > > >> > > Thanks Marton,
> > > >> > > >> > >
> > > >> > > >> > > Current created 'hadoop-thirdparty' repo is empty right
> > now.
> > > >> > > >> > > Whether to use that repo  for shaded artifact or not will
> > be
> > > >> > > >> monitored in
> > > >> > > >> > > HADOOP-13363 umbrella jira. Please feel free to join the
> > > >> > discussion.
> > > >> > > >> > >
> > > >> > > >> > > There is no existing codebase is being moved out of
> hadoop
> > > >> repo.
> > > >> > So
> > > >> > > I
> > > >> > > >> > think
> > > >> > > >> > > right now we are good to go.
> > > >> > > >> > >
> > > >> > > >> > > -Vinay
> > > >> > > >> > >
> > > >> > > >> > > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek <
> > > e...@apache.org>
> > > >> > > wrote:
> > > >> > > >> > >
> > > >> > > >> > >>
> > > >> > > >> > >> I am not sure if it's defined when is a vote required.
> > > >> > > >> > >>
> > > >> > > >> > >> https://www.apache.org/foundation/voting.html
> > > >> > > >> > >>
> > > >> > > >> > >> Personally I think it's a big enough change to send a
> > > >> > notification
> > > >> > > to
> > > >> > > >> > the
> > > >> > > >> > >> dev lists with a 'lazy consensus'  closure
> > > >> > > >> > >>
> > > >> > > >> > >> Marton
> > > >> > > >> > >>
> > > >> > > >> > >> On 2019/09/23 17:46:37, Vinayakumar B <
> > > >> vinayakum...@apache.org>
> > > >> > > >> wrote:
> > > >> > > >> > >>> Hi,
> > > >> > > >> > >>>
> > > >> > > >> > >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may
> > be
> > > >> more
> > > >> > in
> > > >> > > >> > >> future)
> > > >> > > >> > >>> will be kept as a shaded artifact in a separate repo,
> > which
> > > >> will
> > > >> > > be
> > > >> > > >> > >>> referred as dependency in hadoop modules.  This
> approach
> > > >> avoids
> > > >> > > >> shading
> > > >> > > >> > >> of
> > > >> > > >> > >>> every submodule during build.
> > > >> > > >> > >>>
> > > >> > > >> > >>> So question is does any VOTE required before asking to
> > > >> create a
> > > >> > > git
> > > >> > > >> > repo?
> > > >> > > >> > >>>
> > > >> > > >> > >>> On selfserve platform
> > > >> > > https://gitbox.apache.org/setup/newrepo.html
> > > >> > > >> > >>> I can access see that, requester should be PMC.
> > > >> > > >> > >>>
> > > >> > > >> > >>> Wanted to confirm here first.
> > > >> > > >> > >>>
> > > >> > > >> > >>> -Vinay
> > > >> > > >> > >>>
> > > >> > > >> > >>
> > > >> > > >> > >>
> > > >> > >
> > > -
> > > >> > > >> > >> To unsubscribe, e-mail:
> > > private-unsubscr...@hadoop.apache.org
> > > >> > > >> > >> For additional commands, e-mail:
> > > >> private-h...@hadoop.apache.org
> > > >> > > >> > >>
> > > >> > > >> > >>
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] drop mapreduce-pipes

[Brahma Reddy Battula]

+1 on deleting this .


On Mon, 6 Jan 2020 at 7:39 PM, Steve Loughran 
wrote:

> It has been many years since anyone did any work on mapreduce-pipes and
> given the large number of hadoop-cluster streaming frameworks in the ASF
> alone, probably the same number of years since anyone actually launched a
> pipe job other than during testing.
>
> As such, it's a background noise maintenance problem and extra delays in
> test runs.
>
> What do people think about deleting it?
>
> -Steve
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] Hadoop 2019 Release Planning

[Brahma Reddy Battula]

Thanks Akira for resuming this..

 I am interested on 3.3 release ..will act as RM .will update the wiki as
well..



On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota 
wrote:

> I'm interested in doing a release of hadoop.
> The version we need an RM is 3.1.3 right? What's the target date for that?
>
> Thanks,
> Gabor
>
> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka  wrote:
>
> > Thank you Wangda.
> >
> > Now it's 2020. Let's release Hadoop 3.3.0.
> > I created a wiki page for tracking blocker/critical issues for 3.3.0 and
> > I'll check the issues in the list.
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
> > If you find blocker/critical issues in trunk, please set the target
> version
> > to 3.3.0 for tracking.
> >
> > > We still need RM for 3.3.0 and 3.1.3.
> > I can work as a release manager for 3.3.0. Is there anyone who wants to
> be
> > a RM?
> >
> > Thanks and regards,
> > Akira
> >
> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
> > wrote:
> >
> > > Thanks Wangda for bring this up!
> > >
> > > I ran the submarine 0.2.0 release before with a lot of help from folks
> > > especially Sunil. :D
> > > And this time I would like to help to release the 3.1.4. Thanks!
> > >
> > > BR,
> > > Zhankun
> > >
> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
> > >
> > > > Hi Wangda,
> > > > Thanks for bringing this up!
> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is
> a
> > > > problem.
> > > > Hope commiters watch and review these issues, Thanks
> > > > https://issues.apache.org/jira/browse/HDFS-13596
> > > > https://issues.apache.org/jira/browse/HDFS-14396
> > > >
> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > Hope this email finds you well
> > > > >
> > > > > I want to hear your thoughts about what should be the release plan
> > for
> > > > > 2019.
> > > > >
> > > > > In 2018, we released:
> > > > > - 1 maintenance release of 2.6
> > > > > - 3 maintenance releases of 2.7
> > > > > - 3 maintenance releases of 2.8
> > > > > - 3 releases of 2.9
> > > > > - 4 releases of 3.0
> > > > > - 2 releases of 3.1
> > > > >
> > > > > Total 16 releases in 2018.
> > > > >
> > > > > In 2019, by far we only have two releases:
> > > > > - 1 maintenance release of 3.1
> > > > > - 1 minor release of 3.2.
> > > > >
> > > > > However, the community put a lot of efforts to stabilize features
> of
> > > > > various release branches.
> > > > > There're:
> > > > > - 217 fixed patches in 3.1.3 [1]
> > > > > - 388 fixed patches in 3.2.1 [2]
> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> > > > >
> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and
> do a
> > > > minor
> > > > > release for 3.3.0.
> > > > >
> > > > > In addition, I saw community discussion to do a 2.8.6 release for
> > > > security
> > > > > fixes.
> > > > >
> > > > > Any other releases? I think there're release plans for Ozone as
> well.
> > > And
> > > > > please add your thoughts.
> > > > >
> > > > > Volunteers welcome! If you have interests to run a release as
> Release
> > > > > Manager (or co-Resource Manager), please respond to this email
> thread
> > > so
> > > > we
> > > > > can coordinate.
> > > > >
> > > > > Thanks,
> > > > > Wangda Tan
> > > > >
> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.1.3
> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.2.1
> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.3.0
> > > > >
> > > >
> > >
> >
>
-- 



--Brahma Reddy Battula

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Brahma Reddy Battula]

t; classes, instead it will have original classes, resulting in
> >> > compilation
> >> > > >> failure. Workaround, build thirdparty submodule first and exclude
> >> > > >> "thirdparty" submodule in other executions. This will be a
> complex
> >> > > process
> >> > > >> compared to keeping in a separate repo.
> >> > > >>
> >> > > >>  7c. Separate repo, will be a subproject of Hadoop, using the
> >> > same
> >> > > >> HADOOP jira project, with different versioning prefixed with
> >> > > "thirdparty-"
> >> > > >> (ex: thirdparty-1.0.0).
> >> > > >>  7d. Separate will have same release process as Hadoop.
> >> > > >>
> >> > > >>HADOOP-13363 (
> >> https://issues.apache.org/jira/browse/HADOOP-13363)
> >> > > is
> >> > > >> an
> >> > > >> umbrella jira tracking the changes to protobuf upgrade.
> >> > > >>
> >> > > >>PR (https://github.com/apache/hadoop-thirdparty/pull/1) has
> >> been
> >> > > >> raised
> >> > > >> for separate repo creation in (HADOOP-16595 (
> >> > > >> https://issues.apache.org/jira/browse/HADOOP-16595)
> >> > > >>
> >> > > >>Please provide your inputs for the proposal and review the PR
> >> to
> >> > > >> proceed with the proposal.
> >> > > >>
> >> > > >>
> >> > > >-Thanks,
> >> > > >>Vinay
> >> > > >>
> >> > > >> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli <
> >> > > >> vino...@apache.org>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Moving the thread to the dev lists.
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > +Vinod
> >> > > >> >
> >> > > >> > > On Sep 23, 2019, at 11:43 PM, Vinayakumar B <
> >> > > vinayakum...@apache.org>
> >> > > >> > wrote:
> >> > > >> > >
> >> > > >> > > Thanks Marton,
> >> > > >> > >
> >> > > >> > > Current created 'hadoop-thirdparty' repo is empty right now.
> >> > > >> > > Whether to use that repo  for shaded artifact or not will be
> >> > > >> monitored in
> >> > > >> > > HADOOP-13363 umbrella jira. Please feel free to join the
> >> > discussion.
> >> > > >> > >
> >> > > >> > > There is no existing codebase is being moved out of hadoop
> >> repo.
> >> > So
> >> > > I
> >> > > >> > think
> >> > > >> > > right now we are good to go.
> >> > > >> > >
> >> > > >> > > -Vinay
> >> > > >> > >
> >> > > >> > > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek <
> e...@apache.org>
> >> > > wrote:
> >> > > >> > >
> >> > > >> > >>
> >> > > >> > >> I am not sure if it's defined when is a vote required.
> >> > > >> > >>
> >> > > >> > >> https://www.apache.org/foundation/voting.html
> >> > > >> > >>
> >> > > >> > >> Personally I think it's a big enough change to send a
> >> > notification
> >> > > to
> >> > > >> > the
> >> > > >> > >> dev lists with a 'lazy consensus'  closure
> >> > > >> > >>
> >> > > >> > >> Marton
> >> > > >> > >>
> >> > > >> > >> On 2019/09/23 17:46:37, Vinayakumar B <
> >> vinayakum...@apache.org>
> >> > > >> wrote:
> >> > > >> > >>> Hi,
> >> > > >> > >>>
> >> > > >> > >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be
> >> more
> >> > in
> >> > > >> > >> future)
> >> > > >> > >>> will be kept as a shaded artifact in a separate repo, which
> >> will
> >> > > be
> >> > > >> > >>> referred as dependency in hadoop modules.  This approach
> >> avoids
> >> > > >> shading
> >> > > >> > >> of
> >> > > >> > >>> every submodule during build.
> >> > > >> > >>>
> >> > > >> > >>> So question is does any VOTE required before asking to
> >> create a
> >> > > git
> >> > > >> > repo?
> >> > > >> > >>>
> >> > > >> > >>> On selfserve platform
> >> > > https://gitbox.apache.org/setup/newrepo.html
> >> > > >> > >>> I can access see that, requester should be PMC.
> >> > > >> > >>>
> >> > > >> > >>> Wanted to confirm here first.
> >> > > >> > >>>
> >> > > >> > >>> -Vinay
> >> > > >> > >>>
> >> > > >> > >>
> >> > > >> > >>
> >> > >
> -
> >> > > >> > >> To unsubscribe, e-mail:
> private-unsubscr...@hadoop.apache.org
> >> > > >> > >> For additional commands, e-mail:
> >> private-h...@hadoop.apache.org
> >> > > >> > >>
> >> > > >> > >>
> >> > > >> >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >

-- 



--Brahma Reddy Battula

[jira] [Resolved] (HADOOP-16310) Log of a slow RPC request should contain the parameter of the request

[Brahma Reddy Battula (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula resolved HADOOP-16310.
---
Resolution: Duplicate

> Log of a slow RPC request should contain the parameter of the request
> -
>
> Key: HADOOP-16310
> URL: https://issues.apache.org/jira/browse/HADOOP-16310
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: rpc-server
>Affects Versions: 3.1.1, 2.7.7, 3.1.2
>Reporter: lindongdong
>Priority: Minor
>
>  Now, the log of  a slow RPC request just contains the 
> *methodName*,*processingTime* and *client*. Code is here:
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
>   if(LOG.isWarnEnabled()) {
> String client = CurCall.get().toString();
> LOG.warn(
> "Slow RPC : " + methodName + " took " + processingTime +
> " milliseconds to process from client " + client);
>   }
>   rpcMetrics.incrSlowRpc();
> }{code}
>  
> It is not enough to analyze why the RPC request is slow. 
> The parameter of the request is a very important thing, and need to be logged.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: [VOTE] Moving Submarine to a separate Apache project proposal

[Brahma Reddy Battula]

+1, Thanks for Wangda's proposal.

I am interested to participate in this project. Please include me


-Original Message-
From: Wanqiang Ji [mailto:wanqiang...@gmail.com] 
Sent: Wednesday, September 04, 2019 6:53 PM
To: Wangda Tan 
Cc: submarine-dev ; yarn-dev 
; Hdfs-dev ; 
mapreduce-dev ; Hadoop Common 
; private 
Subject: Re: [VOTE] Moving Submarine to a separate Apache project proposal

+1

Thanks for Wangda's proposal.

It is indeed amazing to see the growth and development of submarine. As the TLP 
will attract more developers to join.
I will put more energy into it and contribute more feature. Look forward to the 
next change in submarine.

Thanks,
Wanqiang Ji

On Wed, Sep 4, 2019 at 3:09 PM Bibin Chundatt 
wrote:

> +1
> Thank you  for the proposal.
> I am interested in project. Please include me as well in the project.
>
> Thanks,
> Bibin
>
> On Tue, Sep 3, 2019 at 6:41 PM Ayush Saxena 
> wrote:
>
> > +1
> > Thanx for the proposal.
> >
> > I would even like to participate in the project.
> > Please add me as well.
> >
> > -Ayush
> >
> >
> > > On 03-Sep-2019, at 6:00 PM, Vinayakumar B 
> > > 
> > wrote:
> > >
> > > +1
> > >
> > > Thanks for the proposal.
> > > Its very interesting project and looks very promising one by 
> > > looking at
> > the
> > > participations from various companies and the speed of development.
> > >
> > > I would also like to participate in the project.
> > > Please add me as well.
> > >
> > > Thanks,
> > > -Vinay
> > >
> > > On Tue, 3 Sep 2019, 12:38 pm Rakesh Radhakrishnan, 
> > >  >
> > > wrote:
> > >
> > >> +1, Thanks for the proposal.
> > >>
> > >> I am interested to participate in this project. Please include me 
> > >> as
> > well
> > >> in the project.
> > >>
> > >> Thanks,
> > >> Rakesh
> > >>
> > >> On Tue, Sep 3, 2019 at 11:59 AM zhankun tang 
> > >> 
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Thanks for Wangda's proposal.
> > >>>
> > >>> The submarine project is born within Hadoop, but not limited to
> Hadoop.
> > >> It
> > >>> began with a trainer on YARN but it quickly realized that only a
> > trainer
> > >> is
> > >>> not enough to meet the AI platform requirements. But now there's 
> > >>> no user-friendly open-source solution covers the whole AI 
> > >>> pipeline like
> > data
> > >>> engineering, training, and serving. And the underlying data
> > >> infrastructure
> > >>> itself is also evolving, for instance, many people love k8s. Not
> > >> mentioning
> > >>> there're many AI domain problems in this area to be solved.
> > >>> It's almost for sure that building such an ML platform would 
> > >>> utilize various other open-source components taking ML into 
> > >>> consideration initially.
> > >>>
> > >>> I see submarine grows rapidly towards an enterprise-grade ML 
> > >>> platform
> > >> which
> > >>> could potentially enable AI ability for data engineer and scientist.
> > This
> > >>> is an exciting thing for both the community and the industry.
> > >>>
> > >>> BR,
> > >>> Zhankun
> > >>>
> > >>>
> >  On Tue, 3 Sep 2019 at 13:34, Xun Liu  wrote:
> > 
> >  +1
> > 
> >  Hello everyone, I am a member of the submarine development team.
> >  I have been contributing to submarine for more than a year.
> >  I have seen the progress of submarine development very fast.
> >  In more than a year, there are 9 long-term developers of 
> >  different companies. Contributing, submarine cumulative code 
> >  has more than 200,000 lines of code, is
> > >> growing
> >  very fast,
> >  and is used in the production environment of multiple companies.
> > 
> >  In the submarine development group, there are 5 PMCs and 
> >  7committer
> > >>> members
> >  from Hadoop, spark, zeppelin projects.
> >  They are very familiar with the development process and
> specifications
> > >> of
> >  the apache community,
> >  and can well grasp the project development progress and project
> > >> quality.
> >  So I recommend submarine to be a TLP project directly.
> > 
> >  We will continue to contribute to the submarine project. :-)
> > 
> >  Xun Liu
> >  Regards
> > 
> > > On Tue, 3 Sep 2019 at 12:01, Devaraj K  wrote:
> > >
> > > +1
> > >
> > > Thanks Wangda for the proposal.
> > > I would like to participate in this project, Please add me 
> > > also to
> > >> the
> > > project.
> > >
> > > Regards
> > > Devaraj K
> > >
> > > On Mon, Sep 2, 2019 at 8:50 PM zac yuan 
> > > 
> > >> wrote:
> > >
> > >> +1
> > >>
> > >> Submarine will be a complete solution for AI service development.
> > >> It
> >  can
> > >> take advantage of two best cluster systems: yarn and k8s, 
> > >> which
> > >> will
> >  help
> > >> more and more people get AI ability. To be a separate Apache
> > >> project,
> > > will
> > >> accelerate the procedure of development apparently.
> > >>
> > >> 

Re: [Notice] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Merged. thanks to Arpit for quoting the merge process.

From: Brahma Reddy Battula 
Sent: Tuesday, June 25, 2019 12:39 AM
To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; 
Hdfs-dev; Iñigo Goiri; Hadoop Common
Subject: [Notice] Merge HDFS-13891(RBF) to trunk

Rebased HDFS-13891 against trunk and ran the jenkins (1).Only two four 
testcases are failing which flaky and there is already jira to track this. 
(HDFS-14461)


Going to merge the branch soon which contains 77 commits,so please bear with us.



1.https://builds.apache.org/job/PreCommit-HDFS-Build/27044/console



From: Brahma Reddy Battula 
Sent: Sunday, June 9, 2019 9:56 PM
To: Xiaoqiao He
Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; 
Hdfs-dev; Iñigo Goiri
Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Hi All,

Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

Summary of code changes:
1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
2. Added Security support for RBF
3. Added Missing Client Protocol API's
4. Bug Fixes/ Improvments


 The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He  wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota 
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> giovanni.fumar...@gmail.com> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri  wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka 
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > bra...@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > &g

[Notice] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Rebased HDFS-13891 against trunk and ran the jenkins (1).Only two four 
testcases are failing which flaky and there is already jira to track this. 
(HDFS-14461)


Going to merge the branch soon which contains 77 commits,so please bear with us.



1.https://builds.apache.org/job/PreCommit-HDFS-Build/27044/console



From: Brahma Reddy Battula 
Sent: Sunday, June 9, 2019 9:56 PM
To: Xiaoqiao He
Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; 
Hdfs-dev; Iñigo Goiri
Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk

Hi All,

Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

Summary of code changes:
1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
2. Added Security support for RBF
3. Added Missing Client Protocol API's
4. Bug Fixes/ Improvments


 The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He  wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota 
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> giovanni.fumar...@gmail.com> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri  wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka 
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > bra...@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > > Adding Missing Client API's(2)
>> > > > > Improvements/Bug Fixing
>> > > > >   Critical - HDFS-13637, HDFS-13834
>> > > > >
>> > > > > *Commits:*
>> > > > > 
>> > > > >
>> > > > > No of JIRAs Resolved: 72
>> > > > >
>> > > > > All this 

[RESULT][VOTE] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Vote is passed with 3 binding +1's  and no -1 ( 5 nonbinding votes) .

thanks to all for voting.

Will try to merge this week..


On Tue, Jun 11, 2019 at 1:13 PM Takanobu Asanuma 
wrote:

> +1(non-binding).
>
> Regards,
> - Takanobu
>
> 
> From: Xiaoqiao He 
> Sent: Monday, June 10, 2019 15:47
> To: Ranith Sardar
> Cc: Brahma Reddy Battula; Hadoop Common; Hdfs-dev
> Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
>
> +1 (non-binding)
>
> - Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or
> failure.
> - Built from merged sources.
> - Ran 85 RBF test class at local and result shows: (Tests run: 639,
> Failures: 2, Errors: 2, Skipped: 2) - failed tests include
> #TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't
> think it is blocked issue.
>
> Thanks Brahma for organizing the merge.
>
>
> On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar 
> wrote:
>
> > +1 (Non-binding)
> >
> > -Original Message-
> > From: Brahma Reddy Battula [mailto:bra...@apache.org]
> > Sent: 09 June 2019 19:31
> > To: Hadoop Common ; Hdfs-dev <
> > hdfs-...@hadoop.apache.org>
> > Subject: [VOTE] Merge HDFS-13891(RBF) to trunk
> >
> > Updated mail...
> >
> > -- Forwarded message -
> > From: Brahma Reddy Battula 
> > Date: Sun, Jun 9, 2019 at 7:26 PM
> > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
> > To: Xiaoqiao He 
> > Cc: Akira Ajisaka , Chittaranjan Hota <
> > chitts.h...@gmail.com>, Giovanni Matteo Fumarola <
> > giovanni.fumar...@gmail.com>, Hadoop Common <
> common-dev@hadoop.apache.org>,
> > Hdfs-dev , Iñigo Goiri 
> >
> >
> > Hi All,
> >
> > Given the positive response to the discussion thread [1], here is the
> > formal vote thread to merge HDFS-13891 in to trunk.
> >
> > Summary of code changes:
> > 1. Code changes for this branch are done in the hadoop-hdfs-rbf
> > subproject, there is no impact to hadoop-hdfs and hadoop-common.
> > 2. Added Security support for RBF
> > 3. Added Missing Client Protocol API's
> > 4. Bug Fixes/ Improvments
> >
> >
> >  The vote will run for 7 days, ending Sat June 15th. I will start
> this
> > vote with my +1.
> >
> > Regards,
> > Brahma Reddy Battula
> >
> > 1).
> >
> >
> https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He  wrote:
> >
> > > Thanks Brahma for starting the thread.
> > > +1 for merging.
> > >
> > > He Xiaoqiao
> > >
> > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota
> > > 
> > > wrote:
> > >
> > >> Thanks Brahma initiating this.
> > >> +1(non-binding) for merge.
> > >>
> > >> @Uber we have almost all changes specially rbf security in production
> > >> for a while now without issues.
> > >>
> > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
> > >> giovanni.fumar...@gmail.com> wrote:
> > >>
> > >> > +1 on merging.
> > >> >
> > >> > Thanks Brahma for starting the thread.
> > >> >
> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri 
> > wrote:
> > >> >
> > >> > > Thank you Brahma for pushing this.
> > >> > >
> > >> > > As you mentioned, we have already taken most of the changes into
> > >> > > production.
> > >> > > I want to highlight that the main contribution is the addition of
> > >> > security.
> > >> > > We have been able to test this at a smaller scale (~500 servers
> > >> > > and 4
> > >> > > subclusters) and the performance is great with our current
> > >> > > ZooKeeper deployment.
> > >> > > I would also like to highlight that all the changes are
> > >> > > constrained to hadoop-hdfs-rbf and there is no differences in
> > commons or HDFS.
> > >> > >
> > >> > > +1 on merging
> > >> > >
> > >> > > Inigo
> > >> > >
> > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka
> > >> > > 
> > >> > wrote:
> > >>

Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Dear Arpit,

Thanks for taking look into it.

ECBlockGroupStats.merge() is Utility method which is moved to
hadoop-hdfs-client module. Ideally it could have been seperate jira. But
this changes will not induce any issues, will take necessary action for
this.

On Mon, Jun 10, 2019 at 8:40 PM Arpit Agarwal  wrote:

> I scanned the merge payload for changes to non-RBF code. The changes are
> minimal, which is good.
>
> The only commit that I didn’t understand was:
> https://issues.apache.org/jira/browse/HDFS-14268
>
> The jira description doesn’t make it clear why ECBlockGroupStats is
> modified.
>
> +0 apart from that.
>
>
> On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula 
> wrote:
>
> Dear Hadoop Developers
>
> I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> been working on this feature from last several months.
> This feature work received the contributions from different companies. All
> of the feature development happened smoothly and collaboratively in JIRAs.
>
> Kindly do take a look at the branch and raise issues/concerns that need to
> be addressed before the merge.
>
> *Highlights of HDFS-13891 Branch:*
> =
>
> Adding Security to RBF(1)
> Adding Missing Client API's(2)
> Improvements/Bug Fixing
>  Critical - HDFS-13637, HDFS-13834
>
> *Commits:*
> 
>
> No of JIRAs Resolved: 72
>
> All this commits are in RBF Module. No changes in hdfs/common.
>
> *Tested Cluster:*
> =
>
> Most of these changes verified at Uber,Microsoft,Huawei and some other
> compaines.
>
> *Uber*: Most changes are running in production @Uber including the critical
> security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> Zookeeper as a state store to hold delegation tokens were also stress
> tested to hold more than 2 Million tokens. --CR Hota
>
> *MicroSoft*: Most of these changes are currently running in production at
> Microsoft.The security has also been tested in a 500 server cluster with 4
> subclsuters. --Inigo Goiri
>
> *Huawei* : Deployed all this changes in 20 node cluster with 3
> routers.Planning deploy 10K production cluster.
>
> *Contributors:*
> ===
>
> Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
> contributing to this.
>
> *Future Tasks:*
> ====
>
> will cleanup the jira's under this umbrella and contiue to work.
>
> Reference:
> 1) https://issues.apache.org/jira/browse/HDFS-13532
> 2) https://issues.apache.org/jira/browse/HDFS-13655
>
>
>
>
> --Brahma Reddy Battula
>
>
>

-- 



--Brahma Reddy Battula

[VOTE] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Updated mail...

-- Forwarded message -
From: Brahma Reddy Battula 
Date: Sun, Jun 9, 2019 at 7:26 PM
Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk
To: Xiaoqiao He 
Cc: Akira Ajisaka , Chittaranjan Hota <
chitts.h...@gmail.com>, Giovanni Matteo Fumarola <
giovanni.fumar...@gmail.com>, Hadoop Common ,
Hdfs-dev , Iñigo Goiri 


Hi All,

Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

Summary of code changes:
1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
2. Added Security support for RBF
3. Added Missing Client Protocol API's
4. Bug Fixes/ Improvments


 The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He  wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota 
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> giovanni.fumar...@gmail.com> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri  wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka 
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > bra...@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > > Adding Missing Client API's(2)
>> > > > > Improvements/Bug Fixing
>> > > > >   Critical - HDFS-13637, HDFS-13834
>> > > > >
>> > > > > *Commits:*
>> > > > > 
>> > > > >
>> > > > > No of JIRAs Resolved: 72
>> > > > >
>> > > > > All this commits are in RBF Module. No changes in hdfs/common.
>> > > > >
>> > > > > *Tested Cluster:*
>> > > > > =
>> > > > >
>> > > >

Re: [VOTE] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Hi All,

Given the positive response to the discussion thread [1], here is the
formal vote thread to merge HDFS-13891 in to trunk.

Summary of code changes:
1. Code changes for this branch are done in the hadoop-hdfs-rbf
subproject, there is no impact to hadoop-hdfs and hadoop-common.
2. Added Security support for RBF
3. Added Missing Client Protocol API's
4. Bug Fixes/ Improvments


 The vote will run for 7 days, ending Sat June 15th. I will start this
vote with my +1.

Regards,
Brahma Reddy Battula

1).
https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He  wrote:

> Thanks Brahma for starting the thread.
> +1 for merging.
>
> He Xiaoqiao
>
> On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota 
> wrote:
>
>> Thanks Brahma initiating this.
>> +1(non-binding) for merge.
>>
>> @Uber we have almost all changes specially rbf security in production for
>> a
>> while now without issues.
>>
>> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola <
>> giovanni.fumar...@gmail.com> wrote:
>>
>> > +1 on merging.
>> >
>> > Thanks Brahma for starting the thread.
>> >
>> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri  wrote:
>> >
>> > > Thank you Brahma for pushing this.
>> > >
>> > > As you mentioned, we have already taken most of the changes into
>> > > production.
>> > > I want to highlight that the main contribution is the addition of
>> > security.
>> > > We have been able to test this at a smaller scale (~500 servers and 4
>> > > subclusters) and the performance is great with our current ZooKeeper
>> > > deployment.
>> > > I would also like to highlight that all the changes are constrained to
>> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS.
>> > >
>> > > +1 on merging
>> > >
>> > > Inigo
>> > >
>> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka 
>> > wrote:
>> > >
>> > > > Thanks Brahma for starting the discussion.
>> > > > I'm +1 for merging this.
>> > > >
>> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
>> > > > with 2 routers (not in production) and running several tests.
>> > > >
>> > > > Regards,
>> > > > Akira
>> > > >
>> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <
>> > bra...@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > Dear Hadoop Developers
>> > > > >
>> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk.
>> We
>> > > have
>> > > > > been working on this feature from last several months.
>> > > > > This feature work received the contributions from different
>> > companies.
>> > > > All
>> > > > > of the feature development happened smoothly and collaboratively
>> in
>> > > > JIRAs.
>> > > > >
>> > > > > Kindly do take a look at the branch and raise issues/concerns that
>> > need
>> > > > to
>> > > > > be addressed before the merge.
>> > > > >
>> > > > > *Highlights of HDFS-13891 Branch:*
>> > > > > =
>> > > > >
>> > > > > Adding Security to RBF(1)
>> > > > > Adding Missing Client API's(2)
>> > > > > Improvements/Bug Fixing
>> > > > >   Critical - HDFS-13637, HDFS-13834
>> > > > >
>> > > > > *Commits:*
>> > > > > 
>> > > > >
>> > > > > No of JIRAs Resolved: 72
>> > > > >
>> > > > > All this commits are in RBF Module. No changes in hdfs/common.
>> > > > >
>> > > > > *Tested Cluster:*
>> > > > > =
>> > > > >
>> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some
>> > other
>> > > > > compaines.
>> > > > >
>> > > > > *Uber*: Most changes are running in production @Uber including the
>> > > > critical
>> > > > > security changes, HDFS Clusters are 4000+ n

[DISCUSS] Merge HDFS-13891(RBF) to trunk

[Brahma Reddy Battula]

Dear Hadoop Developers

I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
been working on this feature from last several months.
This feature work received the contributions from different companies. All
of the feature development happened smoothly and collaboratively in JIRAs.

Kindly do take a look at the branch and raise issues/concerns that need to
be addressed before the merge.

*Highlights of HDFS-13891 Branch:*
=

Adding Security to RBF(1)
Adding Missing Client API's(2)
Improvements/Bug Fixing
  Critical - HDFS-13637, HDFS-13834

*Commits:*


No of JIRAs Resolved: 72

All this commits are in RBF Module. No changes in hdfs/common.

*Tested Cluster:*
=

Most of these changes verified at Uber,Microsoft,Huawei and some other
compaines.

*Uber*: Most changes are running in production @Uber including the critical
security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
Zookeeper as a state store to hold delegation tokens were also stress
tested to hold more than 2 Million tokens. --CR Hota

*MicroSoft*: Most of these changes are currently running in production at
Microsoft.The security has also been tested in a 500 server cluster with 4
subclsuters. --Inigo Goiri

*Huawei* : Deployed all this changes in 20 node cluster with 3
routers.Planning deploy 10K production cluster.

*Contributors:*
===

Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and
contributing to this.

*Future Tasks:*


 will cleanup the jira's under this umbrella and contiue to work.

Reference:
1) https://issues.apache.org/jira/browse/HDFS-13532
2) https://issues.apache.org/jira/browse/HDFS-13655




--Brahma Reddy Battula

[RESULT][VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)

[Brahma Reddy Battula]

Here is my +1 too.

Thanks to everyone that participated. The vote to “unprotect the HDFS-13891
branch”
is now closed. It PASSED
with 4(+1 binding) votes and no 0 or -1 votes:

Binding Votes:
===

 Íñigo Goiri
Akira Ajisaka
Anu Engineer
Brahma Reddy Battula

Non Binding votes :
=

Giovanni Matteo Fumarola
Takanobu Asanuma
Ajay Kumar
Dinesh Chitlangia
Aaron Fabbri.






On Thu, May 16, 2019 at 2:02 AM, Aaron Fabbri  wrote:

> +1 to unprotect feature branch (in general) for rebasing against trunk.
>
> On Tue, May 14, 2019 at 7:53 PM Dinesh Chitlangia
>  wrote:
>
> > +1(non-binding) for branch.
> >
> > -Dinesh
> >
> >
> >
> >
> > On Tue, May 14, 2019 at 10:04 PM Brahma Reddy Battula  >
> > wrote:
> >
> > > Yes Arpit,it’s not for trunk..
> > >
> > >
> > > On Wed, May 15, 2019 at 2:11 AM, Arpit Agarwal 
> > > wrote:
> > >
> > > > The request is specific to HDFS-13891, correct?
> > > >
> > > > We should not allow force push on trunk.
> > > >
> > > >
> > > > > On May 14, 2019, at 8:07 AM, Anu Engineer  > > .INVALID>
> > > > wrote:
> > > > >
> > > > > Is it possible to unprotect the branches and not the trunk?
> > Generally,
> > > a
> > > > > force push to trunk indicates a mistake and we have had that in the
> > > past.
> > > > > This is just a suggestion,  even if this request is not met, I am
> > still
> > > > +1.
> > > > >
> > > > > Thanks
> > > > > Anu
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 14, 2019 at 4:58 AM Takanobu Asanuma <
> > > tasan...@yahoo-corp.jp
> > > > >
> > > > > wrote:
> > > > >
> > > > >> +1.
> > > > >>
> > > > >> Thanks!
> > > > >> - Takanobu
> > > > >>
> > > > >> 
> > > > >> From: Akira Ajisaka 
> > > > >> Sent: Tuesday, May 14, 2019 4:26:30 PM
> > > > >> To: Giovanni Matteo Fumarola
> > > > >> Cc: Iñigo Goiri; Brahma Reddy Battula; Hadoop Common; Hdfs-dev
> > > > >> Subject: Re: [VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)
> > > > >>
> > > > >> +1 to unprotect the branch.
> > > > >>
> > > > >> Thanks,
> > > > >> Akira
> > > > >>
> > > > >> On Tue, May 14, 2019 at 3:11 PM Giovanni Matteo Fumarola
> > > > >>  wrote:
> > > > >>>
> > > > >>> +1 to unprotect the branches for rebases.
> > > > >>>
> > > > >>> On Mon, May 13, 2019 at 11:01 PM Iñigo Goiri 
> > > > wrote:
> > > > >>>
> > > > >>>> Syncing the branch to trunk should be a fairly standard task.
> > > > >>>> Is there a way to do this without rebasing and forcing the push?
> > > > >>>> As far as I know this has been the standard for other branches
> > and I
> > > > >> don't
> > > > >>>> know of any alternative.
> > > > >>>> We should clarify the process as having to get PMC consensus to
> > > rebase
> > > > >> a
> > > > >>>> branch seems a little overkill to me.
> > > > >>>>
> > > > >>>> +1 from my side to un protect the branch to do the rebase.
> > > > >>>>
> > > > >>>> On Mon, May 13, 2019, 22:46 Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hi Folks,
> > > > >>>>>
> > > > >>>>> INFRA-18181 made all the Hadoop branches are protected.
> > > > >>>>> Badly HDFS-13891 branch needs to rebased as we contribute core
> > > > >> patches
> > > > >>>>> trunk..So,currently we are stuck with rebase as it’s not
> allowed
> > to
> > > > >> force
> > > > >>>>> push.Hence raised INFRA-18361.
> > > > >>>>>
> > > > >>>>> Can we have a quick vote for INFRA sign-off to proceed as this
> is
> > > > >>>> blocking
> > > > >>>>> all branch commits??
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --Brahma Reddy Battula
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > -
> > > > >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > > >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > > > >>
> > > > >>
> > > >
> > > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>
-- 



--Brahma Reddy Battula

Re: [VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)

[Brahma Reddy Battula]

Yes Arpit,it’s not for trunk..


On Wed, May 15, 2019 at 2:11 AM, Arpit Agarwal 
wrote:

> The request is specific to HDFS-13891, correct?
>
> We should not allow force push on trunk.
>
>
> > On May 14, 2019, at 8:07 AM, Anu Engineer 
> wrote:
> >
> > Is it possible to unprotect the branches and not the trunk? Generally, a
> > force push to trunk indicates a mistake and we have had that in the past.
> > This is just a suggestion,  even if this request is not met, I am still
> +1.
> >
> > Thanks
> > Anu
> >
> >
> >
> > On Tue, May 14, 2019 at 4:58 AM Takanobu Asanuma  >
> > wrote:
> >
> >> +1.
> >>
> >> Thanks!
> >> - Takanobu
> >>
> >> ____________
> >> From: Akira Ajisaka 
> >> Sent: Tuesday, May 14, 2019 4:26:30 PM
> >> To: Giovanni Matteo Fumarola
> >> Cc: Iñigo Goiri; Brahma Reddy Battula; Hadoop Common; Hdfs-dev
> >> Subject: Re: [VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)
> >>
> >> +1 to unprotect the branch.
> >>
> >> Thanks,
> >> Akira
> >>
> >> On Tue, May 14, 2019 at 3:11 PM Giovanni Matteo Fumarola
> >>  wrote:
> >>>
> >>> +1 to unprotect the branches for rebases.
> >>>
> >>> On Mon, May 13, 2019 at 11:01 PM Iñigo Goiri 
> wrote:
> >>>
> >>>> Syncing the branch to trunk should be a fairly standard task.
> >>>> Is there a way to do this without rebasing and forcing the push?
> >>>> As far as I know this has been the standard for other branches and I
> >> don't
> >>>> know of any alternative.
> >>>> We should clarify the process as having to get PMC consensus to rebase
> >> a
> >>>> branch seems a little overkill to me.
> >>>>
> >>>> +1 from my side to un protect the branch to do the rebase.
> >>>>
> >>>> On Mon, May 13, 2019, 22:46 Brahma Reddy Battula 
> >>>> wrote:
> >>>>
> >>>>> Hi Folks,
> >>>>>
> >>>>> INFRA-18181 made all the Hadoop branches are protected.
> >>>>> Badly HDFS-13891 branch needs to rebased as we contribute core
> >> patches
> >>>>> trunk..So,currently we are stuck with rebase as it’s not allowed to
> >> force
> >>>>> push.Hence raised INFRA-18361.
> >>>>>
> >>>>> Can we have a quick vote for INFRA sign-off to proceed as this is
> >>>> blocking
> >>>>> all branch commits??
> >>>>>
> >>>>> --
> >>>>>
> >>>>>
> >>>>>
> >>>>> --Brahma Reddy Battula
> >>>>>
> >>>>
> >>
> >> -
> >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>
> >>
>
> --



--Brahma Reddy Battula

[VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)

[Brahma Reddy Battula]

Hi Folks,

INFRA-18181 made all the Hadoop branches are protected.
Badly HDFS-13891 branch needs to rebased as we contribute core patches
trunk..So,currently we are stuck with rebase as it’s not allowed to force
push.Hence raised INFRA-18361.

Can we have a quick vote for INFRA sign-off to proceed as this is blocking
all branch commits??

-- 



--Brahma Reddy Battula

Re: [ANNOUNCE] Eric Badger is now a committer!

[Brahma Reddy Battula]

Congratulations Eric!!!


On Tue, Mar 5, 2019 at 10:50 PM, Eric Payne 
wrote:

> It is my pleasure to announce that Eric Badger has accepted an invitation
> to become a Hadoop Core committer.
>
> Congratulations, Eric! This is well-deserved!
>
> -Eric Payne
>
-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Brahma Reddy Battula]

Sunil thanks for driving this.

+1 ,(binding)

--Built from Source.
--Installed pseudo cluster.
--Ran basic shell commands
--Ran sample jobs pi,wordcount.



On Tue, Jan 8, 2019 at 5:12 PM Sunil G  wrote:

> Hi folks,
>
>
> Thanks to all of you who helped in this release [1] and for helping to vote
> for RC0. I have created second release candidate (RC1) for Apache Hadoop
> 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>
>
> RC tag in git is release-3.2.0-RC1.
>
>
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>
>
> This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>


-- 
--Brahma Reddy Battula

RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Brahma Reddy Battula]

My late +1. Really it's useful feature.. Great work.

-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common ; hdfs-dev 

Cc: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent 
Reads from Standby

Obviously +1 from me.

With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.

Thanks,
--Konst

On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch 
> HDFS-12943 for Consistent Reads from Standby Node. The feature is 
> intended to scale read RPC workloads. On large clusters reads comprise 
> 95% of all RPCs to the NameNode. We should be able to accommodate 
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple 
> ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent 
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including 
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873), 
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>

Re: [DISCUSS] Move to gitbox

[Brahma Reddy Battula]

+1



On Sat, Dec 8, 2018 at 1:26 PM, Akira Ajisaka  wrote:

> Hi all,
>
> Apache Hadoop git repository is in git-wip-us server and it will be
> decommissioned.
> If there are no objection, I'll file a JIRA ticket with INFRA to
> migrate to https://gitbox.apache.org/ and update documentation.
>
> According to ASF infra team, the timeframe is as follows:
>
> > - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> relocation
> > - January 9th -> February 6th: Mandated (coordinated) relocation
> > - February 7th: All remaining repositories are mass migrated.
> > This timeline may change to accommodate various scenarios.
>
> If we got consensus by January 9th, I can file a ticket with INFRA and
> migrate it.
> Even if we cannot got consensus, the repository will be migrated by
> February 7th.
>
> Regards,
> Akira
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
> --



--Brahma Reddy Battula

RE: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

[Brahma Reddy Battula]

Akira, Thanks for driving this release.

+1 (binding)

-- Built from source
-- Installed HA cluster
--Verified basic operations
--Ran Sample Jobs
--Browsed the UI


-Brahma Reddy Battula

-Original Message-
From: Akira Ajisaka [mailto:aajis...@apache.org] 
Sent: Wednesday, November 14, 2018 6:33 AM
To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

Hi folks,

I have put together a release candidate (RC0) for Hadoop 2.9.2. It includes 204 
bug fixes and improvements since 2.9.1. [1]

The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/
Git signed tag is release-2.9.2-RC0 and the checksum is
826afbeae31ca687bc2f8471dc841b66ed2c6704
The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1166/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Please try the release and vote. The vote will run for 5 days.

[1] https://s.apache.org/2.9.2-fixed-jiras

Thanks,
Akira

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Brahma Reddy Battula]

+1

It’s better to new version link in old version.


Brahma Reddy Battula

On Fri, Aug 31, 2018 at 9:59 PM, Sangjin Lee  wrote:

> +1. Thanks for the work, Marton!
>
> On Fri, Aug 31, 2018 at 8:37 AM Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> > Is there no way to host the new site and the old site concurrently? And
> > link back & forth?
> >
> > +Vinod
> >
> >
> > > On Aug 31, 2018, at 1:07 AM, Elek, Marton  wrote:
> > >
> > > Bumping this thread at last time.
> > >
> > > I have the following proposal:
> > >
> > > 1. I will request a new git repository hadoop-site.git and import the
> > new site to there (which has exactly the same content as the existing
> site).
> > >
> > > 2. I will ask infra to use the new repository as the source of
> > hadoop.apache.org
> > >
> > > 3. I will sync manually all of the changes in the next two months back
> > to the svn site from the git (release announcements, new committers)
> > >
> > > IN CASE OF ANY PROBLEM we can switch back to the svn without any
> problem.
> > >
> > > If no-one objects within three days, I'll assume lazy consensus and
> > start with this plan. Please comment if you have objections.
> > >
> > > Again: it allows immediate fallback at any time as svn repo will be
> kept
> > as is (+ I will keep it up-to-date in the next 2 months)
> > >
> > > Thanks,
> > > Marton
> > >
> > >
> > > On 06/21/2018 09:00 PM, Elek, Marton wrote:
> > >> Thank you very much to bump up this thread.
> > >> About [2]: (Just for the clarification) the content of the proposed
> > website is exactly the same as the old one.
> > >> About [1]. I believe that the "mvn site" is perfect for the
> > documentation but for website creation there are more simple and powerful
> > tools.
> > >> Hugo has more simple compared to jekyll. Just one binary, without
> > dependencies, works everywhere (mac, linux, windows)
> > >> Hugo has much more powerful compared to "mvn site". Easier to
> > create/use more modern layout/theme, and easier to handle the content
> (for
> > example new release announcements could be generated as part of the
> release
> > process)
> > >> I think it's very low risk to try out a new approach for the site (and
> > easy to rollback in case of problems)
> > >> Marton
> > >> ps: I just updated the patch/preview site with the recent releases:
> > >> ***
> > >> * http://hadoop.anzix.net *
> > >> ***
> > >> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> > >>> Got pinged about this offline.
> > >>>
> > >>> Thanks for keeping at it, Marton!
> > >>>
> > >>> I think there are two road-blocks here
> > >>>   (1) Is the mechanism using which the website is built good enough -
> > mvn-site / hugo etc?
> > >>>   (2) Is the new website good enough?
> > >>>
> > >>> For (1), I just think we need more committer attention and get
> > feedback rapidly and get it in.
> > >>>
> > >>> For (2), how about we do it in a different way in the interest of
> > progress?
> > >>>   - We create a hadoop.apache.org/new-site/ where this new site
> goes.
> > >>>   - We then modify the existing web-site to say that there is a new
> > site/experience that folks can click on a link and navigate to
> > >>>   - As this new website matures and gets feedback & fixes, we finally
> > pull the plug at a later point of time when we think we are good to go.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> +Vinod
> > >>>
> > >>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I would like to bump this thread up.
> > >>>>
> > >>>> TLDR; There is a proposed version of a new hadoop site which is
> > available from here: https://elek.github.io/hadoop-site-proposal/ and
> > https://issues.apache.org/jira/browse/HADOOP-14163
> > >>>>
> > >>>> Please let me know what you think about it.
> > >>>>
> > >>>>
> > >>>> Longer version:
> > >>>&

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Brahma Reddy Battula]

+1 ,(binding).

Wangda Tan thanks for driving this.


--Built with source.
--Installed the HA cluster
--Ran basic hdfs operations through shell/API
--Ran Sample jobs like pi, wordcount
--Browsed UI.

On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan  wrote:

> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>

RE: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

[Brahma Reddy Battula]

Thanks Junping for driving this release.


+1  (binding)


-- Build successfully from the source code
-- Start HA cluster
-- Verified basic shell operations
-- Ran pi,wordcount
-- Browsed the NN and RM UI

  


-Brahma Reddy Battula

-Original Message-
From: 俊平堵 [mailto:junping...@apache.org] 
Sent: 09 May 2018 01:41
To: Hadoop Common <common-dev@hadoop.apache.org>; Hdfs-dev 
<hdfs-...@hadoop.apache.org>; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

Hi all,
 I've created the first release candidate (RC0) for Apache Hadoop 2.8.4. 
This is our next maint release to follow up 2.8.3. It includes 77 important 
fixes and improvements.

The RC artifacts are available at:
http://home.apache.org/~junping_du/hadoop-2.8.4-RC0

The RC tag in git is: release-2.8.4-RC0

The maven artifacts are available via repository.apache.org< 
http://repository.apache.org> at:
https://repository.apache.org/content/repositories/orgapachehadoop-1118

Please try the release and vote; the vote will run for the usual 5 working 
days, ending on 5/14/2018 PST time.

Thanks,

Junping

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: Apache Hadoop 3.1.1 release plan

[Brahma Reddy Battula]

Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better addition 
to 3.1 line release for improving quality.

Looks only following two are pending which are in review state. Hope you are 
monitoring these two.

https://issues.apache.org/jira/browse/YARN-8265
https://issues.apache.org/jira/browse/YARN-8236



Note : https://issues.apache.org/jira/browse/YARN-8247==> committed branch-3.1


-Original Message-
From: Wangda Tan [mailto:wheele...@gmail.com] 
Sent: 19 April 2018 17:49
To: Hadoop Common ; 
mapreduce-...@hadoop.apache.org; Hdfs-dev ; 
yarn-...@hadoop.apache.org
Subject: Apache Hadoop 3.1.1 release plan

Hi, All

We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the quality 
of the release, we plan to release 3.1.1 at May 06. The focus of 3.1.1 will be 
fixing blockers / critical bugs and other enhancements. So far there are 100 
JIRAs [1] have fix version marked to 3.1.1.

We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce") AND 
fixVersion = 3.1.1

RE: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Brahma Reddy Battula]

Konstantin thanks for driving this.

+1 (binding)


--Built from the source
--Installed HA cluster
-Verified the basic shell commands
-Ran sample jobs like pi,wordcount
-Browsed the UI's



-Original Message-
From: Konstantin Shvachko [mailto:shv.had...@gmail.com] 
Sent: 10 April 2018 07:14
To: Hadoop Common ; hdfs-dev 
; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

Hi everybody,

This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.5 
was released on December 14, 2017.
Release 2.7.6 includes critical bug fixes and optimizations. See more details 
in Release Note:
http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days 
ending 04/16/2018.

My up to date public key is available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
--Konstantin

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Brahma Reddy Battula]

Wangda thanks for driving this.

+1(binding)

--Built from source
--Installed HA cluster
--Verified Basic Shell commands
--Ran Sample Jobs
--Browsed the UI's.


On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

RE: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Brahma Reddy Battula]

Thanks Lei Xu for driving this.

+1(binding)


--Built from the source
--Installed the HA Cluster
--Verified hdfs operations
--Ran sample jobs
--Verified the UI's



--Brahma Reddy Battula


-Original Message-
From: Lei Xu [mailto:l...@apache.org] 
Sent: 18 March 2018 12:11
To: Hadoop Common <common-dev@hadoop.apache.org>; Hdfs-dev 
<hdfs-...@hadoop.apache.org>; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

Hi, all

I've created release candidate RC-1 for Apache Hadoop 3.0.1

Apache Hadoop 3.0.1 will be the first bug fix release for Apache Hadoop 3.0 
release. It includes 49 bug fixes and security fixes, which include 12 blockers 
and 17 are critical.

Please note:
* HDFS-12990. Change default NameNode RPC port back to 8020. It makes 
incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache Hadoop 
3.0.0 will be deprecated due to this change.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/

The git tag is release-3.0.1-RC1, and the latest commit is
496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1081/

Please try the release and vote; the vote will run for the usual 5 days, ending 
on 3/22/2017 6pm PST time.

Thanks!

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org