[jira] [Created] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config

[Wangda Tan (Jira)]

Wangda Tan created MAPREDUCE-7309:
-

 Summary: Improve performance of reading resource request for 
mapper/reducers from config
 Key: MAPREDUCE-7309
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: applicationmaster
Affects Versions: 3.3.0, 3.2.0, 3.1.0, 3.0.0
Reporter: Wangda Tan
 Attachments: MAPREDUCE-7309.001.patch

This is an issue could affect all the releases which includes YARN-6927. 

Basically, we use regex match repeatly when we read mapper/reducer resource 
request from config files. When we have large config file, and large number of 
splits, it could take a long time.  

We saw AM could take hours to parse config when we have 200k+ splits, with a 
large config file (hundreds of kbs). 

We should do proper caching for pre-configured resource requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Moving Ozone to a separated Apache project

[Wangda Tan]

+1,

Thanks,
Wangda Tan

On Tue, Sep 29, 2020 at 10:10 AM Aravindan Vijayan
 wrote:

> +1, thank you Marton.
>
> On Tue, Sep 29, 2020 at 9:17 AM Bharat Viswanadham 
> wrote:
>
> > +1
> > Thank You @Elek, Marton  for driving this.
> >
> >
> > Thanks,
> > Bharat
> >
> >
> > On Mon, Sep 28, 2020 at 10:54 AM Vivek Ratnavel <
> vivekratna...@apache.org>
> > wrote:
> >
> > > +1 for moving Ozone to a separated Top-Level Apache Project.
> > >
> > > Thanks,
> > > Vivek Subramanian
> > >
> > > On Mon, Sep 28, 2020 at 8:30 AM Hanisha Koneru
> > > 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Thanks,
> > > > Hanisha
> > > >
> > > > > On Sep 27, 2020, at 11:48 PM, Akira Ajisaka 
> > > wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > Thanks,
> > > > > Akira
> > > > >
> > > > > On Fri, Sep 25, 2020 at 3:00 PM Elek, Marton  >  > > > e...@apache.org>> wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> Thank you for all the feedback and requests,
> > > > >>
> > > > >> As we discussed in the previous thread(s) [1], Ozone is proposed
> to
> > > be a
> > > > >> separated Apache Top Level Project (TLP)
> > > > >>
> > > > >> The proposal with all the details, motivation and history is here:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Hadoop+subproject+to+Apache+TLP+proposal
> > > > >>
> > > > >> This voting runs for 7 days and will be concluded at 2nd of
> October,
> > > 6AM
> > > > >> GMT.
> > > > >>
> > > > >> Thanks,
> > > > >> Marton Elek
> > > > >>
> > > > >> [1]:
> > > > >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
> > > > >>
> > > > >>
> > -
> > > > >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > >  > > > yarn-dev-unsubscr...@hadoop.apache.org>
> > > > >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > > > <mailto:yarn-dev-h...@hadoop.apache.org>
> > > > >>
> > > > >
> > > > >
> -
> > > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > > <mailto:common-dev-unsubscr...@hadoop.apache.org>
> > > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > > > <mailto:common-dev-h...@hadoop.apache.org>
> > > >
> > >
> >
>
>
> --
> Thanks & Regards,
> Aravindan
>

Re: [ANNOUNCE] New Apache Hadoop Committer - He Xiaoqiao

[Wangda Tan]

Congrats!

Best,
Wangda Tan

On Thu, Jun 11, 2020 at 5:55 PM Tao Yang  wrote:

> Congratulations Xiaoqiao!
>
> Best,
> Tao
>
>
> --
> 发件人：Weiwei Yang 
> 发送时间：2020年6月12日(星期五) 02:24
> 收件人：Sree Vaddi 
> 抄 送：Ayush Saxena ; Chao Sun ;
> Wei-Chiu Chuang ; Hadoop Common <
> common-...@hadoop.apache.org>; Hdfs-dev ;
> yarn-dev ; mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>; ozone-dev 
> 主 题：Re: [ANNOUNCE] New Apache Hadoop Committer - He Xiaoqiao
>
> Congratulations Xiaoqiao!
>
> On Thu, Jun 11, 2020 at 11:20 AM Sree Vaddi  .invalid>
> wrote:
>
> > Congratulations, He Xiaoqiao.
> >
> > Thank you./Sree
> >
> >
> >
> > On Thursday, June 11, 2020, 9:54:32 AM PDT, Chao Sun <
> > sunc...@apache.org> wrote:
> >
> >  Congrats Xiaoqiao!
> >
> > On Thu, Jun 11, 2020 at 9:36 AM Ayush Saxena  wrote:
> >
> > > Congratulations He Xiaoqiao!!!
> > >
> > > -Ayush
> > >
> > > > On 11-Jun-2020, at 9:30 PM, Wei-Chiu Chuang 
> > wrote:
> > > >
> > > >  In bcc: general@
> > > >
> > > > It's my pleasure to announce that He Xiaoqiao has been elected as a
> > > > committer on the Apache Hadoop project recognizing his continued
> > > > contributions to the
> > > > project.
> > > >
> > > > Please join me in congratulating him.
> > > >
> > > > Hearty Congratulations & Welcome aboard Xiaoqiao!
> > > >
> > > > Wei-Chiu Chuang
> > > > (On behalf of the Hadoop PMC)
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
>
>

Re: [DISCUSS] making Ozone a separate Apache project

[Wangda Tan]

+1,

Thanks,
Wangda

On Thu, May 14, 2020 at 7:01 AM Tsuyoshi Ozawa  wrote:

> +1
>
> Thanks
> - Tsuyoshi
>
> On Thu, May 14, 2020 at 19:42 Rakesh Radhakrishnan <
> rakeshr.oz...@gmail.com>
> wrote:
>
> > +1
> >
> > Thanks,
> > Rakesh
> >
> > On Wed, May 13, 2020 at 1:23 PM Elek, Marton  wrote:
> >
> > >
> > >
> > > I would like to start a discussion to make a separate Apache project
> for
> > > Ozone
> > >
> > >
> > >
> > > ### HISTORY [1]
> > >
> > >   * Apache Hadoop Ozone development started on a feature branch of
> > > Hadoop repository (HDFS-7240)
> > >
> > >   * In the October of 2017 a discussion has been started to merge it to
> > > the Hadoop main branch
> > >
> > >   * After a long discussion it's merged to Hadoop trunk at the March of
> > > 2018
> > >
> > >   * During the discussion of the merge, it was suggested multiple times
> > > to create a separated project for the Ozone. But at that time:
> > >  1). Ozone was tightly integrated with Hadoop/HDFS
> > >  2). There was an active plan to use Block layer of Ozone (HDDS or
> > > HDSL at that time) as the block level of HDFS
> > >  3). The community of Ozone was a subset of the HDFS community
> > >
> > >   * The first beta release of Ozone was just released. Seems to be a
> > > good time before the first GA to make a decision about the future.
> > >
> > >
> > >
> > > ### WHAT HAS BEEN CHANGED
> > >
> > >   During the last years Ozone became more and more independent both at
> > > the community and code side. The separation has been suggested again
> and
> > > again (for example by Owen [2] and Vinod [3])
> > >
> > >
> > >
> > >   From COMMUNITY point of view:
> > >
> > >
> > >* Fortunately more and more new contributors are helping Ozone.
> > > Originally the Ozone community was a subset of HDFS project. But now a
> > > bigger and bigger part of the community is related to Ozone only.
> > >
> > >* It seems to be easier to _build_ the community as a separated
> > project.
> > >
> > >* A new, younger project might have different practices
> > > (communication, commiter criteria, development style) compared to old,
> > > mature project
> > >
> > >* It's easier to communicate (and improve) these standards in a
> > > separated projects with clean boundaries
> > >
> > >* Separated project/brand can help to increase the adoption rate and
> > > attract more individual contributor (AFAIK it has been seen in
> Submarine
> > > after a similar move)
> > >
> > >   * Contribution process can be communicated more easily, we can make
> > > first time contribution more easy
> > >
> > >
> > >
> > >   From CODE point of view Ozone became more and more independent:
> > >
> > >
> > >   * Ozone has different release cycle
> > >
> > >   * Code is already separated from Hadoop code base
> > > (apache/hadoop-ozone.git)
> > >
> > >   * It has separated CI (github actions)
> > >
> > >   * Ozone uses different (more strict) coding style (zero toleration of
> > > unit test / checkstyle errors)
> > >
> > >   * The code itself became more and more independent from Hadoop on
> > > Maven level. Originally it was compiled together with the in-tree
> latest
> > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > > Configuration...)
> > >
> > >   * It starts to use multiple version of Hadoop (on client side)
> > >
> > >   * Volume of resolved issues are already very high on Ozone side
> (Ozone
> > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > > together in the last 2-3 months)
> > >
> > >
> > > Summary: Before the first Ozone GA release, It seems to be a good time
> > > to discuss the long-term future of Ozone. Managing it as a separated
> TLP
> > > project seems to have more benefits.
> > >
> > >
> > > Please let me know what your opinion is...
> > >
> > > Thanks a lot,
> > > Marton
> > >
> > >
> > >
> > >
> > >
> > > [1]: For more details, see:
> > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> > >
> > > [2]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> > >
> > > [3]:
> > >
> > >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> > >
> > > -
> > > To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [VOTE] EOL Hadoop branch-2.8

[Wangda Tan]

+1

On Mon, Mar 2, 2020 at 8:50 PM Akira Ajisaka  wrote:

> +1
>
> -Akira
>
> On Tue, Mar 3, 2020 at 4:55 AM Ayush Saxena  wrote:
>
> > +1 for marking 2.8 EOL
> >
> > -Ayush
> >
> > > On 03-Mar-2020, at 12:18 AM, Wei-Chiu Chuang 
> wrote:
> > >
> > > I am sorry I forgot to start a VOTE thread.
> > >
> > > This is the "official" vote thread to mark branch-2.8 End of Life. This
> > is
> > > based on the following thread and the tracking jira (HADOOP-16880
> > > ).
> > >
> > > This vote will run for 7 days and conclude on March 9th (Mon) 11am PST.
> > >
> > > Please feel free to share your thoughts.
> > >
> > > Thanks,
> > > Weichiu
> > >
> > >> On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang <
> weic...@cloudera.com>
> > >> wrote:
> > >>
> > >> Looking at the EOL policy wiki:
> > >>
> >
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> > >>
> > >> The Hadoop community can still elect to make security update for
> EOL'ed
> > >> releases.
> > >>
> > >> I think the EOL is to give more clarity to downstream applications
> (such
> > >> as HBase) the guidance of which Hadoop release lines are still active.
> > >> Additionally, I don't think it is sustainable to maintain 6 concurrent
> > >> release lines in this big project, which is why I wanted to start this
> > >> discussion.
> > >>
> > >> Thoughts?
> > >>
> > >>> On Mon, Feb 24, 2020 at 10:22 AM Sunil Govindan 
> > wrote:
> > >>>
> > >>> Hi Wei-Chiu
> > >>>
> > >>> Extremely sorry for the late reply here.
> > >>> Cud u pls help to add more clarity on defining what will happen for
> > >>> branch-2.8 when we call EOL.
> > >>> Does this mean that, no more release coming out from this branch, or
> > some
> > >>> more additional guidelines?
> > >>>
> > >>> - Sunil
> > >>>
> > >>>
> > >>> On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang
> > >>>  wrote:
> > >>>
> >  This thread has been running for 7 days and no -1.
> > 
> >  Don't think we've established a formal EOL process, but to publicize
> > the
> >  EOL, I am going to file a jira, update the wiki and post the
> > >>> announcement
> >  to general@ and user@
> > 
> >  On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia <
> > >>> dineshc@gmail.com>
> >  wrote:
> > 
> > > Thanks Wei-Chiu for initiating this.
> > >
> > > +1 for 2.8 EOL.
> > >
> > > On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka <
> aajis...@apache.org>
> > > wrote:
> > >
> > >> Thanks Wei-Chiu for starting the discussion,
> > >>
> > >> +1 for the EoL.
> > >>
> > >> -Akira
> > >>
> > >> On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena 
> >  wrote:
> > >>
> > >>> Thanx Wei-Chiu for initiating this
> > >>> +1 for marking 2.8 EOL
> > >>>
> > >>> -Ayush
> > >>>
> >  On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang <
> > >>> weic...@apache.org>
> > >> wrote:
> > 
> >  The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th
> >  2018.
> > 
> >  It's been 17 months since the release and the community by and
> >  large
> > >> have
> >  moved up to 2.9/2.10/3.x.
> > 
> >  With Hadoop 3.3.0 over the horizon, is it time to start the EOL
> > >>> discussion
> >  and reduce the number of active branches?
> > >>>
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >>> For additional commands, e-mail:
> > >>> common-dev-h...@hadoop.apache.org
> > >>>
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Hi guys,

Previously we are using
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap for 3.1 and 3.2.
So to reduce confusing, I suggest to use one of the
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap or
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
for the release plans.

Thanks,
Wangda

On Wed, Jan 8, 2020 at 5:56 AM Ayush Saxena  wrote:

> Hi Gabor
>
> Seams like you need to create one, I too can't find anything related.
>
> Most I could find was :
>
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.1.3
>
> -Ayush
>
>
> On Wed, 8 Jan 2020 at 18:45, Gabor Bota 
> wrote:
>
> > I'm volunteering to be the release manager for 3.1.4.
> > Where can I find a wiki page for the 3.1 line? I can not find one here:
> >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
> >
> > On Tue, Jan 7, 2020 at 3:41 PM Steve Loughran
>  > >
> > wrote:
> >
> > > I'm thinking of doing a backport of most of the hadoop-aws changes to
> > > branch-3.2, for the next 3.2.x release; they are all self contained and
> > > will benefit many (they will need to cope with the older mockito
> version,
> > > but I have to deal with that in-house already).
> > >
> > > one change is the new openFile() builder API. I'd like to wrap that up
> > with
> > > a little improvement
> https://issues.apache.org/jira/browse/HADOOP-16759;
> > > That way for all releases with the API, it's consistent.
> > >
> > > (that withStatus() feature gives extra performance and ensures that
> > > etag/version can be used to get the explicit version you want.)
> > >
> > > On Tue, Jan 7, 2020 at 2:18 AM Akira Ajisaka 
> > wrote:
> > >
> > > > >  I am interested on 3.3 release ..will act as RM .will update the
> > wiki
> > > as
> > > > well..
> > > >
> > > > Thanks Brahma for your reply. I'll help you as co-RM.
> > > > We will send announcements (cutting branches, code freeze, and so on)
> > in
> > > > another thread.
> > > >
> > > > Thanks,
> > > > Akira
> > > >
> > > > On Tue, Jan 7, 2020 at 4:32 AM Wangda Tan 
> wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > Thanks for the update and for volunteering to be RM.
> > > > >
> > > > > I just did a quick check:
> > > > > 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21)
> > > > > 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22)
> > > > > 3.3.0 has .. many patches sitting here so we definitely need a
> > release.
> > > > >
> > > > > If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be
> > > great.
> > > > >
> > > > > Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I
> saw
> > > > > community started to have large prod deployment on 3.2.x, Gabor if
> > you
> > > > have
> > > > > bandwidth to help releases, I think we can do 3.2.2 first then
> 3.1.4.
> > > > >
> > > > > Thoughts?
> > > > > - Wangda
> > > > >
> > > > > On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > >> Thanks Akira for resuming this..
> > > > >>
> > > > >>  I am interested on 3.3 release ..will act as RM .will update the
> > wiki
> > > > as
> > > > >> well..
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota <
> gabor.b...@cloudera.com
> > > > .invalid>
> > > > >> wrote:
> > > > >>
> > > > >>> I'm interested in doing a release of hadoop.
> > > > >>> The version we need an RM is 3.1.3 right? What's the target date
> > for
> > > > >>> that?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Gabor
> > > > >>>
> > > > >>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka <
> aajis...@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Thank you Wangda.
> > > &

Re: [DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Hi guys,

Thanks for the update and for volunteering to be RM.

I just did a quick check:
3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21)
3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22)
3.3.0 has .. many patches sitting here so we definitely need a release.

If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be great.

Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw
community started to have large prod deployment on 3.2.x, Gabor if you have
bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4.

Thoughts?
- Wangda

On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula 
wrote:

> Thanks Akira for resuming this..
>
>  I am interested on 3.3 release ..will act as RM .will update the wiki as
> well..
>
>
>
> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota 
> wrote:
>
>> I'm interested in doing a release of hadoop.
>> The version we need an RM is 3.1.3 right? What's the target date for that?
>>
>> Thanks,
>> Gabor
>>
>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka  wrote:
>>
>> > Thank you Wangda.
>> >
>> > Now it's 2020. Let's release Hadoop 3.3.0.
>> > I created a wiki page for tracking blocker/critical issues for 3.3.0 and
>> > I'll check the issues in the list.
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
>> > If you find blocker/critical issues in trunk, please set the target
>> version
>> > to 3.3.0 for tracking.
>> >
>> > > We still need RM for 3.3.0 and 3.1.3.
>> > I can work as a release manager for 3.3.0. Is there anyone who wants to
>> be
>> > a RM?
>> >
>> > Thanks and regards,
>> > Akira
>> >
>> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
>> > wrote:
>> >
>> > > Thanks Wangda for bring this up!
>> > >
>> > > I ran the submarine 0.2.0 release before with a lot of help from folks
>> > > especially Sunil. :D
>> > > And this time I would like to help to release the 3.1.4. Thanks!
>> > >
>> > > BR,
>> > > Zhankun
>> > >
>> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
>> > >
>> > > > Hi Wangda,
>> > > > Thanks for bringing this up!
>> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade
>> is a
>> > > > problem.
>> > > > Hope commiters watch and review these issues, Thanks
>> > > > https://issues.apache.org/jira/browse/HDFS-13596
>> > > > https://issues.apache.org/jira/browse/HDFS-14396
>> > > >
>> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Hope this email finds you well
>> > > > >
>> > > > > I want to hear your thoughts about what should be the release plan
>> > for
>> > > > > 2019.
>> > > > >
>> > > > > In 2018, we released:
>> > > > > - 1 maintenance release of 2.6
>> > > > > - 3 maintenance releases of 2.7
>> > > > > - 3 maintenance releases of 2.8
>> > > > > - 3 releases of 2.9
>> > > > > - 4 releases of 3.0
>> > > > > - 2 releases of 3.1
>> > > > >
>> > > > > Total 16 releases in 2018.
>> > > > >
>> > > > > In 2019, by far we only have two releases:
>> > > > > - 1 maintenance release of 3.1
>> > > > > - 1 minor release of 3.2.
>> > > > >
>> > > > > However, the community put a lot of efforts to stabilize features
>> of
>> > > > > various release branches.
>> > > > > There're:
>> > > > > - 217 fixed patches in 3.1.3 [1]
>> > > > > - 388 fixed patches in 3.2.1 [2]
>> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
>> > > > >
>> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and
>> do a
>> > > > minor
>> > > > > release for 3.3.0.
>> > > > >
>> > > > > In addition, I saw community discussion to do a 2.8.6 release for
>> > > > security
>> > > > > fixes.
>> > > > >
>> > > > > Any other releases? I think there're release plans for Ozone as
>> well.
>> > > And
>> > > > > please add your thoughts.
>> > > > >
>> > > > > Volunteers welcome! If you have interests to run a release as
>> Release
>> > > > > Manager (or co-Resource Manager), please respond to this email
>> thread
>> > > so
>> > > > we
>> > > > > can coordinate.
>> > > > >
>> > > > > Thanks,
>> > > > > Wangda Tan
>> > > > >
>> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>> Fixed
>> > > AND
>> > > > > fixVersion = 3.1.3
>> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>> Fixed
>> > > AND
>> > > > > fixVersion = 3.2.1
>> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>> Fixed
>> > > AND
>> > > > > fixVersion = 3.3.0
>> > > > >
>> > > >
>> > >
>> >
>>
> --
>
>
>
> --Brahma Reddy Battula
>

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Wangda Tan]

+1, thanks Jonathan for bringing this up!

On Fri, Nov 15, 2019 at 11:41 AM epa...@apache.org 
wrote:

> Thanks Jonathan for opening the discussion.
>
> I am not in favor of this proposal. 2.10 was very recently released, and
> moving to 2.10 will take some time for the community. It seems premature to
> make a decision at this point that there will never be a need for a 2.11
> release.
>
> -Eric
>
>
>  On Thursday, November 14, 2019, 8:51:59 PM CST, Jonathan Hung <
> jyhung2...@gmail.com> wrote:
>
> Hi folks,
>
> Given the release of 2.10.0, and the fact that it's intended to be a bridge
> release to Hadoop 3.x [1], I'm proposing we make 2.10.x the last minor
> release line in branch-2. Currently, the main issue is that there's many
> fixes going into branch-2 (the theoretical 2.11.0) that's not going into
> branch-2.10 (which will become 2.10.1), so the fixes in branch-2 will
> likely never see the light of day unless they are backported to
> branch-2.10.
>
> To do this, I propose we:
>
>   - Delete branch-2.10
>   - Rename branch-2 to branch-2.10
>   - Set version in the new branch-2.10 to 2.10.1-SNAPSHOT
>
> This way we get all the current branch-2 fixes into the 2.10.x release
> line. Then the commit chain will look like: trunk -> branch-3.2 ->
> branch-3.1 -> branch-2.10 -> branch-2.9 -> branch-2.8
>
> Thoughts?
>
> Jonathan Hung
>
> [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg29479.html
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [Discuss] Hadoop-Ozone repository mailing list configurations

[Wangda Tan]

We're going to fix the Submarine email list issues once spin-off works
start

On Wed, Oct 23, 2019 at 2:39 PM Matt Foley 
wrote:

> Definitely yes on ‘ozone-issues’.  Whether we want to keep ozone-dev and
> hdfs-dev together or separate, I’m neutral.
> Thanks,
> —Matt
>
> On Oct 23, 2019, at 2:11 PM, Elek, Marton  wrote:
>
> Thanks to report this problem Rohith,
>
> Yes, it seems to be configured with the wrong mailing list.
>
> I think the right fix is to create ozone-dev@ and ozone-issues@ and use
> them instead of hdfs-(dev/issues).
>
> Is there any objections against creating new ozone-* mailing lists?
>
> Thanks,
> Marton
>
>
> On 10/21/19 6:03 AM, Rohith Sharma K S wrote:
> > + common/yarn and mapreduce/submarine
> > Looks like same issue in submarine repository also !
> > On Mon, 21 Oct 2019 at 09:30, Rohith Sharma K S <
> rohithsharm...@apache.org>
> > wrote:
> >> Folks,
> >>
> >> In Hadoop world, any mailing list has its own purposes as below
> >> 1. hdfs/common/yarn/mapreduce-*dev *mailing list is meant for developer
> >> discussion purpose.
> >> 2. hdfs/common/yarn/mapreduce*-issues* mailing list used for comments
> >> made in the issues.
> >>
> >>  It appears Hadoop-Ozone repository configured *hdfs-dev *mailing list
> >> for *hdfs-issues* list also. As a result hdfs-dev mailing list is
> >> bombarded with every comment made in hadoop-ozone repository.
> >>
> >>
> >> Could it be fixed?
> >>
> >> -Rohith Sharma K S
> >>
> >>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>

[NEED HELP] Hadoop 3.x Production Deployment Can be Publicly Talk About?

[Wangda Tan]

Hi devs and users,

Tomorrow (sorry for the short notice) we will do a presentation at Strata
Data Conf @ NY for a Community update of Hadoop 3.x. I'm thinking to create
a slide about existing production deployment on Hadoop 3.x. Basically, I
want to put a logo wall with a list of big names so we can encourage more
users to upgrade to 3.x.

I knew there are tons of users are on 3.x already, but only a few of them
have public slides, I don't get a permission to put other non-public use
cases to the slide. So if you are:
- Using 3.x in production. (Ideally large scale, using some new features or
running in new environment like on cloud, using GPU, etc.).
- When I saying 3.x it could be Apache Hadoop 3.x, HDP 3.x, CDH 6.x or any
distribution with Apache Hadoop 3.x as base.

Please respond your company name w/ logo (it it is not too obvious) to this
email (either public or private to me). If you could include a short
summary (which can be publicly shared) of the use cases that will be very
helpful.

If the number of companies responded is too low, I may skip putting a logo
wall.

Hope to get your feedbacks soon.

Thanks,
Wangda Tan

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Wangda Tan]

+1 (binding).

>From my experiences of Submarine project, I think moving to a separate repo
helps.

- Wangda

On Tue, Sep 17, 2019 at 11:41 AM Subru Krishnan  wrote:

> +1 (binding).
>
> IIUC, there will not be an Ozone module in trunk anymore as that was my
> only concern from the original discussion thread? IMHO, this should be the
> default approach for new modules.
>
> On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
> slamendo...@bloomberg.net> wrote:
>
> > +1
> >
> > From: e...@apache.org At: 09/17/19 05:48:32To:
> hdfs-...@hadoop.apache.org,
> > mapreduce-dev@hadoop.apache.org,  common-...@hadoop.apache.org,
> > yarn-...@hadoop.apache.org
> > Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
> > source tree
> >
> >
> > TLDR; I propose to move Ozone related code out from Hadoop trunk and
> > store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
> >
> >
> > When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
> > be part of the source tree but with separated release cadence, mainly
> > because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
> >
> > During the last Ozone releases this dependency is removed to provide
> > more stable releases. Instead of using the latest trunk/SNAPSHOT build
> > from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
> >
> > As we have no more strict dependency between Hadoop trunk SNAPSHOT and
> > Ozone trunk I propose to separate the two code base from each other with
> > creating a new Hadoop git repository (apache/hadoop-ozone.git):
> >
> > With moving Ozone to a separated git repository:
> >
> >   * It would be easier to contribute and understand the build (as of now
> > we always need `-f pom.ozone.xml` as a Maven parameter)
> >   * It would be possible to adjust build process without breaking
> > Hadoop/Ozone builds.
> >   * It would be possible to use different Readme/.asf.yaml/github
> > template for the Hadoop Ozone and core Hadoop. (For example the current
> > github template [2] has a link to the contribution guideline [3]. Ozone
> > has an extended version [4] from this guideline with additional
> > information.)
> >   * Testing would be more safe as it won't be possible to change core
> > Hadoop and Hadoop Ozone in the same patch.
> >   * It would be easier to cut branches for Hadoop releases (based on the
> > original consensus, Ozone should be removed from all the release
> > branches after creating relase branches from trunk)
> >
> >
> > What do you think?
> >
> > Thanks,
> > Marton
> >
> > [1]:
> >
> >
> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
> > c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
> > [2]:
> >
> >
> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
> > [3]:
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
> > [4]:
> >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Thanks everyone for voting, support, and suggestions!

Best,
Wangda


On Sun, Sep 8, 2019 at 10:57 PM Sree Vaddi  wrote:

> Congratulations, Wangda. Thank you.
>
> Let's work towards it's better adoption.
>
>
> Thank you.
> /Sree
>
>
>
>
> On Sunday, September 8, 2019, 10:30:49 PM PDT, Wangda Tan <
> wheele...@gmail.com> wrote:
>
>
> Thanks everybody for voting! Let me conclude the voting thread:
>
> We got 16 binding +1s from PMCs. And 29 +1s from Committers/Contributors.
> With no veto.
>
> The vote now passed, we will work with the Apache community to get new
> project created and continue the spin-off process.
>
> Best,
> Wangda Tan
>
>
> On Sun, Sep 8, 2019 at 11:24 AM Chen Liang  wrote:
>
> > Late +1.  I am interested in the project too. Please include me.
> >
> > Regards,
> > Chen
> >
> > Varun Saxena  于2019年9月7日周六 上午10:35写道：
> >
> >> Belated +1.
> >> I too am interested in the project. Please include me.
> >>
> >> Regards,
> >> Varun Saxena
> >>
> >> On Sat, Sep 7, 2019 at 9:35 PM Sunil Govindan 
> wrote:
> >>
> >> > +1 to the proposal.
> >> > Thanks to the community for the great response.
> >> >
> >> > - Sunil
> >> >
> >> > On Sun, Sep 1, 2019 at 10:49 AM Wangda Tan 
> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > As we discussed in the previous thread [1],
> >> > >
> >> > > I just moved the spin-off proposal to CWIKI and completed all TODO
> >> parts.
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
> >> > >
> >> > > If you have interests to learn more about this. Please review the
> >> > proposal
> >> > > let me know if you have any questions/suggestions for the proposal.
> >> This
> >> > > will be sent to board post voting passed. (And please note that the
> >> > > previous voting thread [2] to move Submarine to a separate Github
> repo
> >> > is a
> >> > > necessary effort to move Submarine to a separate Apache project but
> >> not
> >> > > sufficient so I sent two separate voting thread.)
> >> > >
> >> > > Please let me know if I missed anyone in the proposal, and reply if
> >> you'd
> >> > > like to be included in the project.
> >> > >
> >> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
> >> PDT.
> >> > >
> >> > > Thanks,
> >> > > Wangda Tan
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> >> > > [2]
> >> > >
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
> >> > >
> >> >
> >>
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Thanks everybody for voting! Let me conclude the voting thread:

We got 16 binding +1s from PMCs. And 29 +1s from Committers/Contributors.
With no veto.

The vote now passed, we will work with the Apache community to get new
project created and continue the spin-off process.

Best,
Wangda Tan


On Sun, Sep 8, 2019 at 11:24 AM Chen Liang  wrote:

> Late +1.   I am interested in the project too. Please include me.
>
> Regards,
> Chen
>
> Varun Saxena  于2019年9月7日周六 上午10:35写道：
>
>> Belated +1.
>> I too am interested in the project. Please include me.
>>
>> Regards,
>> Varun Saxena
>>
>> On Sat, Sep 7, 2019 at 9:35 PM Sunil Govindan  wrote:
>>
>> > +1 to the proposal.
>> > Thanks to the community for the great response.
>> >
>> > - Sunil
>> >
>> > On Sun, Sep 1, 2019 at 10:49 AM Wangda Tan  wrote:
>> >
>> > > Hi all,
>> > >
>> > > As we discussed in the previous thread [1],
>> > >
>> > > I just moved the spin-off proposal to CWIKI and completed all TODO
>> parts.
>> > >
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>> > >
>> > > If you have interests to learn more about this. Please review the
>> > proposal
>> > > let me know if you have any questions/suggestions for the proposal.
>> This
>> > > will be sent to board post voting passed. (And please note that the
>> > > previous voting thread [2] to move Submarine to a separate Github repo
>> > is a
>> > > necessary effort to move Submarine to a separate Apache project but
>> not
>> > > sufficient so I sent two separate voting thread.)
>> > >
>> > > Please let me know if I missed anyone in the proposal, and reply if
>> you'd
>> > > like to be included in the project.
>> > >
>> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
>> PDT.
>> > >
>> > > Thanks,
>> > > Wangda Tan
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> > > [2]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>> > >
>> >
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

I think I missed Rohith:

* Rohith Sharma K S (rohithsharm...@apache.org) (Hadoop PMC)

To add to initial committers.

On Fri, Sep 6, 2019 at 10:56 PM Dinesh Chitlangia 
wrote:

> +1
>
> -Dinesh
>
>
>
>
> On Fri, Sep 6, 2019 at 11:23 PM 俊平堵  wrote:
>
>> +1. Please include me also.
>>
>> Thanks,
>>
>> Junping
>>
>> Wangda Tan  于2019年9月1日周日 下午1:19写道：
>>
>> > Hi all,
>> >
>> > As we discussed in the previous thread [1],
>> >
>> > I just moved the spin-off proposal to CWIKI and completed all TODO
>> parts.
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>> >
>> > If you have interests to learn more about this. Please review the
>> proposal
>> > let me know if you have any questions/suggestions for the proposal. This
>> > will be sent to board post voting passed. (And please note that the
>> > previous voting thread [2] to move Submarine to a separate Github repo
>> is a
>> > necessary effort to move Submarine to a separate Apache project but not
>> > sufficient so I sent two separate voting thread.)
>> >
>> > Please let me know if I missed anyone in the proposal, and reply if
>> you'd
>> > like to be included in the project.
>> >
>> > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>> > [1]
>> >
>> >
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> > [2]
>> >
>> >
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Thanks everyone for voting! And for whoever has interests to join
Submarine, you're always welcome!

And thanks to Owen for the kind help offered, I just added you to PMC list
in the proposal. It will be a great help to the community if you could
join!

For existing Hadoop committers who have interests to join, I plan to add
you to the initial list after discussed with other proposed initial
Submarine PMC members. The list I saw is:

* Naganarasimha Garla (naganarasimha_gr at apache dot org) (Hadoop PMC)
* Devaraj K (devaraj at apache dot org) (Hadoop PMC)
* Rakesh Radhakrishnan (rakeshr at apache dot org) (bookkeeper PMC, Hadoop
PMC, incubator, Mnemonic PMC, Zookeeper PMC)
* Vinayakumar B (vinayakumarb at apache dot org) (Hadoop PMC, incubator PMC)
* Ayush Saxena (ayushsaxena at apache dot org) (Hadoop Committer)
* Bibin Chundatt (bibinchundatt at apache dot org) (Hadoop PMC)
* Bharat Viswanadham (bharat at apache dot org) (Hadoop)
* Brahma Reddy Battula (brahma at apache dot org)) (Hadoop PMC)
* Abhishek Modi (abmodi at apache dot org) (Hadoop Committer)
* Wei-Chiu Chuang (weichiu at apache dot org) (Hadoop PMC)
* Junping Du (junping_du at apache dot org) (Hadoop PMC, member)

We'd like to see some reasonable contributions to the projects from all our
committers who will join now. Please join the weekly call or mailing lists
(once established) and share your inputs to the project. Members of
Submarine will reach out to all of you individually to understand the areas
you wish to contribute and will help in same. please let me know if you
DON'T want to add to the committer list.

Best,
Wangda Tan

On Fri, Sep 6, 2019 at 3:54 PM Wei-Chiu Chuang  wrote:

> +1
> I've involved myself in Submarine dev and I'd like to be included in the
> future.
>
> Thanks
>
> On Sat, Sep 7, 2019 at 5:27 AM Owen O'Malley 
> wrote:
>
>> Since you don't have any Apache Members, I'll join to provide Apache
>> oversight.
>>
>> .. Owen
>>
>> On Fri, Sep 6, 2019 at 1:38 PM Owen O'Malley 
>> wrote:
>>
>> > +1 for moving to a new project.
>> >
>> > On Sat, Aug 31, 2019 at 10:19 PM Wangda Tan 
>> wrote:
>> >
>> >> Hi all,
>> >>
>> >> As we discussed in the previous thread [1],
>> >>
>> >> I just moved the spin-off proposal to CWIKI and completed all TODO
>> parts.
>> >>
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>> >>
>> >> If you have interests to learn more about this. Please review the
>> proposal
>> >> let me know if you have any questions/suggestions for the proposal.
>> This
>> >> will be sent to board post voting passed. (And please note that the
>> >> previous voting thread [2] to move Submarine to a separate Github repo
>> is
>> >> a
>> >> necessary effort to move Submarine to a separate Apache project but not
>> >> sufficient so I sent two separate voting thread.)
>> >>
>> >> Please let me know if I missed anyone in the proposal, and reply if
>> you'd
>> >> like to be included in the project.
>> >>
>> >> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
>> PDT.
>> >>
>> >> Thanks,
>> >> Wangda Tan
>> >>
>> >> [1]
>> >>
>> >>
>> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2]
>> >>
>> >>
>> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Hi Sree,

I put it to the proposal, please let me know what you think:

The traditional path at Apache would have been to create an incubator
> project, but the code is already being released by Apache and most of the
> developers are familiar with Apache rules and guidelines. In particular,
> the proposed PMC has 2 Apache TLP PMCs and proposed initial committers
> have 4 Apache TLP PMCs (from 3 different companies). They will provide
> oversight and guidance for the developers that are less experienced in the
> Apache Way. Therefore, the Submarine project would like to propose becoming
> a Top Level Project at Apache.
>

To me, putting to TLP has mostly pros, it is an easier process (same as ORC
spin-off from Hive), much less overhead to both dev community and Apache
side.

Thanks,
Wangda

On Sun, Sep 1, 2019 at 2:04 PM Sree Vaddi  wrote:

> +1 to move submarine to separate apache project.
>
> It is not clear in the proposal, if submarine majority voted to move to a
> seperate apache project,
> will it go through the incubation and TLP (top level project) later ?
> 1. Incubation
> pros and cons
> efforts towards making it a TLP
>
> 2. direct TLP
>
>
> Thank you.
> /Sree
>
>
>
>
> On Saturday, August 31, 2019, 10:19:06 PM PDT, Wangda Tan <
> wheele...@gmail.com> wrote:
>
>
> Hi all,
>
> As we discussed in the previous thread [1],
>
> I just moved the spin-off proposal to CWIKI and completed all TODO parts.
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>
> If you have interests to learn more about this. Please review the proposal
> let me know if you have any questions/suggestions for the proposal. This
> will be sent to board post voting passed. (And please note that the
> previous voting thread [2] to move Submarine to a separate Github repo is a
> necessary effort to move Submarine to a separate Apache project but not
> sufficient so I sent two separate voting thread.)
>
> Please let me know if I missed anyone in the proposal, and reply if you'd
> like to be included in the project.
>
> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>

[VOTE] Moving Submarine to a separate Apache project proposal

[Wangda Tan]

Hi all,

As we discussed in the previous thread [1],

I just moved the spin-off proposal to CWIKI and completed all TODO parts.

https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal

If you have interests to learn more about this. Please review the proposal
let me know if you have any questions/suggestions for the proposal. This
will be sent to board post voting passed. (And please note that the
previous voting thread [2] to move Submarine to a separate Github repo is a
necessary effort to move Submarine to a separate Apache project but not
sufficient so I sent two separate voting thread.)

Please let me know if I missed anyone in the proposal, and reply if you'd
like to be included in the project.

This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.

Thanks,
Wangda Tan

[1]
https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
[2]
https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Wangda Tan]

Got some trouble in creating git repo using Apache self-service. Created
https://issues.apache.org/jira/browse/INFRA-18964

Thanks,
Wangda

On Thu, Aug 29, 2019 at 11:52 PM Wangda Tan  wrote:

> Thanks everyone for voting.
>
> We got 12 binding +1s from committers. The vote passed.
>
> We will go ahead to create a repo "submarine", and will allow existing
> Hadoop committers to commit code to submarine.
>
> Best,
> Wangda
>
> On Thu, Aug 29, 2019 at 10:50 PM Arpit Agarwal 
> wrote:
>
>> +1
>>
>> > On Aug 23, 2019, at 7:05 PM, Wangda Tan  wrote:
>> >
>> > Hi devs,
>> >
>> > This is a voting thread to move Submarine source code, documentation
>> from
>> > Hadoop repo to a separate Apache Git repo. Which is based on
>> discussions of
>> >
>> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>> > Contributors who have permissions to push to Hadoop Git repository will
>> > have permissions to push to the new Submarine repository.
>> >
>> > This voting thread will run for 7 days and will end at Aug 30th.
>> >
>> > Please let me know if you have any questions.
>> >
>> > Thanks,
>> > Wangda Tan
>>
>>

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Wangda Tan]

Thanks everyone for voting.

We got 12 binding +1s from committers. The vote passed.

We will go ahead to create a repo "submarine", and will allow existing
Hadoop committers to commit code to submarine.

Best,
Wangda

On Thu, Aug 29, 2019 at 10:50 PM Arpit Agarwal 
wrote:

> +1
>
> > On Aug 23, 2019, at 7:05 PM, Wangda Tan  wrote:
> >
> > Hi devs,
> >
> > This is a voting thread to move Submarine source code, documentation from
> > Hadoop repo to a separate Apache Git repo. Which is based on discussions
> of
> >
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
> >
> > Contributors who have permissions to push to Hadoop Git repository will
> > have permissions to push to the new Submarine repository.
> >
> > This voting thread will run for 7 days and will end at Aug 30th.
> >
> > Please let me know if you have any questions.
> >
> > Thanks,
> > Wangda Tan
>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

I just put it to CWIKI:
https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches

Any thoughts? If no I will go ahead to send it to user/dev/general email
lists. (Should we send it to general?)

On Wed, Aug 28, 2019 at 5:03 PM Wangda Tan  wrote:

> Hi folks,
> Thanks all for sharing thoughts.
>
> I suggest moving branch-2 last release related discussion to a separate
> thread.
>
> We got 10 binding +1s. And no -1 to the proposal. I'm going to put this to
> WIKI first. And feel free to create a ticket and update our website.
>
> Thanks,
> Wangda
>
> On Wed, Aug 28, 2019 at 2:45 AM Konstantin Shvachko 
> wrote:
>
>> +1 for the proposal.
>>
>> I thought we already EOL-ed 2.6 though some time ago.
>>
>> Thanks,
>> --Konstantin
>>
>> On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
>>
>> > Hi all,
>> >
>> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
>> > and 3.0 EOL. This is based on discussions of [1]
>> >
>> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
>> >
>> > Please feel free to share your thoughts.
>> >
>> > Thanks,
>> > Wangda
>> >
>> > [1]
>> >
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
>> > ,
>> >
>>
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

Hi folks,
Thanks all for sharing thoughts.

I suggest moving branch-2 last release related discussion to a separate
thread.

We got 10 binding +1s. And no -1 to the proposal. I'm going to put this to
WIKI first. And feel free to create a ticket and update our website.

Thanks,
Wangda

On Wed, Aug 28, 2019 at 2:45 AM Konstantin Shvachko 
wrote:

> +1 for the proposal.
>
> I thought we already EOL-ed 2.6 though some time ago.
>
> Thanks,
> --Konstantin
>
> On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Hi all,

We received comments and suggestions from contributors, committers and PMC
members regarding the proposal:
https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit?ts=5d529ec0

@Vinod Kumar Vavilapalli  could you provide suggestions
regarding what we should do next? Could you help to send this to the ASF
board?

Thanks,
Wangda Tan

On Tue, Aug 13, 2019 at 4:36 PM Wangda Tan  wrote:

> Hi folks,
>
> I just drafted a proposal which is targetted to send to PMC list and board
> for thoughts. Thanks Xun Liu for providing thoughts about future
> directions/architecture, and reviews from Keqiu Hu.
>
> Title: "Apache Submarine for Apache Top-Level Project"
>
>
> https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit
>
> I plan to send it to PMC list/board next Monday, so any
> comments/suggestions are welcome.
>
> Thanks,
> Wangda
>
>
> On Tue, Jul 30, 2019 at 6:01 PM 俊平堵  wrote:
>
>> Thanks Vinod for these great suggestions. I agree most of your comments
>> above.
>>  "For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1?". IIUC, this should be treated as
>> feature branch merge, so may be 3 committer +1 is needed here according to
>> https://hadoop.apache.org/bylaws.html?
>>
>> bq. Can somebody who have cycles and been on the ASF lists for a while
>> look into the process here?
>> I can check with ASF members who has experience on this if no one haven't
>> yet.
>>
>> Thanks,
>>
>> Junping
>>
>> Vinod Kumar Vavilapalli  于2019年7月29日周一 下午9:46写道：
>>
>>> Looks like there's a meaningful push behind this.
>>>
>>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>>> this enthusiasm turns into building a real, independent but more
>>> importantly a sustainable community.
>>>
>>> Given that there were two official releases off the Apache Hadoop
>>> project, I doubt if you'd need to go through the incubator process. Instead
>>> you can directly propose a new TLP at ASF board. The last few times this
>>> happened was with ORC, and long before that with Hive, HBase etc. Can
>>> somebody who have cycles and been on the ASF lists for a while look into
>>> the process here?
>>>
>>> For the Apache Hadoop community, this will be treated simply as
>>> code-change and so need a committer +1? You can be more gently by formally
>>> doing a vote once a process doc is written down.
>>>
>>> Back to the sustainable community point, as part of drafting this
>>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>>> PMC/Committers can exercise their will to join this new project as
>>> PMC/Committers respectively without any additional constraints.
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
>>> >
>>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>>> from
>>> > 20+ contributors!
>>> >
>>> > So I think we should move it forward, any suggestions about what we
>>> should
>>> > do?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
>>> >
>>> >> +1, This is neo from TiDB & TiKV community.
>>> >> Thanks Xun for bring this up.
>>> >>
>>> >> Our CNCF project's open source distributed KV storage system TiKV,
>>> >> Hadoop submarine's machine learning engine helps us to optimize data
>>> >> storage,
>>> >> helping us solve some problems in data hotspots and data shuffers.
>>> >>
>>> >> We are ready to improve the performance of TiDB in our open source
>>> >> distributed relational database TiDB and also using the hadoop
>>> submarine
>>> >> machine learning engine.
>>> >>
>>> >> I think if submarine can be independent, it will develop faster and
>>> better.
>>> >> Thanks to the hadoop community for developing submarine!
>>> >>
>>> >> Best Regards,
>>> >> neo
>>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>>> >> https://github.com/tikv
>>> >>
>>> >> Xun Liu  于2019年7月22日周一 下午4:07写道：
>>> >>
>>> >>> @adam.antal
>>> >>>
>&g

[VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Wangda Tan]

Hi devs,

This is a voting thread to move Submarine source code, documentation from
Hadoop repo to a separate Apache Git repo. Which is based on discussions of
https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E

Contributors who have permissions to push to Hadoop Git repository will
have permissions to push to the new Submarine repository.

This voting thread will run for 7 days and will end at Aug 30th.

Please let me know if you have any questions.

Thanks,
Wangda Tan

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

Hi Steve,

The proposal is to EOL for the following branches:

[2.0.x - 2.7.x]
[3.0.x]

2.8.x, 2.9.x, 2.10.x (not released yet), 3.1.x (and later) are not EOL.

Thoughts?

On Sat, Aug 24, 2019 at 1:40 AM Steve Loughran  wrote:

>
>
> On Wed, Aug 21, 2019 at 4:03 AM Wangda Tan  wrote:
>
>> Hi all,
>>
>> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
>> and 3.0 EOL. This is based on discussions of [1]
>>
>
> 3.0 inclusive? i.e the non EOl ones being 2.8+ and 3.1+?
>

[VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Wangda Tan]

Hi all,

This is a vote thread to mark any versions smaller than 2.7 (inclusive),
and 3.0 EOL. This is based on discussions of [1]

This discussion runs for 7 days and will conclude on Aug 28 Wed.

Please feel free to share your thoughts.

Thanks,
Wangda

[1]
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
,

Re: Make EOL branches to public?

[Wangda Tan]

Thank you all for suggestions. Let me send a vote email to mark 2.6, 2.7,
3.0 EOL.

- Wangda

On Wed, Aug 21, 2019 at 9:34 AM Akira Ajisaka  wrote:

> +1
>
> Thank you for the discussion.
>
> -Akira
>
> On Wed, Aug 21, 2019 at 5:51 AM Wei-Chiu Chuang 
> wrote:
> >
> > +1
> > I feel like one year of inactivity is a good sign that the community is
> not
> > interested in the branch any more.
> >
> > On Fri, Aug 16, 2019 at 3:14 AM Wangda Tan  wrote:
> >
> > > Hi folks,
> > >
> > > Want to hear your thoughts about what we should do to make some
> branches
> > > EOL. We discussed a couple of times before in dev lists and PMC list.
> > > However, we couldn't get a formal process of EOL. According to the
> > > discussion. It is hard to decide it based on time like "After 1st
> release,
> > > EOL in 2 years". Because community members still want to maintain it
> and
> > > developers still want to get a newer version released.
> > >
> > > However, without a public place to figure out which release will be
> EOL, it
> > > is very hard for users to choose the right releases to upgrade and
> develop.
> > >
> > > So I want to propose to make an agreement about making a public EOL
> wiki
> > > page and create a process to EOL a release:
> > >
> > > The process I'm thinking is very simple: If no volunteer to do a
> > > maintenance release in a short to mid-term (like 3 months to 1 or 1.5
> > > year). We will claim a release is EOL. After EOL community can still
> choose
> > > to do a security-only release.
> > >
> > > Here's a list which I can think about:
> > >
> > > 1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
> > > 2) 2.7.x (Last released at Apr 2018)
> > > 4) 3.0.x (Last released at May 2018)
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Wangda
> > >
>

Make EOL branches to public?

[Wangda Tan]

Hi folks,

Want to hear your thoughts about what we should do to make some branches
EOL. We discussed a couple of times before in dev lists and PMC list.
However, we couldn't get a formal process of EOL. According to the
discussion. It is hard to decide it based on time like "After 1st release,
EOL in 2 years". Because community members still want to maintain it and
developers still want to get a newer version released.

However, without a public place to figure out which release will be EOL, it
is very hard for users to choose the right releases to upgrade and develop.

So I want to propose to make an agreement about making a public EOL wiki
page and create a process to EOL a release:

The process I'm thinking is very simple: If no volunteer to do a
maintenance release in a short to mid-term (like 3 months to 1 or 1.5
year). We will claim a release is EOL. After EOL community can still choose
to do a security-only release.

Here's a list which I can think about:

1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
2) 2.7.x (Last released at Apr 2018)
4) 3.0.x (Last released at May 2018)

Thoughts?

Thanks,
Wangda

Re: [DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Thanks all for the suggestions and volunteering run these releases:

I just updated roadmap of Hadoop wiki:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

Now I put 5 releases and planned date is 2019. We have 3 releases have
volunteered RM (Release Manager):

- 3.2.1: Rohith
- 2.8.6: Junping
- 2.10.0: Jonathan

We still need RM for 3.3.0 and 3.1.3.

Also, from my past experience, it will be helpful to get a co-RM (or shadow
RM) to do release together since release will have some effort which two
RMs can share some workload.

Can you help to update Roadmap wiki and put an estimated release date,
feature freeze date, etc. Which we can starting release cycle sooner if
possible?

Thanks,
Wangda


On Fri, Aug 16, 2019 at 5:00 PM Rohith Sharma K S 
wrote:

> Hi Wangda,
>
> Thanks for initiating this. I would like to nominate myself for 3.2.1
> Release Management.
>
> -Rohith Sharma K S
>
> On Sat, 10 Aug 2019 at 08:29, Wangda Tan  wrote:
>
> > Hi all,
> >
> > Hope this email finds you well
> >
> > I want to hear your thoughts about what should be the release plan for
> > 2019.
> >
> > In 2018, we released:
> > - 1 maintenance release of 2.6
> > - 3 maintenance releases of 2.7
> > - 3 maintenance releases of 2.8
> > - 3 releases of 2.9
> > - 4 releases of 3.0
> > - 2 releases of 3.1
> >
> > Total 16 releases in 2018.
> >
> > In 2019, by far we only have two releases:
> > - 1 maintenance release of 3.1
> > - 1 minor release of 3.2.
> >
> > However, the community put a lot of efforts to stabilize features of
> > various release branches.
> > There're:
> > - 217 fixed patches in 3.1.3 [1]
> > - 388 fixed patches in 3.2.1 [2]
> > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> >
> > I think it is the time to do maintenance releases of 3.1/3.2 and do a
> minor
> > release for 3.3.0.
> >
> > In addition, I saw community discussion to do a 2.8.6 release for
> security
> > fixes.
> >
> > Any other releases? I think there're release plans for Ozone as well. And
> > please add your thoughts.
> >
> > Volunteers welcome! If you have interests to run a release as Release
> > Manager (or co-Resource Manager), please respond to this email thread so
> we
> > can coordinate.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > fixVersion = 3.1.3
> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > fixVersion = 3.2.1
> > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > fixVersion = 3.3.0
> >
>

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Hi folks,

I just drafted a proposal which is targetted to send to PMC list and board
for thoughts. Thanks Xun Liu for providing thoughts about future
directions/architecture, and reviews from Keqiu Hu.

Title: "Apache Submarine for Apache Top-Level Project"

https://docs.google.com/document/d/1kE_f-r-ANh9qOeapdPwQPHhaJTS7IMiqDQAS8ESi4TA/edit

I plan to send it to PMC list/board next Monday, so any
comments/suggestions are welcome.

Thanks,
Wangda


On Tue, Jul 30, 2019 at 6:01 PM 俊平堵  wrote:

> Thanks Vinod for these great suggestions. I agree most of your comments
> above.
>  "For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1?". IIUC, this should be treated as
> feature branch merge, so may be 3 committer +1 is needed here according to
> https://hadoop.apache.org/bylaws.html?
>
> bq. Can somebody who have cycles and been on the ASF lists for a while
> look into the process here?
> I can check with ASF members who has experience on this if no one haven't
> yet.
>
> Thanks,
>
> Junping
>
> Vinod Kumar Vavilapalli  于2019年7月29日周一 下午9:46写道：
>
>> Looks like there's a meaningful push behind this.
>>
>> Given the desire is to fork off Apache Hadoop, you'd want to make sure
>> this enthusiasm turns into building a real, independent but more
>> importantly a sustainable community.
>>
>> Given that there were two official releases off the Apache Hadoop
>> project, I doubt if you'd need to go through the incubator process. Instead
>> you can directly propose a new TLP at ASF board. The last few times this
>> happened was with ORC, and long before that with Hive, HBase etc. Can
>> somebody who have cycles and been on the ASF lists for a while look into
>> the process here?
>>
>> For the Apache Hadoop community, this will be treated simply as
>> code-change and so need a committer +1? You can be more gently by formally
>> doing a vote once a process doc is written down.
>>
>> Back to the sustainable community point, as part of drafting this
>> proposal, you'd definitely want to make sure all of the Apache Hadoop
>> PMC/Committers can exercise their will to join this new project as
>> PMC/Committers respectively without any additional constraints.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
>> >
>> > Thanks everybody for sharing your thoughts. I saw positive feedbacks
>> from
>> > 20+ contributors!
>> >
>> > So I think we should move it forward, any suggestions about what we
>> should
>> > do?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
>> >
>> >> +1, This is neo from TiDB & TiKV community.
>> >> Thanks Xun for bring this up.
>> >>
>> >> Our CNCF project's open source distributed KV storage system TiKV,
>> >> Hadoop submarine's machine learning engine helps us to optimize data
>> >> storage,
>> >> helping us solve some problems in data hotspots and data shuffers.
>> >>
>> >> We are ready to improve the performance of TiDB in our open source
>> >> distributed relational database TiDB and also using the hadoop
>> submarine
>> >> machine learning engine.
>> >>
>> >> I think if submarine can be independent, it will develop faster and
>> better.
>> >> Thanks to the hadoop community for developing submarine!
>> >>
>> >> Best Regards,
>> >> neo
>> >> www.pingcap.com / https://github.com/pingcap/tidb /
>> >> https://github.com/tikv
>> >>
>> >> Xun Liu  于2019年7月22日周一 下午4:07写道：
>> >>
>> >>> @adam.antal
>> >>>
>> >>> The submarine development team has completed the following
>> preparations:
>> >>> 1. Established a temporary test repository on Github.
>> >>> 2. Change the package name of hadoop submarine from
>> org.hadoop.submarine
>> >> to
>> >>> org.submarine
>> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>> >>> 4. On the Github docked travis-ci system, all test cases have been
>> >> tested;
>> >>> 5. Several Hadoop submarine users completed the system test using the
>> >> code
>> >>> in this repository.
>> >>>
>> >>> 赵欣  于2019年7月22日周一 上午9:38写道：
>> >>>
>> >>>> Hi
>> >>>>
>

[DISCUSS] Hadoop 2019 Release Planning

[Wangda Tan]

Hi all,

Hope this email finds you well

I want to hear your thoughts about what should be the release plan for
2019.

In 2018, we released:
- 1 maintenance release of 2.6
- 3 maintenance releases of 2.7
- 3 maintenance releases of 2.8
- 3 releases of 2.9
- 4 releases of 3.0
- 2 releases of 3.1

Total 16 releases in 2018.

In 2019, by far we only have two releases:
- 1 maintenance release of 3.1
- 1 minor release of 3.2.

However, the community put a lot of efforts to stabilize features of
various release branches.
There're:
- 217 fixed patches in 3.1.3 [1]
- 388 fixed patches in 3.2.1 [2]
- 1172 fixed patches in 3.3.0 [3] (OMG!)

I think it is the time to do maintenance releases of 3.1/3.2 and do a minor
release for 3.3.0.

In addition, I saw community discussion to do a 2.8.6 release for security
fixes.

Any other releases? I think there're release plans for Ozone as well. And
please add your thoughts.

Volunteers welcome! If you have interests to run a release as Release
Manager (or co-Resource Manager), please respond to this email thread so we
can coordinate.

Thanks,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
fixVersion = 3.1.3
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
fixVersion = 3.2.1
[3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
fixVersion = 3.3.0

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Thanks Vinod, the proposal to make it be TLP definitely a great suggestion.
I will draft a proposal and keep the thread posted.

Best,
Wangda

On Mon, Jul 29, 2019 at 3:46 PM Vinod Kumar Vavilapalli 
wrote:

> Looks like there's a meaningful push behind this.
>
> Given the desire is to fork off Apache Hadoop, you'd want to make sure
> this enthusiasm turns into building a real, independent but more
> importantly a sustainable community.
>
> Given that there were two official releases off the Apache Hadoop project,
> I doubt if you'd need to go through the incubator process. Instead you can
> directly propose a new TLP at ASF board. The last few times this happened
> was with ORC, and long before that with Hive, HBase etc. Can somebody who
> have cycles and been on the ASF lists for a while look into the process
> here?
>
> For the Apache Hadoop community, this will be treated simply as
> code-change and so need a committer +1? You can be more gently by formally
> doing a vote once a process doc is written down.
>
> Back to the sustainable community point, as part of drafting this
> proposal, you'd definitely want to make sure all of the Apache Hadoop
> PMC/Committers can exercise their will to join this new project as
> PMC/Committers respectively without any additional constraints.
>
> Thanks
> +Vinod
>
> > On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
> >
> > Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> > 20+ contributors!
> >
> > So I think we should move it forward, any suggestions about what we
> should
> > do?
> >
> > Best,
> > Wangda
> >
> > On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
> >
> >> +1, This is neo from TiDB & TiKV community.
> >> Thanks Xun for bring this up.
> >>
> >> Our CNCF project's open source distributed KV storage system TiKV,
> >> Hadoop submarine's machine learning engine helps us to optimize data
> >> storage,
> >> helping us solve some problems in data hotspots and data shuffers.
> >>
> >> We are ready to improve the performance of TiDB in our open source
> >> distributed relational database TiDB and also using the hadoop submarine
> >> machine learning engine.
> >>
> >> I think if submarine can be independent, it will develop faster and
> better.
> >> Thanks to the hadoop community for developing submarine!
> >>
> >> Best Regards,
> >> neo
> >> www.pingcap.com / https://github.com/pingcap/tidb /
> >> https://github.com/tikv
> >>
> >> Xun Liu  于2019年7月22日周一 下午4:07写道：
> >>
> >>> @adam.antal
> >>>
> >>> The submarine development team has completed the following
> preparations:
> >>> 1. Established a temporary test repository on Github.
> >>> 2. Change the package name of hadoop submarine from
> org.hadoop.submarine
> >> to
> >>> org.submarine
> >>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> >>> 4. On the Github docked travis-ci system, all test cases have been
> >> tested;
> >>> 5. Several Hadoop submarine users completed the system test using the
> >> code
> >>> in this repository.
> >>>
> >>> 赵欣  于2019年7月22日周一 上午9:38写道：
> >>>
> >>>> Hi
> >>>>
> >>>> I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> >> are
> >>>> a major in electrical engineering. Our teaching teams and students use
> >>>> bigoop submarine for big data analysis and automation control of
> >>> electrical
> >>>> equipment.
> >>>>
> >>>> Many thanks to the hadoop community for providing us with machine
> >>> learning
> >>>> tools like submarine.
> >>>>
> >>>> I wish hadoop submarine is getting better and better.
> >>>>
> >>>>
> >>>> ==
> >>>> 赵欣
> >>>> 东南大学电气工程学院
> >>>>
> >>>> -
> >>>>
> >>>> Zhao XIN
> >>>>
> >>>> School of Electrical Engineering
> >>>>
> >>>> ==
> >>>> 2019-07-18
> >>>>
> >>>>
> >>>> *From:* Xun Liu 
> >>>> *Date:* 2019-07-18 09:46
> >>>> *To:* xinzhao 
> >>>> *Subject:* Fwd: 

Re: Any thoughts making Submarine a separate Apache project?

[Wangda Tan]

Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu  于2019年7月22日周一 下午4:07写道：
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣  于2019年7月22日周一 上午9:38写道：
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu 
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao 
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > -- Forwarded message -
> > > 发件人： dashuiguailu...@gmail.com 
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth , runlin zhang <
> > > runlin...@gmail.com>
> > > Cc: Xun Liu , common-dev <
> > common-...@hadoop.apache.org>,
> > > yarn-dev , hdfs-dev <
> > > hdfs-...@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-...@hadoop.apache.org>
> > >
> > >
> > > +1 ，Good idea, we are very much looking forward to it.
> > >
> > > --
> > > dashuiguailu...@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth 
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang 
> > > *CC:* Xun Liu ; Hadoop Common
> > > ; yarn-dev ;
> > > Hdfs-dev ; mapreduce-dev
> > > ; submarine-dev
> > > 
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang  wrote:
> > >
> > > > +1 ，That will be great ！
> > > >
> > > > > 在 2019年7月10日，下午3:34，Xun Liu  写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the 

Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang

[Wangda Tan]

Congrats!

Best,
Wangda

On Tue, Jul 16, 2019 at 10:37 AM 杨弢(杨弢)  wrote:

> Thanks everyone.
> I'm so honored to be an Apache Hadoop Committer, I will keep working on
> this great project and contribute more. Thanks.
>
> Best Regards,
> Tao Yang
>
>
> --
> 发件人：Naganarasimha Garla 
> 发送时间：2019年7月15日(星期一) 17:55
> 收件人：Weiwei Yang 
> 抄 送：yarn-dev ; Hadoop Common <
> common-...@hadoop.apache.org>; mapreduce-dev <
> mapreduce-dev@hadoop.apache.org>; Hdfs-dev 
> 主 题：Re: [ANNOUNCE] New Apache Hadoop Committer - Tao Yang
>
> Congrats and welcome Tao Yang!
>
> Regards
> + Naga
>
> On Mon, 15 Jul 2019, 17:54 Weiwei Yang,  wrote:
>
> > Hi Dear Apache Hadoop Community
> >
> > It's my pleasure to announce that Tao Yang has been elected as an Apache
> > Hadoop committer, this is to recognize his contributions to Apache Hadoop
> > YARN project.
> >
> > Congratulations and welcome on board!
> >
> > Weiwei
> > (On behalf of the Apache Hadoop PMC)
> >
>
>

Re: [Announcement] Apache Hadoop Submarine 0.2.0 released!

[Wangda Tan]

That's great news! Thanks for everybody who helping this release!

Best,
Wangda

On Tue, Jul 2, 2019 at 11:40 PM Zhankun Tang  wrote:

> Hi folks,
>
> I'm glad to announce that the Apache Hadoop community
> has voted to release Apache Hadoop Submarine 0.2.0.
>
> Apache Hadoop Submarine is a project to allows infra engineer/data
> scientist
> to run deep learning applications resource management platform (like YARN,
> K8s).
>
> This release contains below new features:
>
>1. Linkedin’s TonY runtime support in Submarine
>2. PyTorch enabled in Submarine with both YARN native service runtime
>(single node) and TonY runtime
>3. Support uber jar of Submarine to submit the job
>4. The YAML file to describe a job
>5. The Notebook support (by Apache Zeppelin Submarine interpreter)
>
> For more information and to download, please check:
> https://hadoop.apache.org/submarine/
>
> For submarine jar files as project dependencies, please check:
> https://mvnrepository.com/search?q=submarine
>
> Tons of thanks to everyone who contributed to the release, and everyone in
> the Apache Hadoop community! The release is a result of work from many
> contributors. Thank you for all of them.
>
> BR,
> Zhankun
>

Re: Agenda & More Information about Hadoop Community Meetup @ Palo Alto, June 26

[Wangda Tan]

A friendly reminder,

The meetup will take place tomorrow at 9:00 AM PDT to 4:00 PM PDT.

The address is: 395 Page Mill Rd, Palo Alto, CA 94306
We’ll be in the Bigtop conference room on the 1st floor. Go left after
coming through the main entrance, and it will be on the right.

Zoom: https://cloudera.zoom.us/j/606607666

Please let me know if you have any questions. If you haven't RSVP yet,
please go ahead and RSVP so we can better prepare food, seat, etc.

Thanks,
Wangda

On Wed, Jun 19, 2019 at 4:49 PM Wangda Tan  wrote:

> Hi All,
>
> I want to let you know that we have confirmed most of the agenda for
> Hadoop Community Meetup. It will be a whole day event.
>
> Agenda & Dial-In info because see below, *please RSVP
> at https://www.meetup.com/Hadoop-Contributors/events/262055924/
> <https://www.meetup.com/Hadoop-Contributors/events/262055924/>*
>
> Huge thanks to Daniel Templeton, Wei-Chiu Chuang, Christina Vu for helping
> with organizing and logistics.
>
> *Please help to promote meetup information on Twitter, LinkedIn, etc.
> Appreciated! *
>
> Best,
> Wangda
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *AM:9:00: Arrival and check-in--9:30 -
> 10:15:-Talk: Hadoop storage in cloud-native
> environmentsAbstract: Hadoop is a mature storage system but designed years
> before the cloud-native movement. Kubernetes and other cloud-native tools
> are emerging solutions for containerized environments but sometimes they
> require different approaches.In this presentation we would like to share
> our experiences to run Apache Hadoop Ozone in Kubernetes and the connection
> point to other cloud-native ecosystem elements. We will compare the
> benefits and drawbacks to use Kubernetes and Hadoop storage together and
> show our current achievements and future plans.Speaker: Marton Elek
> (Cloudera)10:20 - 11:00:--Talk: Selective Wire Encryption In
> HDFSAbstract: Wire data encryption is a key component of the Hadoop
> Distributed File System (HDFS). However, such encryption enforcement comes
> in as an all-or-nothing feature. In our use case at LinkedIn, we would like
> to selectively expose fast unencrypted access to fully managed internal
> clients, which can be trusted, while only expose encrypted access to
> clients outside of the trusted circle with higher security risks. That way
> we minimize performance overhead for trusted internal clients while still
> securing data from potential outside threats. Our design extends HDFS
> NameNode to run on multiple ports, connecting to different NameNode ports
> would end up with different levels of encryption protection. This
> protection then gets enforced for both NameNode RPC and the subsequent data
> transfers to/from DataNode. This approach comes with minimum operational
> and performance overhead.Speaker: Konstantin Shvachko (LinkedIn), Chen
> Liang (LinkedIn)11:10 - 11:55:-Talk: YuniKorn: Next Generation
> Scheduling for YARN and K8sAbstract: We will talk about our open source
> work - YuniKorn scheduler project (Y for YARN, K for K8s, uni- for Unified)
> brings long-wanted features such as hierarchical queues, fairness between
> users/jobs/queues, preemption to Kubernetes; and it brings service
> scheduling enhancements to YARN. Any improvements to this scheduler can
> benefit both Kubernetes and YARN community.Speaker: Wangda Tan
> (Cloudera)PM:12:00 - 12:55 Lunch Break (Provided by
> Cloudera)1:00 -
> 1:25---Talk: Yarn Efficiency at UberAbstract: We will present the
> work done at Uber to improve YARN cluster utilization and job SOA with
> elastic resource management, low compute workload on passive datacenter,
> preemption, larger container, etc. We will also go through YARN upgrade in
> order to adopt new features and talk about the challenges.Speaker: Aihua Xu
> (Uber), Prashant Golash (Uber)1:30 - 2:10 One more
> talk-2:20 - 4:00---BoF sessions &
> Breakout Sessions & Group discussions: Talk about items like JDK 11
> support, next releases (2.10.0, 3.3.0, etc.), Hadoop on Cloud, etc.4:00:
> Reception provided by
> Cloudera.==Join Zoom
> Meetinghttps://cloudera.zoom.us/j/116816195
> <https://cloudera.zoom.us/j/116816195>*
>

Re: [VOTE] Release Apache Hadoop Submarine 0.2.0 - RC0

[Wangda Tan]

+1 Binding. Tested in local cluster and reviewed docs.

Thanks!

On Wed, Jun 19, 2019 at 3:20 AM Sunil Govindan  wrote:

> +1 binding
>
> - tested in local cluster.
> - tried tony run time as well
> - doc seems fine now.
>
> - Sunil
>
>
> On Thu, Jun 6, 2019 at 6:53 PM Zhankun Tang  wrote:
>
> > Hi folks,
> >
> > Thanks to all of you who have contributed in this submarine 0.2.0
> release.
> > We now have a release candidate (RC0) for Apache Hadoop Submarine 0.2.0.
> >
> >
> > The Artifacts for this Submarine-0.2.0 RC0 are available here:
> >
> > https://home.apache.org/~ztang/submarine-0.2.0-rc0/
> >
> >
> > It's RC tag in git is "submarine-0.2.0-RC0".
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1221/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on 13th June at 11:59 pm
> > PST.
> >
> >
> >
> > The highlights of this release.
> >
> > 1. Linkedin's TonY runtime support in Submarine
> >
> > 2. PyTorch enabled in Submarine with both YARN native service runtime
> > (single node) and TonY runtime
> >
> > 3. Support uber jar of Submarine to submit the job
> >
> > 4. The YAML file to describe a job
> >
> > 5. The Notebook support (by Apache Zeppelin Submarine interpreter)
> >
> >
> > Thanks to Sunil, Wangda, Xun, Zac, Keqiu, Szilard for helping me in
> > preparing the release.
> >
> > I have done a few testing with my pseudo cluster. My +1 (non-binding) to
> > start.
> >
> >
> >
> > Regards,
> > Zhankun
> >
>

Agenda & More Information about Hadoop Community Meetup @ Palo Alto, June 26

[Wangda Tan]

Hi All,

I want to let you know that we have confirmed most of the agenda for Hadoop
Community Meetup. It will be a whole day event.

Agenda & Dial-In info because see below, *please RSVP
at https://www.meetup.com/Hadoop-Contributors/events/262055924/
<https://www.meetup.com/Hadoop-Contributors/events/262055924/>*

Huge thanks to Daniel Templeton, Wei-Chiu Chuang, Christina Vu for helping
with organizing and logistics.

*Please help to promote meetup information on Twitter, LinkedIn, etc.
Appreciated! *

Best,
Wangda

























































*AM:9:00: Arrival and check-in--9:30 -
10:15:-Talk: Hadoop storage in cloud-native
environmentsAbstract: Hadoop is a mature storage system but designed years
before the cloud-native movement. Kubernetes and other cloud-native tools
are emerging solutions for containerized environments but sometimes they
require different approaches.In this presentation we would like to share
our experiences to run Apache Hadoop Ozone in Kubernetes and the connection
point to other cloud-native ecosystem elements. We will compare the
benefits and drawbacks to use Kubernetes and Hadoop storage together and
show our current achievements and future plans.Speaker: Marton Elek
(Cloudera)10:20 - 11:00:--Talk: Selective Wire Encryption In
HDFSAbstract: Wire data encryption is a key component of the Hadoop
Distributed File System (HDFS). However, such encryption enforcement comes
in as an all-or-nothing feature. In our use case at LinkedIn, we would like
to selectively expose fast unencrypted access to fully managed internal
clients, which can be trusted, while only expose encrypted access to
clients outside of the trusted circle with higher security risks. That way
we minimize performance overhead for trusted internal clients while still
securing data from potential outside threats. Our design extends HDFS
NameNode to run on multiple ports, connecting to different NameNode ports
would end up with different levels of encryption protection. This
protection then gets enforced for both NameNode RPC and the subsequent data
transfers to/from DataNode. This approach comes with minimum operational
and performance overhead.Speaker: Konstantin Shvachko (LinkedIn), Chen
Liang (LinkedIn)11:10 - 11:55:-Talk: YuniKorn: Next Generation
Scheduling for YARN and K8sAbstract: We will talk about our open source
work - YuniKorn scheduler project (Y for YARN, K for K8s, uni- for Unified)
brings long-wanted features such as hierarchical queues, fairness between
users/jobs/queues, preemption to Kubernetes; and it brings service
scheduling enhancements to YARN. Any improvements to this scheduler can
benefit both Kubernetes and YARN community.Speaker: Wangda Tan
(Cloudera)PM:12:00 - 12:55 Lunch Break (Provided by
Cloudera)1:00 -
1:25---Talk: Yarn Efficiency at UberAbstract: We will present the
work done at Uber to improve YARN cluster utilization and job SOA with
elastic resource management, low compute workload on passive datacenter,
preemption, larger container, etc. We will also go through YARN upgrade in
order to adopt new features and talk about the challenges.Speaker: Aihua Xu
(Uber), Prashant Golash (Uber)1:30 - 2:10 One more
talk-2:20 - 4:00---BoF sessions &
Breakout Sessions & Group discussions: Talk about items like JDK 11
support, next releases (2.10.0, 3.3.0, etc.), Hadoop on Cloud, etc.4:00:
Reception provided by
Cloudera.==Join Zoom
Meetinghttps://cloudera.zoom.us/j/116816195
<https://cloudera.zoom.us/j/116816195>*

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Thanks @Wei-Chiu Chuang  . updated gdoc

On Tue, Jun 18, 2019 at 7:35 PM Wei-Chiu Chuang  wrote:

> Thanks Wangda,
>
> I just like to make a correction -- the .ics calendar file says the first
> Wednesday for HDFS/cloud connector is in Mandarin whereas on the gdoc is to
> host it on the third Wednesday.
>
> On Tue, Jun 18, 2019 at 5:29 PM Wangda Tan  wrote:
>
> > Hi Folks,
> >
> > I just updated doc:
> >
> >
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
> > with
> > dial-in information, notes, etc.
> >
> > Here's a calendar to subscribe:
> >
> >
> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
> >
> > I'm thinking to give it a try from next week, any suggestions?
> >
> > Thanks,
> > Wangda
> >
> > On Fri, Jun 14, 2019 at 4:02 PM Wangda Tan  wrote:
> >
> > > And please let me know if you can help with coordinate logistics stuff,
> > > cross-checking, etc. Let's spend some time next week to get it
> finalized.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan 
> wrote:
> > >
> > >> Hi Folks,
> > >>
> > >> Yufei: Agree with all your opinions.
> > >>
> > >> Anu: it might be more efficient to use Google doc to track meeting
> > >> minutes and we can put them together.
> > >>
> > >> I just put the proposal to
> > >>
> >
> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ
> > ,
> > >> you can check if the proposal time works or not. If you agree, we can
> go
> > >> ahead to add meeting link, google doc, etc.
> > >>
> > >> If you want to have edit permissions, please drop a private email to
> me
> > >> so I will add you.
> > >>
> > >> We still need more hosts, in each track, ideally we should have at
> least
> > >> 3 hosts per track just like HDFS blocks :), please volunteer, so we
> can
> > >> have enough members to run the meeting.
> > >>
> > >> Let's shoot by end of the next week, let's get all logistics done and
> > >> starting community sync up series from the week of Jun 25th.
> > >>
> > >> Thanks,
> > >> Wangda
> > >>
> > >> Thanks,
> > >> Wangda
> > >>
> > >>
> > >>
> > >> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer  >
> > >> wrote:
> > >>
> > >>> For Ozone, we have started using the Wiki itself as the agenda and
> > after
> > >>> the meeting is over, we convert it into the meeting notes.
> > >>> Here is an example, the project owner can edit and maintain it, it is
> > >>> like 10 mins work - and allows anyone to add stuff into the agenda
> too.
> > >>>
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
> > >>>
> > >>> --Anu
> > >>>
> > >>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu 
> > wrote:
> > >>>
> > >>>> +1 for this idea. Thanks Wangda for bringing this up.
> > >>>>
> > >>>> Some comments to share:
> > >>>>
> > >>>>- Agenda needed to be posted ahead of meeting and welcome any
> > >>>> interested
> > >>>>party to contribute to topics.
> > >>>>- We should encourage more people to attend. That's whole point
> of
> > >>>> the
> > >>>>meeting.
> > >>>>- Hopefully, this can mitigate the situation that some patches
> are
> > >>>>waiting for review for ever, which turns away new contributors.
> > >>>>- 30m per session sounds a little bit short, we can try it out
> and
> > >>>> see
> > >>>>if extension is needed.
> > >>>>
> > >>>> Best,
> > >>>>
> > >>>> Yufei
> > >>>>
> > >>>> `This is not a contribution`
> > >>>>
> > >>>>
> > >>>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan 
> > wrote:
> > >>>>
> > >>>> > Hi Hadoop-devs,
> > &

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Hi Folks,

I just updated doc:
https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
with
dial-in information, notes, etc.

Here's a calendar to subscribe:
https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics

I'm thinking to give it a try from next week, any suggestions?

Thanks,
Wangda

On Fri, Jun 14, 2019 at 4:02 PM Wangda Tan  wrote:

> And please let me know if you can help with coordinate logistics stuff,
> cross-checking, etc. Let's spend some time next week to get it finalized.
>
> Thanks,
> Wangda
>
> On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan  wrote:
>
>> Hi Folks,
>>
>> Yufei: Agree with all your opinions.
>>
>> Anu: it might be more efficient to use Google doc to track meeting
>> minutes and we can put them together.
>>
>> I just put the proposal to
>> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ,
>> you can check if the proposal time works or not. If you agree, we can go
>> ahead to add meeting link, google doc, etc.
>>
>> If you want to have edit permissions, please drop a private email to me
>> so I will add you.
>>
>> We still need more hosts, in each track, ideally we should have at least
>> 3 hosts per track just like HDFS blocks :), please volunteer, so we can
>> have enough members to run the meeting.
>>
>> Let's shoot by end of the next week, let's get all logistics done and
>> starting community sync up series from the week of Jun 25th.
>>
>> Thanks,
>> Wangda
>>
>> Thanks,
>> Wangda
>>
>>
>>
>> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer 
>> wrote:
>>
>>> For Ozone, we have started using the Wiki itself as the agenda and after
>>> the meeting is over, we convert it into the meeting notes.
>>> Here is an example, the project owner can edit and maintain it, it is
>>> like 10 mins work - and allows anyone to add stuff into the agenda too.
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
>>>
>>> --Anu
>>>
>>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu  wrote:
>>>
>>>> +1 for this idea. Thanks Wangda for bringing this up.
>>>>
>>>> Some comments to share:
>>>>
>>>>- Agenda needed to be posted ahead of meeting and welcome any
>>>> interested
>>>>party to contribute to topics.
>>>>- We should encourage more people to attend. That's whole point of
>>>> the
>>>>meeting.
>>>>- Hopefully, this can mitigate the situation that some patches are
>>>>waiting for review for ever, which turns away new contributors.
>>>>- 30m per session sounds a little bit short, we can try it out and
>>>> see
>>>>if extension is needed.
>>>>
>>>> Best,
>>>>
>>>> Yufei
>>>>
>>>> `This is not a contribution`
>>>>
>>>>
>>>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:
>>>>
>>>> > Hi Hadoop-devs,
>>>> >
>>>> > Previous we have regular YARN community sync up (1 hr, biweekly, but
>>>> not
>>>> > open to public). Recently because of changes in our schedules, Less
>>>> folks
>>>> > showed up in the sync up for the last several months.
>>>> >
>>>> > I saw the K8s community did a pretty good job to run their sig
>>>> meetings,
>>>> > there's regular meetings for different topics, notes, agenda, etc.
>>>> Such as
>>>> >
>>>> >
>>>> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>>>> >
>>>> >
>>>> > For Hadoop community, there are less such regular meetings open to the
>>>> > public except for Ozone project and offline meetups or
>>>> Bird-of-Features in
>>>> > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
>>>> > Summit at Washington DC and Barcelona, and lots (50+) of folks join
>>>> the
>>>> > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it
>>>> is
>>>> > important to open such conversations to the public and let more
>>>> > folk/companies join.
>>>> >
>>>> > Discussed a small group of community members and wrote a short
>>>> proposal
>>>> > about the form, time and topic of the community sync up, thanks for
>>>> > everybody who have contributed to the proposal! Please feel free to
>>>> add
>>>> > your thoughts to the Proposal Google doc
>>>> > <
>>>> >
>>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>>> > >
>>>> > .
>>>> >
>>>> > Especially for the following parts:
>>>> > - If you have interests to run any of the community sync-ups, please
>>>> put
>>>> > your name to the table inside the proposal. We need more volunteers
>>>> to help
>>>> > run the sync-ups in different timezones.
>>>> > - Please add suggestions to the time, frequency and themes and feel
>>>> free to
>>>> > share your thoughts if we should do sync ups for other topics which
>>>> are not
>>>> > covered by the proposal.
>>>> >
>>>> > Link to the Proposal Google doc
>>>> > <
>>>> >
>>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>>> > >
>>>> >
>>>> > Thanks,
>>>> > Wangda Tan
>>>> >
>>>>
>>>

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

And please let me know if you can help with coordinate logistics stuff,
cross-checking, etc. Let's spend some time next week to get it finalized.

Thanks,
Wangda

On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan  wrote:

> Hi Folks,
>
> Yufei: Agree with all your opinions.
>
> Anu: it might be more efficient to use Google doc to track meeting minutes
> and we can put them together.
>
> I just put the proposal to
> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ,
> you can check if the proposal time works or not. If you agree, we can go
> ahead to add meeting link, google doc, etc.
>
> If you want to have edit permissions, please drop a private email to me so
> I will add you.
>
> We still need more hosts, in each track, ideally we should have at least 3
> hosts per track just like HDFS blocks :), please volunteer, so we can have
> enough members to run the meeting.
>
> Let's shoot by end of the next week, let's get all logistics done and
> starting community sync up series from the week of Jun 25th.
>
> Thanks,
> Wangda
>
> Thanks,
> Wangda
>
>
>
> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer 
> wrote:
>
>> For Ozone, we have started using the Wiki itself as the agenda and after
>> the meeting is over, we convert it into the meeting notes.
>> Here is an example, the project owner can edit and maintain it, it is
>> like 10 mins work - and allows anyone to add stuff into the agenda too.
>>
>>
>> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
>>
>> --Anu
>>
>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu  wrote:
>>
>>> +1 for this idea. Thanks Wangda for bringing this up.
>>>
>>> Some comments to share:
>>>
>>>- Agenda needed to be posted ahead of meeting and welcome any
>>> interested
>>>party to contribute to topics.
>>>- We should encourage more people to attend. That's whole point of the
>>>meeting.
>>>- Hopefully, this can mitigate the situation that some patches are
>>>waiting for review for ever, which turns away new contributors.
>>>- 30m per session sounds a little bit short, we can try it out and see
>>>if extension is needed.
>>>
>>> Best,
>>>
>>> Yufei
>>>
>>> `This is not a contribution`
>>>
>>>
>>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:
>>>
>>> > Hi Hadoop-devs,
>>> >
>>> > Previous we have regular YARN community sync up (1 hr, biweekly, but
>>> not
>>> > open to public). Recently because of changes in our schedules, Less
>>> folks
>>> > showed up in the sync up for the last several months.
>>> >
>>> > I saw the K8s community did a pretty good job to run their sig
>>> meetings,
>>> > there's regular meetings for different topics, notes, agenda, etc.
>>> Such as
>>> >
>>> >
>>> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>>> >
>>> >
>>> > For Hadoop community, there are less such regular meetings open to the
>>> > public except for Ozone project and offline meetups or
>>> Bird-of-Features in
>>> > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
>>> > Summit at Washington DC and Barcelona, and lots (50+) of folks join the
>>> > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
>>> > important to open such conversations to the public and let more
>>> > folk/companies join.
>>> >
>>> > Discussed a small group of community members and wrote a short proposal
>>> > about the form, time and topic of the community sync up, thanks for
>>> > everybody who have contributed to the proposal! Please feel free to add
>>> > your thoughts to the Proposal Google doc
>>> > <
>>> >
>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>> > >
>>> > .
>>> >
>>> > Especially for the following parts:
>>> > - If you have interests to run any of the community sync-ups, please
>>> put
>>> > your name to the table inside the proposal. We need more volunteers to
>>> help
>>> > run the sync-ups in different timezones.
>>> > - Please add suggestions to the time, frequency and themes and feel
>>> free to
>>> > share your thoughts if we should do sync ups for other topics which
>>> are not
>>> > covered by the proposal.
>>> >
>>> > Link to the Proposal Google doc
>>> > <
>>> >
>>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>>> > >
>>> >
>>> > Thanks,
>>> > Wangda Tan
>>> >
>>>
>>

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Hi Folks,

Yufei: Agree with all your opinions.

Anu: it might be more efficient to use Google doc to track meeting minutes
and we can put them together.

I just put the proposal to
https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ,
you can check if the proposal time works or not. If you agree, we can go
ahead to add meeting link, google doc, etc.

If you want to have edit permissions, please drop a private email to me so
I will add you.

We still need more hosts, in each track, ideally we should have at least 3
hosts per track just like HDFS blocks :), please volunteer, so we can have
enough members to run the meeting.

Let's shoot by end of the next week, let's get all logistics done and
starting community sync up series from the week of Jun 25th.

Thanks,
Wangda

Thanks,
Wangda



On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer 
wrote:

> For Ozone, we have started using the Wiki itself as the agenda and after
> the meeting is over, we convert it into the meeting notes.
> Here is an example, the project owner can edit and maintain it, it is like
> 10 mins work - and allows anyone to add stuff into the agenda too.
>
> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
>
> --Anu
>
> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu  wrote:
>
>> +1 for this idea. Thanks Wangda for bringing this up.
>>
>> Some comments to share:
>>
>>- Agenda needed to be posted ahead of meeting and welcome any
>> interested
>>party to contribute to topics.
>>- We should encourage more people to attend. That's whole point of the
>>meeting.
>>- Hopefully, this can mitigate the situation that some patches are
>>waiting for review for ever, which turns away new contributors.
>>- 30m per session sounds a little bit short, we can try it out and see
>>if extension is needed.
>>
>> Best,
>>
>> Yufei
>>
>> `This is not a contribution`
>>
>>
>> On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:
>>
>> > Hi Hadoop-devs,
>> >
>> > Previous we have regular YARN community sync up (1 hr, biweekly, but not
>> > open to public). Recently because of changes in our schedules, Less
>> folks
>> > showed up in the sync up for the last several months.
>> >
>> > I saw the K8s community did a pretty good job to run their sig meetings,
>> > there's regular meetings for different topics, notes, agenda, etc. Such
>> as
>> >
>> >
>> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>> >
>> >
>> > For Hadoop community, there are less such regular meetings open to the
>> > public except for Ozone project and offline meetups or Bird-of-Features
>> in
>> > Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
>> > Summit at Washington DC and Barcelona, and lots (50+) of folks join the
>> > Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
>> > important to open such conversations to the public and let more
>> > folk/companies join.
>> >
>> > Discussed a small group of community members and wrote a short proposal
>> > about the form, time and topic of the community sync up, thanks for
>> > everybody who have contributed to the proposal! Please feel free to add
>> > your thoughts to the Proposal Google doc
>> > <
>> >
>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>> > >
>> > .
>> >
>> > Especially for the following parts:
>> > - If you have interests to run any of the community sync-ups, please put
>> > your name to the table inside the proposal. We need more volunteers to
>> help
>> > run the sync-ups in different timezones.
>> > - Please add suggestions to the time, frequency and themes and feel
>> free to
>> > share your thoughts if we should do sync ups for other topics which are
>> not
>> > covered by the proposal.
>> >
>> > Link to the Proposal Google doc
>> > <
>> >
>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
>> > >
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>>
>

[DISCUSS] A unified and open Hadoop community sync up schedule?

[Wangda Tan]

Hi Hadoop-devs,

Previous we have regular YARN community sync up (1 hr, biweekly, but not
open to public). Recently because of changes in our schedules, Less folks
showed up in the sync up for the last several months.

I saw the K8s community did a pretty good job to run their sig meetings,
there's regular meetings for different topics, notes, agenda, etc. Such as
https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit


For Hadoop community, there are less such regular meetings open to the
public except for Ozone project and offline meetups or Bird-of-Features in
Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
Summit at Washington DC and Barcelona, and lots (50+) of folks join the
Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
important to open such conversations to the public and let more
folk/companies join.

Discussed a small group of community members and wrote a short proposal
about the form, time and topic of the community sync up, thanks for
everybody who have contributed to the proposal! Please feel free to add
your thoughts to the Proposal Google doc
<https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#>
.

Especially for the following parts:
- If you have interests to run any of the community sync-ups, please put
your name to the table inside the proposal. We need more volunteers to help
run the sync-ups in different timezones.
- Please add suggestions to the time, frequency and themes and feel free to
share your thoughts if we should do sync ups for other topics which are not
covered by the proposal.

Link to the Proposal Google doc
<https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#>

Thanks,
Wangda Tan

Re: [VOTE] Release Apache Hadoop Submarine 0.2.0 - RC0

[Wangda Tan]

Thanks Zhankun and everybody who helped this release.

Could you share the link to the user doc?

Best,
Wangda

On Thu, Jun 6, 2019 at 8:55 AM Xun Liu  wrote:

> Zhankun,
>
> Thank you release submarine 0.2.0
>
> I tested it with submarine-0.2.0-rc0. The functions can be used normally.
> But I found a little problem, The command parameter --verbose does not
> take effect.
> But this does not affect the use, It is recommended that after
> submarine-0.2.0-rc0 is released, Can be improved.
>
> +1
>
>
> > On Jun 6, 2019, at 9:23 PM, Zhankun Tang  wrote:
> >
> > Hi folks,
> >
> > Thanks to all of you who have contributed in this submarine 0.2.0
> release.
> > We now have a release candidate (RC0) for Apache Hadoop Submarine 0.2.0.
> >
> >
> > The Artifacts for this Submarine-0.2.0 RC0 are available here:
> >
> > https://home.apache.org/~ztang/submarine-0.2.0-rc0/
> >
> >
> > It's RC tag in git is "submarine-0.2.0-RC0".
> >
> >
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1221/
> >
> >
> > This vote will run 7 days (5 weekdays), ending on 13th June at 11:59 pm
> PST.
> >
> >
> >
> > The highlights of this release.
> >
> > 1. Linkedin's TonY runtime support in Submarine
> >
> > 2. PyTorch enabled in Submarine with both YARN native service runtime
> > (single node) and TonY runtime
> >
> > 3. Support uber jar of Submarine to submit the job
> >
> > 4. The YAML file to describe a job
> >
> > 5. The Notebook support (by Apache Zeppelin Submarine interpreter)
> >
> >
> > Thanks to Sunil, Wangda, Xun, Zac, Keqiu, Szilard for helping me in
> > preparing the release.
> >
> > I have done a few testing with my pseudo cluster. My +1 (non-binding) to
> > start.
> >
> >
> >
> > Regards,
> > Zhankun
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [ANNOUNCE] Eric Badger is now a committer!

[Wangda Tan]

Congratulations, Eric.

Welcome aboard!

Best,
Wangda


On Tue, Mar 5, 2019 at 2:26 PM Sree V 
wrote:

> Congratulations, Eric.
>
>
>
> Thank you./Sree
>
>
>
> On Tuesday, March 5, 2019, 12:50:20 PM PST, Ayush Saxena <
> ayush...@gmail.com> wrote:
>
>  Congratulations Eric!!!
>
> -Ayush
>
> > On 05-Mar-2019, at 11:34 PM, Chandni Singh 
> wrote:
> >
> > Congratulations Eric!
> >
> > On Tue, Mar 5, 2019 at 9:32 AM Jim Brennan
> >  wrote:
> >
> >> Congratulations Eric!
> >>
> >> On Tue, Mar 5, 2019 at 11:20 AM Eric Payne  >> .invalid>
> >> wrote:
> >>
> >>> It is my pleasure to announce that Eric Badger has accepted an
> invitation
> >>> to become a Hadoop Core committer.
> >>>
> >>> Congratulations, Eric! This is well-deserved!
> >>>
> >>> -Eric Payne
> >>>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

[Wangda Tan]

Thanks everybody for voting!

We got 10 binding votes and 15 non-binding votes. The vote passed. We will
work with the community to start new subproject.

Best,
Wangda

On Mon, Feb 4, 2019 at 9:06 PM Prabhu Josephraj 
wrote:

> +1 (non-binding)
>
> On Tue, Feb 5, 2019 at 10:30 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
>> +1
>>
>> On Sat, 2 Feb 2019, 09:51 Rohith Sharma K S > wrote:
>>
>> > +1
>> >
>> > On Sat, Feb 2, 2019, 3:54 AM Wangda Tan  wrote:
>> >
>> > > Hi all,
>> > >
>> > > According to positive feedbacks from the thread [1]
>> > >
>> > > This is vote thread to start a new subproject named "hadoop-submarine"
>> > > which follows the release process already established for ozone.
>> > >
>> > > The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
>> > >
>> > > Thanks,
>> > > Wangda Tan
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
>> > >
>> >
>>
>

[ANNOUNCE] Apache Hadoop 3.1.2 release

[Wangda Tan]

It gives us great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.1.2.

IMPORTANT NOTES

3.1.2 is the second stable release of 3.1 line which is considered to be
production-ready.

Hadoop 3.1.2 brings a number of enhancements.

The Hadoop community fixed 325 JIRAs [1] in total as part of the 3.1.2
release. Of these fixes:

Apache Hadoop 3.1.2 contains a number of significant features and
enhancements. A few of them are noted below.

- Nvidia-docker-plugin v2 support for GPU support on YARN.
- YARN service upgrade improvements and bug fixes.
- YARN UIv2 improvements and bug fixes.
- AliyunOSS related improvements and bug fixes.
- Docker on YARN support related improvements and bug fixes.

Please see the Hadoop 3.1.2 CHANGES for the detailed list of issues
resolved. The release news is posted on the Apache Hadoop website too, you
can go to the downloads section.

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! The release is a result of direct and indirect
efforts from many contributors, listed below are the those who contributed
directly by submitting patches and/or reporting issues. (148 contributors,
Sorted by ID)

BilwaST, Charo Zhang, GeLiXin, Harsha1206, Huachao, Jim_Brennan, LiJinglun,
Naganarasimha, OrDTesters, RANith, Rakesh_Shah, Ray Burgemeestre, Sen Zhao,
SoumyaPN, SouryakantaDwivedy, Tao Yang, Zian Chen, abmodi, adam.antal,
ajayydv, ajisakaa, akhilpb, akhilsnaik, amihalyi, arpitagarwal, aw,
ayushtkn, banditka, belugabehr, benlau, bibinchundatt, billie.rinaldi,
boky01, bolke, borisvu, botong, brahmareddy, briandburton, bsteinbach,
candychencan, ccondit-target, charanh, cheersyang, cltlfcjin, collinma,
crh, csingh, csun, daisuke.kobayashi, daryn, dibyendu_hadoop,
dineshchitlangia, ebadger, eepayne, elgoiri, erwaman, eyang, fengchuang,
ferhui, fly_in_gis, gabor.bota, gezapeti, gsaha, haibochen, hexiaoqiao,
hfyang20071, hgadre, jeagles, jhung, jiangjianfei, jianliang.wu,
jira.shegalov, jiwq, jlowe, jojochuang, jonBoone, kanwaljeets, karams,
kennethlnnn, kgyrtkirk, kihwal, knanasi, kshukla, laszlok, leftnoteasy,
leiqiang, liaoyuxiangqin, linyiqun, ljain, lukmajercak, maniraj...@gmail.com,
masatana, nandakumar131, oliverhuh...@gmail.com, oshevchenko, pbacsko,
peruguusha, photogamrun, pj.fanning, prabham, pradeepambati, pranay_singh,
revans2, rkanter, rohithsharma, shaneku...@gmail.com, shubham.dewan,
shuzirra, shv, simonprewo, sinago, smeng, snemeth, sodonnell,
sreenivasulureddy, ssath...@hortonworks.com, ssulav, ste...@apache.org,
study, suma.shivaprasad, sunilg, surendrasingh, tangzhankun, tarunparimi,
tasanuma0829, templedf, thinktaocs, tlipcon, tmarquardt, trjianjianjiao,
uranus, varun_saxena, vinayrpet, vrushalic, wilfreds,
write2kish...@gmail.com, wujinhu, xiaochen, xiaoheipangzi, xkrogen,
yangjiandan, yeshavora, yiran, yoelee, yuzhih...@gmail.com, zichensun,
zvenczel

Wangda Tan and Sunil Govind

[1] JIRA query: project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
Fixed AND fixVersion = 3.1.2 ORDER BY key ASC, updated ASC, created DESC,
priority DESC

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

[Wangda Tan]

+1, make sense to me.

On Tue, Feb 5, 2019 at 3:29 PM Konstantin Shvachko 
wrote:

> +1 Makes sense to me.
>
> Thanks,
> --Konst
>
> On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:
>
> > Hello,
> >
> > Starting a vote based on the discuss thread [1] for moving branch-2
> > precommit/nightly test builds to openjdk8. After this change, the test
> > phase for precommit builds [2] and branch-2 nightly build [3] will run on
> > openjdk8. To maintain source compatibility, these builds will still run
> > their compile phase for branch-2 on openjdk7 as they do now (in addition
> to
> > compiling on openjdk8).
> >
> > Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
> >
> > [2]
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
> >
> > [3]
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> >
> > Jonathan Hung
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Wangda Tan]

Thanks everyone for voting, we have 6 binding votes and 6 non-binding
votes, no vetos. The vote passed. We will go ahead to stage this release.

- Wangda and Sunil.

On Tue, Feb 5, 2019 at 12:05 PM Billie Rinaldi 
wrote:

> Thanks Sunil, the new source tarball matches the RC tag and its checksum
> and signature look good.
>
> Billie
>
> On Tue, Feb 5, 2019 at 10:50 AM Sunil G  wrote:
>
> > Thanks Billie for pointing out.
> > I have updated source by removing patchprocess and extra line create
> > release.
> >
> > Also updated checksum as well.
> >
> > @bil...@apache.org   @Wangda Tan 
> > please help to verify this changed bit once.
> >
> > Thanks
> > Sunil
> >
> > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi 
> > wrote:
> >
> >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> >> patchprocess directory with some yetus code in it. Also, the file
> >> dev-support/bin/create-release file has the following line added:
> >>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> >> gpg-agent):1"
> >>
> >> I think we are probably due for an overall review of LICENSE and NOTICE.
> >> I saw some idiosyncrasies there but nothing that looked like a blocker.
> >>
> >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> >>
> >>> Hi Folks,
> >>>
> >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> >>>
> >>> The artifacts are available here:
> >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> >>>
> >>> The RC tag in git is release-3.1.2-RC1:
> >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> >>>
> >>> This vote will run 5 days from now.
> >>>
> >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> >>>
> >>> We have done testing with a pseudo cluster and distributed shell job.
> >>>
> >>> My +1 to start.
> >>>
> >>> Best,
> >>> Wangda Tan and Sunil Govindan
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.2)
> >>> ORDER BY priority DESC
> >>>
> >>
>

[VOTE] Propose to start new Hadoop sub project "submarine"

[Wangda Tan]

Hi all,

According to positive feedbacks from the thread [1]

This is vote thread to start a new subproject named "hadoop-submarine"
which follows the release process already established for ozone.

The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.

Thanks,
Wangda Tan

[1]
https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E

Re: [DISCUSS] Making submarine to different release model like Ozone

[Wangda Tan]

Eric,
Thanks for your reconsideration. We will definitely try best to not break
compatibilities, etc. like how we did to other components!

Really appreciate everybody's support, thoughts, suggestions shared on this
thread. Given the discussion went very positive, I will go ahead to send a
voting thread.

Best,
Wangda

On Fri, Feb 1, 2019 at 2:06 PM Eric Yang  wrote:

> If HDFS or YARN breaks compatibility with Submarine, it will require to
> make release to catch up with the latest Hadoop changes.  On
> hadoop.apache.org website, the latest news may always have Submarine on
> top to repair compatibility with latest of Hadoop.  This may overwhelm any
> interesting news that may happen in Hadoop space.  I don’t like to see that
> happen, but unavoidable with independent release cycle.  Maybe there is a
> good way to avoid this with help of release manager to ensure that
> Hadoop/Submarine don’t break compatibility frequently.
>
>
>
> For me to lift my veto, release managers of independent release cycles
> need to take responsibility to ensure X version of Hadoop is tested with Y
> version of Submarine.  Release managers will have to do more work to ensure
> the defined combination works.  With the greater responsibility of release
> management comes with its own reward.  Seasoned PMC may be nominated to
> become Apache Member, which will help with Submarine to enter Apache
> Incubator when time is right.  Hence, I will withdraw my veto and let
> Submarine set its own course.
>
>
>
> Good luck Wangda.
>
>
>
> Regards,
>
> Eric
>
>
>
> *From: *Wangda Tan 
> *Date: *Friday, February 1, 2019 at 10:52 AM
> *To: *Eric Yang 
> *Cc: *Weiwei Yang , Xun Liu ,
> Hadoop Common , "yarn-...@hadoop.apache.org"
> , Hdfs-dev , "
> mapreduce-dev@hadoop.apache.org" 
> *Subject: *Re: [DISCUSS] Making submarine to different release model like
> Ozone
>
>
>
> Thanks everyone for sharing thoughts!
>
>
>
> Eric, appreciate your suggestions. But there are many examples to have
> separate releases, like Hive's storage API, OZone, etc. For loosely coupled
> sub-projects, it gonna be great (at least for most of the users) to have
> separate releases so new features can be faster consumed and iterated. From
> above feedbacks from developers and users, I think it is also what people
> want.
>
>
>
> Another concern you mentioned is Submarine is aligned with Hadoop project
> goals. From feedbacks we can see, it attracts companies continue using
> Hadoop to solve their ML/DL requirements, it also created a good feedback
> loop, many issues faced, and some new functionalities added by Submarine
> went back to Hadoop. Such as localization files, directories. GPU topology
> related enhancement, etc.
>
>
>
> We will definitely use this sub-project opportunity to fast grow both
> Submarine and Hadoop, try to get fast release cycles for both of the
> projects. And for your suggestion about Apache incubator, we can reconsider
> it once Submarine becomes a more independent project, now it is still too
> small and too much overhead to go through the process, I don't want to stop
> the fast-growing community for months to go through incubator process for
> now.
>
>
>
> I really hope my comment can help you reconsider the veto. :)
>
>
>
> Thanks,
>
> Wangda
>
>
>
> On Fri, Feb 1, 2019 at 9:39 AM Eric Yang  wrote:
>
> Submarine is an application built for YARN framework, but it does not have
> strong dependency on YARN development.  For this kind of projects, it would
> be best to enter Apache Incubator cycles to create a new community.  Apache
> commons is the only project other than Incubator that has independent
> release cycles.  The collection is large, and the project goal is
> ambitious.  No one really knows which component works with each other in
> Apache commons.  Hadoop is a much more focused project on distributed
> computing framework and not incubation sandbox.  For alignment with Hadoop
> goals, and we want to prevent Hadoop project to be overloaded while
> allowing good ideas to be carried forwarded in Apache incubator.  Put on my
> Apache Member hat, my vote is -1 to allow more independent subproject
> release cycle in Hadoop project that does not align with Hadoop project
> goals.
>
> Apache incubator process is highly recommended for Submarine:
> https://incubator.apache.org/policy/process.html This allows Submarine to
> develop for older version of Hadoop like Spark works with multiple versions
> of Hadoop.
>
> Regards,
> Eric
>
> On 1/31/19, 10:51 PM, "Weiwei Yang"  wrote:
>
> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the

Re: [DISCUSS] Making submarine to different release model like Ozone

[Wangda Tan]

Thanks everyone for sharing thoughts!

Eric, appreciate your suggestions. But there are many examples to have
separate releases, like Hive's storage API, OZone, etc. For loosely coupled
sub-projects, it gonna be great (at least for most of the users) to have
separate releases so new features can be faster consumed and iterated. From
above feedbacks from developers and users, I think it is also what people
want.

Another concern you mentioned is Submarine is aligned with Hadoop project
goals. From feedbacks we can see, it attracts companies continue using
Hadoop to solve their ML/DL requirements, it also created a good feedback
loop, many issues faced, and some new functionalities added by Submarine
went back to Hadoop. Such as localization files, directories. GPU topology
related enhancement, etc.

We will definitely use this sub-project opportunity to fast grow both
Submarine and Hadoop, try to get fast release cycles for both of the
projects. And for your suggestion about Apache incubator, we can reconsider
it once Submarine becomes a more independent project, now it is still too
small and too much overhead to go through the process, I don't want to stop
the fast-growing community for months to go through incubator process for
now.

I really hope my comment can help you reconsider the veto. :)

Thanks,
Wangda

On Fri, Feb 1, 2019 at 9:39 AM Eric Yang  wrote:

> Submarine is an application built for YARN framework, but it does not have
> strong dependency on YARN development.  For this kind of projects, it would
> be best to enter Apache Incubator cycles to create a new community.  Apache
> commons is the only project other than Incubator that has independent
> release cycles.  The collection is large, and the project goal is
> ambitious.  No one really knows which component works with each other in
> Apache commons.  Hadoop is a much more focused project on distributed
> computing framework and not incubation sandbox.  For alignment with Hadoop
> goals, and we want to prevent Hadoop project to be overloaded while
> allowing good ideas to be carried forwarded in Apache incubator.  Put on my
> Apache Member hat, my vote is -1 to allow more independent subproject
> release cycle in Hadoop project that does not align with Hadoop project
> goals.
>
> Apache incubator process is highly recommended for Submarine:
> https://incubator.apache.org/policy/process.html This allows Submarine to
> develop for older version of Hadoop like Spark works with multiple versions
> of Hadoop.
>
> Regards,
> Eric
>
> On 1/31/19, 10:51 PM, "Weiwei Yang"  wrote:
>
> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last year, the
> community grows fast and quiet collaborative. I can see the reasons to get
> it release faster in its own cycle. And at the same time, the Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at Netease
> Research Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine learning
> engine from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data team's
> hadoop technology stack,
> > It avoids the needs to increase the manpower investment in learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN cluster to run
> machine learning,
> > which makes the utilization of server resources more efficient, and
> reserves a lot of human and material resources in our previous years.
> >
> > Our team have finished the test and deployment of the Submarine and
> will provide the service to our e-commerce department (
> http://www.kaola.com/) shortly.
> >
> > We also plan to provides the Submarine engine in our existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle training,
> > 2) News department (http://www.163.com) needs news recommendation,
> > 3) Mailbox department (http://www.163.com) requires anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5) Education department (http://www.youdao.com) requires voice
> recognition,
> > 6) Massive Open Online Courses (https://open.163.com/) requires
> multilingual translation and so on.
> >
>

[DISCUSS] Making submarine to different release model like Ozone

[Wangda Tan]

Hi devs,

Since we started submarine-related effort last year, we received a lot of
feedbacks, several companies (such as Netease, China Mobile, etc.)  are
trying to deploy Submarine to their Hadoop cluster along with big data
workloads. Linkedin also has big interests to contribute a Submarine TonY (
https://github.com/linkedin/TonY) runtime to allow users to use the same
interface.

>From what I can see, there're several issues of putting Submarine under
yarn-applications directory and have same release cycle with Hadoop:

1) We started 3.2.0 release at Sep 2018, but the release is done at Jan
2019. Because of non-predictable blockers and security issues, it got
delayed a lot. We need to iterate submarine fast at this point.

2) We also see a lot of requirements to use Submarine on older Hadoop
releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a
short time, but the requirement to run deep learning is urgent to them. We
should decouple Submarine from Hadoop version.

And why we wanna to keep it within Hadoop? First, Submarine included some
innovation parts such as enhancements of user experiences for YARN
services/containerization support which we can add it back to Hadoop later
to address common requirements. In addition to that, we have a big overlap
in the community developing and using it.

There're several proposals we have went through during Ozone merge to trunk
discussion:
https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E

I propose to adopt Ozone model: which is the same master branch, different
release cycle, and different release branch. It is a great example to show
agile release we can do (2 Ozone releases after Oct 2018) with less
overhead to setup CI, projects, etc.

*Links:*
- JIRA: https://issues.apache.org/jira/browse/YARN-8135
- Design doc
<https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit>
- User doc
<https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html>
(3.2.0
release)
- Blogposts, {Submarine} : Running deep learning workloads on Apache Hadoop
<https://hortonworks.com/blog/submarine-running-deep-learning-workloads-apache-hadoop/>,
(Chinese Translation: Link <https://www.jishuwen.com/d/2Vpu>)
- Talks: Strata Data Conf NY
<https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68289>

Thoughts?

Thanks,
Wangda Tan

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC0

[Wangda Tan]

Hi Rohith,

Thanks for reporting and fixing this issue, let’s redo the issue and will
send another vote shortly.

Best,
Wangda

On Sun, Jan 27, 2019 at 10:03 PM Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> @Wangda Tan  I have pushed the changes to branch-3.1
> and branch-3.1.2. Verified hadoop-3.1.2 branch build for functionalities.
> Should new RC to be given OR same could be continued with updated
> repositories?
>
> On Mon, 28 Jan 2019 at 10:54, Rohith Sharma K S 
> wrote:
>
>> -1, I found an issue in ATSv2 initialization in NodeManager.  This causes
>> none of the ATSv2 events published from NodeManager. I have created
>> YARN-9242 for tracking the same.
>>
>> -Rohith Sharma K S
>>
>> On Fri, 25 Jan 2019 at 11:40, Wangda Tan  wrote:
>>
>>> Hi folks,
>>>
>>> With tons of help from Sunil, we have created RC0 for Apache Hadoop
>>> 3.1.2.
>>> The artifacts are available here:
>>>
>>> *http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/
>>> <http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/>*
>>>
>>> The RC tag in git is release-3.1.2-RC0:
>>> https://github.com/apache/hadoop/commits/release-3.1.2-RC0
>>>
>>> The maven artifacts are available via repository.apache.org at
>>> *
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1212/
>>> <
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1212/
>>> >*
>>>
>>> This vote will run 5 days from now.
>>>
>>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
>>>
>>> I have done testing with a pseudo cluster and distributed shell job. My
>>> +1
>>> to start.
>>>
>>> Best,
>>> Wangda Tan and Sunil Govind
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
>>> ORDER BY priority DESC
>>>
>> --
Sent from Gmail Mobile

[VOTE] Release Apache Hadoop 3.1.2 - RC0

[Wangda Tan]

Hi folks,

With tons of help from Sunil, we have created RC0 for Apache Hadoop 3.1.2.
The artifacts are available here:

*http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/
<http://home.apache.org/~sunilg/hadoop-3.1.2-RC0/>*

The RC tag in git is release-3.1.2-RC0:
https://github.com/apache/hadoop/commits/release-3.1.2-RC0

The maven artifacts are available via repository.apache.org at
*https://repository.apache.org/content/repositories/orgapachehadoop-1212/
<https://repository.apache.org/content/repositories/orgapachehadoop-1212/>*

This vote will run 5 days from now.

3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan and Sunil Govind

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
ORDER BY priority DESC

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

onfiguration due to changes in [orgapachehadoop-1202
> (staging: open)] made by wangda...
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:40:50 INFO  [99598137-805254] -
> com.sonatype.nexus.staging.internal.task.RepositoryOpenTask - Staging
> repository created: orgapachehadoop-1202
> 2019-01-22 05:41:18 INFO  [99598137-805274] -
> com.sonatype.nexus.staging.internal.DefaultStagingManager - Closing staging
> repositories [orgapachehadoop-1202]
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> com.sonatype.nexus.staging.internal.task.StagingBackgroundTask - STARTED
> Closing staging repositories: [orgapachehadoop-1202]
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:18 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:20 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> org.sonatype.nexus.configuration.ModelUtils - Saving model
> /x1/nexus-work/conf/staging.xml
> 2019-01-22 05:41:22 WARN  [pool-1-thread-7] -
> com.sonatype.nexus.staging.internal.task.RepositoryCloseTask - One or more
> operations failed; aborting:
> com.sonatype.nexus.staging.StagingRulesFailedException: One or more rules
> have failed
> 2019-01-22 05:41:22 INFO  [pool-1-thread-7] -
> com.sonatype.nexus.staging.internal.task.StagingBackgroundTask - FINISHED
> Closing staging repositories: [orgapachehadoop-1202]
>
>
>
>
> On Jan 21, 2019, at 9:42 PM, Wangda Tan  wrote:
>
> Hi Chris,
>
> Thanks for helping the issue,
>
> Now the issue still exists but the process becomes much faster:
>
> failureMessage Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-project/3.1.2/hadoop-project-3.1.2.pom', check
> the logs.
> failureMessage Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-main/3.1.2/hadoop-main-3.1.2.pom', check the
> logs.
> failureMessage Failed to validate the pgp signature of
> '/org/apache/h

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi Chris,

Thanks for helping the issue,

Now the issue still exists but the process becomes much faster:

failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-project/3.1.2/hadoop-project-3.1.2.pom', check
the logs.
failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-main/3.1.2/hadoop-main-3.1.2.pom', check the
logs.
failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-build-tools/3.1.2/hadoop-build-tools-3.1.2.jar',
check the logs.
failureMessage Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-build-tools/3.1.2/hadoop-build-tools-3.1.2.pom',
check the logs.

Above is the error message, is it possible to get the actual failure
message from Nexus server?

Thanks,
Wangda

On Mon, Jan 21, 2019 at 5:18 PM Chris Lambertus  wrote:

> It looks like there are timeouts from some of the keyservers. I’ve trimmed
> the list again to only servers known to be working (ubuntu and
> sks-keyservers.net <http://sks-keyservers.net/>). Can you give it a try
> again?
>
> Brian, there are also a number of timeout errors related to central, but I
> think they are unrelated.
>
> com.sonatype.central.secure.nexus.plugin.internal.AuthtokenFetcherImpl -
> Failed to fetch authtoken: org.apache.http.conn.ConnectTimeoutException:
> Connect to secure.central.sonatype.com:443 [
> secure.central.sonatype.com/207.223.241.90] failed: connect timed out
>
> -Chris
>
>
> > On Jan 21, 2019, at 2:39 PM, Brian Fox  wrote:
> >
> > They keys file is irrelevant to Nexus. The only thing that matters is
> it’s in the mit pgp key ring.
> >
> > --Brian (mobile)
> >
> >
> > On Jan 21, 2019, at 3:34 PM, Wangda Tan  wheele...@gmail.com>> wrote:
> >
> >> I just checked on KEYS file, it doesn't show sig part. I updated KEYS
> file on Apache
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS <
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS> and made
> it be ultimately trusted.
> >>
> >> pub   rsa4096 2018-03-20 [SC]
> >>   4C899853CDDA4E40C60212B5B3FA653D57300D45
> >> uid   [ultimate] Wangda tan  wan...@apache.org>>
> >> sig 3B3FA653D57300D45 2018-03-20  Wangda tan  <mailto:wan...@apache.org>>
> >> sub   rsa4096 2018-03-20 [E]
> >> sig      B3FA653D57300D45 2018-03-20  Wangda tan  <mailto:wan...@apache.org>>
> >> But the error still remains same while closing repo, not sure how to
> get it resolved ..
> >>
> >>
> >> On Mon, Jan 21, 2019 at 9:14 AM Wangda Tan  <mailto:wheele...@gmail.com>> wrote:
> >> Hi David,
> >>
> >> Thanks for helping check this,
> >>
> >> I can see signatures on my key:
> >>
> >> pub  4096R/57300D45 <
> http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
>
> >>   Fingerprint=4C89 9853 CDDA 4E40 C602  12B5 B3FA 653D 5730 0D45
> >>
> >> uid Wangda tan mailto:wan...@apache.org>>
> >> sig  sig3  57300D45 <
> http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
> __ __ [selfsig] <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45
> >
> >> sig  sig   C36C5F0F <
> http://pool.sks-keyservers.net:11371/key/0x255ADF56C36C5F0F> 2018-04-05
> __ __ Vinod Kumar Vavilapalli (I am also known as
> @tshooter.)  <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x255ADF56C36C5F0F
> >
> >> sig  sig   F9CBBD4C <
> http://pool.sks-keyservers.net:11371/key/0x29ED86E6F9CBBD4C> 2018-11-08
> __ __ shikong  <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x29ED86E6F9CBBD4C
> >
> >>
> >> sub  4096R/D0C16F12 2018-03-20
> >> sig sbind  57300D45 <
> http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
> __ __ [] <
> http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45
> >
> >> And gpg --edit-key also shows:
> >>
> >> gpg --edit-key 4C899853CDDA4E40C60212B5B3FA653D57300D45
> >> gpg (GnuPG) 2.2.5; Copyright (C) 2018 Free Software Foundation, Inc.
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.
> >>
> >> Secret key is available.
> >>
> >> sec  rsa4096/B3FA653D57300D45
> >>  created: 2018-03-20  expires: never   usage: SC
> >>  trust: unknown   validity: unkn

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

I just checked on KEYS file, it doesn't show sig part. I updated KEYS file
on Apache https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and
made it be ultimately trusted.

pub   rsa4096 2018-03-20 [SC]
  4C899853CDDA4E40C60212B5B3FA653D57300D45
uid   [ultimate] Wangda tan 
sig 3B3FA653D57300D45 2018-03-20  Wangda tan 
sub   rsa4096 2018-03-20 [E]
sig  B3FA653D57300D45 2018-03-20  Wangda tan 

But the error still remains same while closing repo, not sure how to
get it resolved ..



On Mon, Jan 21, 2019 at 9:14 AM Wangda Tan  wrote:

> Hi David,
>
> Thanks for helping check this,
>
> I can see signatures on my key:
>
> pub  4096R/57300D45 
> <http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20
>Fingerprint=4C89 9853 CDDA 4E40 C602  12B5 B3FA 653D 5730 0D45
> uid Wangda tan 
> sig  sig3  57300D45 
> <http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20 
> __ __ [selfsig] 
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>
> sig  sig   C36C5F0F 
> <http://pool.sks-keyservers.net:11371/key/0x255ADF56C36C5F0F> 2018-04-05 
> __ __ Vinod Kumar Vavilapalli (I am also known as @tshooter.) 
>  
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x255ADF56C36C5F0F>
> sig  sig   F9CBBD4C 
> <http://pool.sks-keyservers.net:11371/key/0x29ED86E6F9CBBD4C> 2018-11-08 
> __ __ shikong  
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x29ED86E6F9CBBD4C>
> sub  4096R/D0C16F12 2018-03-20
> sig sbind  57300D45 
> <http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45> 2018-03-20 
> __ __ [] 
> <http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>
>
> And gpg --edit-key also shows:
>
> gpg --edit-key 4C899853CDDA4E40C60212B5B3FA653D57300D45
> gpg (GnuPG) 2.2.5; Copyright (C) 2018 Free Software Foundation, Inc.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Secret key is available.
>
> sec  rsa4096/B3FA653D57300D45
>  created: 2018-03-20  expires: never       usage: SC
>  trust: unknown   validity: unknown
> ssb  rsa4096/79CD893FD0C16F12
>  created: 2018-03-20  expires: never   usage: E
> [ unknown] (1). Wangda tan 
>
> Thanks,
> Wangda
>
> On Mon, Jan 21, 2019 at 9:08 AM David Nalley  wrote:
>
>> I wonder if it's because there are no signatures on your key.
>>
>> --David
>>
>> On Mon, Jan 21, 2019 at 11:57 AM Wangda Tan  wrote:
>> >
>> > Hi Brian,
>> >
>> > Here're links to my key:
>> >
>> > http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45
>> >
>> > http://pgp.mit.edu/pks/lookup?op=get=0xB3FA653D57300D45
>> >
>> > On Apache SVN:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Thanks,
>> > Wangda
>> >
>> > On Mon, Jan 21, 2019 at 6:51 AM Brian Demers 
>> wrote:
>> >>
>> >> Can you share the link to your key?
>> >>
>> >> -Brian
>> >>
>> >> On Jan 20, 2019, at 11:21 PM, Wangda Tan  wrote:
>> >>
>> >> Still couldn't figure out without locating the log on the Nexus
>> machine. With help from several committers and PMCs, we didn't see anything
>> wrong with my signing key.
>> >>
>> >> I don't want to delay 3.1.2 more because of this. Is it allowed for me
>> to publish artifacts (like tarball, source package, etc.) only and somebody
>> else to push Maven bits to Nexus. I believe Apache bylaw should allow that
>> because there're several releases have more than one release managers. If
>> it is not allowed, please take over the RM role if you have the bandwidth,
>> I think most works have been done except close the Nexus repo.
>> >>
>> >> Thanks,
>> >> Wangda
>> >>
>> >> On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan 
>> wrote:
>> >>>
>> >>> Spent several more hours trying to figure out the issue, still no
>> luck.
>> >>>
>> >>> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
>> appreciate if anybody could add some suggestions.
>> >>>
>> >>> Thanks,
>> >>> Wangda
>> >>>
>> >>> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan 
>> wrote:
>> >>>>
>> >>>> It 

Re: [NOTICE] branch-3.1.2 created for 3.1.2 release

[Wangda Tan]

Hi devs,

Caused by delays in releasing preparing, we plan to roll RC1 tomorrow.
Given the key issue is not resolved yet, I asked Sunil to be co-RM for this
release.

branch-3.1.2 is open for critical/blocker changes only, please add your
fixes to the branch, we will start rolling RC tomorrow (Tue) 9 AM PDT.

Thank,
Wangda


On Mon, Jan 7, 2019 at 10:13 AM Wangda Tan  wrote:

> Hi devs,
>
> I just created branch-3.1.2 for 3.1.2 release and pushed to remote.
> branch-3.1 is still open for new commits required for next 3.1.x release
> (which will be 3.1.3). In the meantime, branch-3.1.2 is closed for new
> commits unless further noticed. I'm working on RC0 of 3.1.2 release now.
>
> Best,
> Wangda
>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi David,

Thanks for helping check this,

I can see signatures on my key:

pub  4096R/57300D45
<http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45>
2018-03-20
 Fingerprint=4C89 9853 CDDA 4E40 C602  12B5 B3FA 653D 5730 0D45
uid Wangda tan 
sig  sig3  57300D45
<http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45>
2018-03-20 __ __ [selfsig]
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>
sig  sig   C36C5F0F
<http://pool.sks-keyservers.net:11371/key/0x255ADF56C36C5F0F>
2018-04-05 __ __ Vinod Kumar Vavilapalli (I am also
known as @tshooter.) 
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x255ADF56C36C5F0F>
sig  sig   F9CBBD4C
<http://pool.sks-keyservers.net:11371/key/0x29ED86E6F9CBBD4C>
2018-11-08 __ __ shikong 
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0x29ED86E6F9CBBD4C>
sub  4096R/D0C16F12 2018-03-20
sig sbind  57300D45
<http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45>
2018-03-20 __ __ []
<http://pool.sks-keyservers.net:11371/search/vindex/fingerprint/0xB3FA653D57300D45>

And gpg --edit-key also shows:

gpg --edit-key 4C899853CDDA4E40C60212B5B3FA653D57300D45
gpg (GnuPG) 2.2.5; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  rsa4096/B3FA653D57300D45
 created: 2018-03-20  expires: never   usage: SC
 trust: unknown   validity: unknown
ssb  rsa4096/79CD893FD0C16F12
 created: 2018-03-20  expires: never   usage: E
[ unknown] (1). Wangda tan 

Thanks,
Wangda

On Mon, Jan 21, 2019 at 9:08 AM David Nalley  wrote:

> I wonder if it's because there are no signatures on your key.
>
> --David
>
> On Mon, Jan 21, 2019 at 11:57 AM Wangda Tan  wrote:
> >
> > Hi Brian,
> >
> > Here're links to my key:
> >
> > http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45
> >
> > http://pgp.mit.edu/pks/lookup?op=get=0xB3FA653D57300D45
> >
> > On Apache SVN:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > Wangda
> >
> > On Mon, Jan 21, 2019 at 6:51 AM Brian Demers 
> wrote:
> >>
> >> Can you share the link to your key?
> >>
> >> -Brian
> >>
> >> On Jan 20, 2019, at 11:21 PM, Wangda Tan  wrote:
> >>
> >> Still couldn't figure out without locating the log on the Nexus
> machine. With help from several committers and PMCs, we didn't see anything
> wrong with my signing key.
> >>
> >> I don't want to delay 3.1.2 more because of this. Is it allowed for me
> to publish artifacts (like tarball, source package, etc.) only and somebody
> else to push Maven bits to Nexus. I believe Apache bylaw should allow that
> because there're several releases have more than one release managers. If
> it is not allowed, please take over the RM role if you have the bandwidth,
> I think most works have been done except close the Nexus repo.
> >>
> >> Thanks,
> >> Wangda
> >>
> >> On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan 
> wrote:
> >>>
> >>> Spent several more hours trying to figure out the issue, still no luck.
> >>>
> >>> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
> appreciate if anybody could add some suggestions.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan 
> wrote:
> >>>>
> >>>> It seems the problem still exists for me:
> >>>>
> >>>> Now the error message only contains:
> >>>>
> >>>> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
> check the logs.
> >>>> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
> check the logs.
> >>>>
> >>>> If anybody has access the Nexus node, could you please help to check
> what is the failure message?
> >>>>
> >>>> Thanks,
> >>>> Wangda
> >>>>
> >>>>
> >>>> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
> >>>>>
> >>>>> Good to know. The pool has occasionally had sync issues, but we're
> talking 3 times in the last 8-9 years.
>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi Brian,

Here're links to my key:

http://pool.sks-keyservers.net:11371/key/0xB3FA653D57300D45

http://pgp.mit.edu/pks/lookup?op=get=0xB3FA653D57300D45

On Apache SVN: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Thanks,
Wangda

On Mon, Jan 21, 2019 at 6:51 AM Brian Demers  wrote:

> Can you share the link to your key?
>
> -Brian
>
> On Jan 20, 2019, at 11:21 PM, Wangda Tan  wrote:
>
> Still couldn't figure out without locating the log on the Nexus machine.
> With help from several committers and PMCs, we didn't see anything wrong
> with my signing key.
>
> I don't want to delay 3.1.2 more because of this. Is it allowed for me to
> publish artifacts (like tarball, source package, etc.) only and somebody
> else to push Maven bits to Nexus. I believe Apache bylaw should allow that
> because there're several releases have more than one release managers. If
> it is not allowed, please take over the RM role if you have the bandwidth,
> I think most works have been done except close the Nexus repo.
>
> Thanks,
> Wangda
>
> On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan  wrote:
>
>> Spent several more hours trying to figure out the issue, still no luck.
>>
>> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
>> appreciate if anybody could add some suggestions.
>>
>> Thanks,
>> Wangda
>>
>> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan  wrote:
>>
>>> It seems the problem still exists for me:
>>>
>>> Now the error message only contains:
>>>
>>> failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
>>> check the logs.
>>> failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
>>> check the logs.
>>>
>>> If anybody has access the Nexus node, could you please help to check
>>> what is the failure message?
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
>>>
>>>> Good to know. The pool has occasionally had sync issues, but we're
>>>> talking 3 times in the last 8-9 years.
>>>>
>>>> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>>>>
>>>>> My key was pushed to the server with pgp about 1 year ago, and it
>>>>> worked
>>>>> well with the last Ratis release. So it should be synced between the
>>>>> key
>>>>> servers.
>>>>>
>>>>> But it seems that the INFRA solved the problem with shuffling the key
>>>>> server order (or it was an intermittent issue): see INFRA-17649
>>>>>
>>>>> Seems to be working now...
>>>>>
>>>>> Marton
>>>>>
>>>>>
>>>>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>>>>> > HI Brain,
>>>>> > Thanks for responding, could u share how to push to keys to Apache
>>>>> pgp pool?
>>>>> >
>>>>> > Best,
>>>>> > Wangda
>>>>> >
>>>>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox 
>>>>> wrote:
>>>>> >
>>>>> >> Did you push your key up to the pgp pool? That's what Nexus is
>>>>> validating
>>>>> >> against. It might take time to propagate if you just pushed it.
>>>>> >>
>>>>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton 
>>>>> wrote:
>>>>> >>
>>>>> >>> Seems to be an INFRA issue for me:
>>>>> >>>
>>>>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>>>>> >>> repository and it was ok, locally I verified it.
>>>>> >>>
>>>>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I
>>>>> got
>>>>> >>> the same problem even if it worked at last year during the 0.3.0
>>>>> >>> release. (I used exactly the same command)
>>>>> >>>
>>>>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>>>>> >>> suggested in the error message:
>>>>> >>>
>>>>> >>> https://issues.apache.org/jira/browse/I

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Still couldn't figure out without locating the log on the Nexus machine.
With help from several committers and PMCs, we didn't see anything wrong
with my signing key.

I don't want to delay 3.1.2 more because of this. Is it allowed for me to
publish artifacts (like tarball, source package, etc.) only and somebody
else to push Maven bits to Nexus. I believe Apache bylaw should allow that
because there're several releases have more than one release managers. If
it is not allowed, please take over the RM role if you have the bandwidth,
I think most works have been done except close the Nexus repo.

Thanks,
Wangda

On Thu, Jan 17, 2019 at 11:18 AM Wangda Tan  wrote:

> Spent several more hours trying to figure out the issue, still no luck.
>
> I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
> appreciate if anybody could add some suggestions.
>
> Thanks,
> Wangda
>
> On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan  wrote:
>
>> It seems the problem still exists for me:
>>
>> Now the error message only contains:
>>
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
>> check the logs.
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
>> check the logs.
>>
>> If anybody has access the Nexus node, could you please help to check what
>> is the failure message?
>>
>> Thanks,
>> Wangda
>>
>>
>> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
>>
>>> Good to know. The pool has occasionally had sync issues, but we're
>>> talking 3 times in the last 8-9 years.
>>>
>>> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>>>
>>>> My key was pushed to the server with pgp about 1 year ago, and it worked
>>>> well with the last Ratis release. So it should be synced between the key
>>>> servers.
>>>>
>>>> But it seems that the INFRA solved the problem with shuffling the key
>>>> server order (or it was an intermittent issue): see INFRA-17649
>>>>
>>>> Seems to be working now...
>>>>
>>>> Marton
>>>>
>>>>
>>>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>>>> > HI Brain,
>>>> > Thanks for responding, could u share how to push to keys to Apache
>>>> pgp pool?
>>>> >
>>>> > Best,
>>>> > Wangda
>>>> >
>>>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox 
>>>> wrote:
>>>> >
>>>> >> Did you push your key up to the pgp pool? That's what Nexus is
>>>> validating
>>>> >> against. It might take time to propagate if you just pushed it.
>>>> >>
>>>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton 
>>>> wrote:
>>>> >>
>>>> >>> Seems to be an INFRA issue for me:
>>>> >>>
>>>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>>>> >>> repository and it was ok, locally I verified it.
>>>> >>>
>>>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I
>>>> got
>>>> >>> the same problem even if it worked at last year during the 0.3.0
>>>> >>> release. (I used exactly the same command)
>>>> >>>
>>>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>>>> >>> suggested in the error message:
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>>>> >>>
>>>> >>> Marton
>>>> >>>
>>>> >>>
>>>> >>> [1]:
>>>> >>>
>>>> >>>
>>>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>>> >>>
>>>> >>>
>>>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>>>> >>>> Uploaded sample file and signature.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan >>> >>>> &l

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Spent several more hours trying to figure out the issue, still no luck.

I just filed https://issues.sonatype.org/browse/OSSRH-45646, really
appreciate if anybody could add some suggestions.

Thanks,
Wangda

On Tue, Jan 15, 2019 at 9:48 AM Wangda Tan  wrote:

> It seems the problem still exists for me:
>
> Now the error message only contains:
>
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
> check the logs.
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
> check the logs.
>
> If anybody has access the Nexus node, could you please help to check what
> is the failure message?
>
> Thanks,
> Wangda
>
>
> On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:
>
>> Good to know. The pool has occasionally had sync issues, but we're
>> talking 3 times in the last 8-9 years.
>>
>> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>>
>>> My key was pushed to the server with pgp about 1 year ago, and it worked
>>> well with the last Ratis release. So it should be synced between the key
>>> servers.
>>>
>>> But it seems that the INFRA solved the problem with shuffling the key
>>> server order (or it was an intermittent issue): see INFRA-17649
>>>
>>> Seems to be working now...
>>>
>>> Marton
>>>
>>>
>>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>>> > HI Brain,
>>> > Thanks for responding, could u share how to push to keys to Apache pgp
>>> pool?
>>> >
>>> > Best,
>>> > Wangda
>>> >
>>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
>>> >
>>> >> Did you push your key up to the pgp pool? That's what Nexus is
>>> validating
>>> >> against. It might take time to propagate if you just pushed it.
>>> >>
>>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>>> >>
>>> >>> Seems to be an INFRA issue for me:
>>> >>>
>>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>>> >>> repository and it was ok, locally I verified it.
>>> >>>
>>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>>> >>> the same problem even if it worked at last year during the 0.3.0
>>> >>> release. (I used exactly the same command)
>>> >>>
>>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>>> >>> suggested in the error message:
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>>> >>>
>>> >>> Marton
>>> >>>
>>> >>>
>>> >>> [1]:
>>> >>>
>>> >>>
>>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>> >>>
>>> >>>
>>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>>> >>>> Uploaded sample file and signature.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan >> >>>> <mailto:wheele...@gmail.com>> wrote:
>>> >>>>
>>> >>>> Actually, among the hundreds of failed messages, the "No public
>>> key"
>>> >>>> issues still occurred several times:
>>> >>>>
>>> >>>> failureMessage  No public key: Key with id:
>>> (b3fa653d57300d45)
>>> >>>> was not able to be located on http://gpg-keyserver.de/.
>>> Upload
>>> >>>> your public key and try the operation again.
>>> >>>> failureMessage  No public key: Key with id:
>>> (b3fa653d57300d45)
>>> >>>> was not able to be located on
>>> >>>> http://pool.sks-keyservers.net:11371. Upload your public
>>> key
>>> >>> and
>>> >>>> try the operation again.
>>> >>>> failureMessage  No public key: Key with id:
>>> (b3fa653d57300d4

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Wangda Tan]

Manually checked the new repo, looks good, thanks Sunil!  +1

Best,
Wangda

On Tue, Jan 15, 2019 at 2:04 PM Sunil G  wrote:

> Thanks folks for voting.
>
> For the point mentioned from Zoltan, I re-ran the same source and deployed
> to Nexus to avail those missing source.
> https://repository.apache.org/content/repositories/orgapachehadoop-1186/
>
> Please help to cross check the same.
>
> Thanks & Regards
> Sunil
>
> On Tue, Jan 15, 2019 at 10:05 AM Wangda Tan  wrote:
>
>> +1 (Binding).
>>
>> Deployed a local cluster from binary, and ran some sample sanity jobs.
>>
>> Thanks Sunil for driving the release.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
>> wrote:
>>
>>> Thanks Sunil and others who have worked on the making this release
>>> happen!
>>>
>>> +1 (non-binding)
>>>
>>> - Built from source
>>> - Deployed a pseudo-distributed one node cluster
>>> - Ran basic wordcount, sort, pi jobs
>>> - Basic HDFS/WebHDFS commands
>>> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>>>
>>> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
>>> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
>>> have been fixed in trunk.
>>> - {{
>>> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
>>> fails with an OutOfMemoryError exception. I see the same failure on
>>> trunk as well.
>>>
>>>
>>> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>>>
>>>> Thanks Sunil to manage this release.
>>>>
>>>> +1 (non-binding)
>>>>
>>>> 1. built from the source (with clean local maven repo)
>>>> 2. verified signatures + checksum
>>>> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
>>>> k8s resources [1]
>>>> 4. Executed basic HDFS commands
>>>> 5. Executed basic yarn example jobs
>>>>
>>>> Marton
>>>>
>>>> [1]: FTR: resources:
>>>> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
>>>> https://github.com/elek/flekszible
>>>>
>>>>
>>>> On 1/8/19 12:42 PM, Sunil G wrote:
>>>> > Hi folks,
>>>> >
>>>> >
>>>> > Thanks to all of you who helped in this release [1] and for helping
>>>> to vote
>>>> > for RC0. I have created second release candidate (RC1) for Apache
>>>> Hadoop
>>>> > 3.2.0.
>>>> >
>>>> >
>>>> > Artifacts for this RC are available here:
>>>> >
>>>> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>>>> >
>>>> >
>>>> > RC tag in git is release-3.2.0-RC1.
>>>> >
>>>> >
>>>> >
>>>> > The maven artifacts are available via repository.apache.org at
>>>> >
>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>>>> >
>>>> >
>>>> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59
>>>> pm PST.
>>>> >
>>>> >
>>>> >
>>>> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
>>>> > additions
>>>> >
>>>> > are the highlights of this release.
>>>> >
>>>> > 1. Node Attributes Support in YARN
>>>> >
>>>> > 2. Hadoop Submarine project for running Deep Learning workloads on
>>>> YARN
>>>> >
>>>> > 3. Support service upgrade via YARN Service API and CLI
>>>> >
>>>> > 4. HDFS Storage Policy Satisfier
>>>> >
>>>> > 5. Support Windows Azure Storage - Blob file system in Hadoop
>>>> >
>>>> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>>>> >
>>>> > 7. Improvements in Router-based HDFS federation
>>>> >
>>>> >
>>>> >
>>>> > Thanks to Wangda, Vinod, Marton for helping me in preparing the
>>>> release.
>>>> >
>>>> > I have done few testing with my pseudo cluster. My +1 to start.
>>>> >
>>>> >
>>>> >
>>>> > Regards,
>>>> >
>>>> > Sunil
>>>> >
>>>> >
>>>> >
>>>> > [1]
>>>> >
>>>> >
>>>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>>>> >
>>>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>>> (3.2.0)
>>>> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status =
>>>> Resolved
>>>> > ORDER BY fixVersion ASC
>>>> >
>>>>
>>>> -
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>>
>>>>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

It seems the problem still exists for me:

Now the error message only contains:

failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-client-check-invariants/3.1.2/hadoop-client-check-invariants-3.1.2.pom',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-resourceestimator/3.1.2/hadoop-resourceestimator-3.1.2-javadoc.jar',
check the logs.

If anybody has access the Nexus node, could you please help to check what
is the failure message?

Thanks,
Wangda


On Tue, Jan 15, 2019 at 9:56 AM Brian Fox  wrote:

> Good to know. The pool has occasionally had sync issues, but we're talking
> 3 times in the last 8-9 years.
>
> On Tue, Jan 15, 2019 at 10:39 AM Elek, Marton  wrote:
>
>> My key was pushed to the server with pgp about 1 year ago, and it worked
>> well with the last Ratis release. So it should be synced between the key
>> servers.
>>
>> But it seems that the INFRA solved the problem with shuffling the key
>> server order (or it was an intermittent issue): see INFRA-17649
>>
>> Seems to be working now...
>>
>> Marton
>>
>>
>> On 1/15/19 5:19 AM, Wangda Tan wrote:
>> > HI Brain,
>> > Thanks for responding, could u share how to push to keys to Apache pgp
>> pool?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:
>> >
>> >> Did you push your key up to the pgp pool? That's what Nexus is
>> validating
>> >> against. It might take time to propagate if you just pushed it.
>> >>
>> >> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>> >>
>> >>> Seems to be an INFRA issue for me:
>> >>>
>> >>> 1. I downloaded a sample jar file [1] + the signature from the
>> >>> repository and it was ok, locally I verified it.
>> >>>
>> >>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>> >>> the same problem even if it worked at last year during the 0.3.0
>> >>> release. (I used exactly the same command)
>> >>>
>> >>> I opened an infra ticket to check the logs of the Nexus as it was
>> >>> suggested in the error message:
>> >>>
>> >>> https://issues.apache.org/jira/browse/INFRA-17649
>> >>>
>> >>> Marton
>> >>>
>> >>>
>> >>> [1]:
>> >>>
>> >>>
>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>> >>>
>> >>>
>> >>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>> >>>> Uploaded sample file and signature.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan > >>>> <mailto:wheele...@gmail.com>> wrote:
>> >>>>
>> >>>> Actually, among the hundreds of failed messages, the "No public
>> key"
>> >>>> issues still occurred several times:
>> >>>>
>> >>>> failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>> >>>>     was not able to be located on http://gpg-keyserver.de/.
>> Upload
>> >>>> your public key and try the operation again.
>> >>>> failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>> >>>> was not able to be located on
>> >>>> http://pool.sks-keyservers.net:11371. Upload your public key
>> >>> and
>> >>>> try the operation again.
>> >>>> failureMessage  No public key: Key with id:
>> (b3fa653d57300d45)
>> >>>> was not able to be located on http://pgp.mit.edu:11371.
>> Upload
>> >>>> your public key and try the operation again.
>> >>>>
>> >>>> Once the close operation returned, I will upload sample files
>> which
>> >>>> may help troubleshoot the issue.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan > >>>> <mailto:wheele...@gmail.com>> wrote:
>> >>>>
>> >>>> Than

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1

[Wangda Tan]

+1 (Binding).

Deployed a local cluster from binary, and ran some sample sanity jobs.

Thanks Sunil for driving the release.

Best,
Wangda


On Mon, Jan 14, 2019 at 11:26 AM Virajith Jalaparti 
wrote:

> Thanks Sunil and others who have worked on the making this release happen!
>
> +1 (non-binding)
>
> - Built from source
> - Deployed a pseudo-distributed one node cluster
> - Ran basic wordcount, sort, pi jobs
> - Basic HDFS/WebHDFS commands
> - Ran all the ABFS driver tests against an ADLS Gen 2 account in EAST US
>
> Non-blockers (AFAICT): The following tests in ABFS (HADOOP-15407) fail:
> - For ACLs ({{ITestAzureBlobFilesystemAcl}}) -- However, I believe these
> have been fixed in trunk.
> - {{
> ITestAzureBlobFileSystemE2EScale#testWriteHeavyBytesToFileAcrossThreads}}
> fails with an OutOfMemoryError exception. I see the same failure on trunk
> as well.
>
>
> On Mon, Jan 14, 2019 at 6:21 AM Elek, Marton  wrote:
>
>> Thanks Sunil to manage this release.
>>
>> +1 (non-binding)
>>
>> 1. built from the source (with clean local maven repo)
>> 2. verified signatures + checksum
>> 3. deployed 3 node cluster to Google Kubernetes Engine with generated
>> k8s resources [1]
>> 4. Executed basic HDFS commands
>> 5. Executed basic yarn example jobs
>>
>> Marton
>>
>> [1]: FTR: resources:
>> https://github.com/flokkr/k8s/tree/master/examples/hadoop , generator:
>> https://github.com/elek/flekszible
>>
>>
>> On 1/8/19 12:42 PM, Sunil G wrote:
>> > Hi folks,
>> >
>> >
>> > Thanks to all of you who helped in this release [1] and for helping to
>> vote
>> > for RC0. I have created second release candidate (RC1) for Apache Hadoop
>> > 3.2.0.
>> >
>> >
>> > Artifacts for this RC are available here:
>> >
>> > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/
>> >
>> >
>> > RC tag in git is release-3.2.0-RC1.
>> >
>> >
>> >
>> > The maven artifacts are available via repository.apache.org at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1178/
>> >
>> >
>> > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm
>> PST.
>> >
>> >
>> >
>> > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature
>> > additions
>> >
>> > are the highlights of this release.
>> >
>> > 1. Node Attributes Support in YARN
>> >
>> > 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>> >
>> > 3. Support service upgrade via YARN Service API and CLI
>> >
>> > 4. HDFS Storage Policy Satisfier
>> >
>> > 5. Support Windows Azure Storage - Blob file system in Hadoop
>> >
>> > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>> >
>> > 7. Improvements in Router-based HDFS federation
>> >
>> >
>> >
>> > Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>> >
>> > I have done few testing with my pseudo cluster. My +1 to start.
>> >
>> >
>> >
>> > Regards,
>> >
>> > Sunil
>> >
>> >
>> >
>> > [1]
>> >
>> >
>> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>> >
>> > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
>> > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
>> > ORDER BY fixVersion ASC
>> >
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

HI Brain,
Thanks for responding, could u share how to push to keys to Apache pgp pool?

Best,
Wangda

On Mon, Jan 14, 2019 at 10:44 AM Brian Fox  wrote:

> Did you push your key up to the pgp pool? That's what Nexus is validating
> against. It might take time to propagate if you just pushed it.
>
> On Mon, Jan 14, 2019 at 9:59 AM Elek, Marton  wrote:
>
>> Seems to be an INFRA issue for me:
>>
>> 1. I downloaded a sample jar file [1] + the signature from the
>> repository and it was ok, locally I verified it.
>>
>> 2. I tested it with an other Apache project (Ratis) and my key. I got
>> the same problem even if it worked at last year during the 0.3.0
>> release. (I used exactly the same command)
>>
>> I opened an infra ticket to check the logs of the Nexus as it was
>> suggested in the error message:
>>
>> https://issues.apache.org/jira/browse/INFRA-17649
>>
>> Marton
>>
>>
>> [1]:
>>
>> https://repository.apache.org/service/local/repositories/orgapachehadoop-1183/content/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-javadoc.jar
>>
>>
>> On 1/13/19 6:27 AM, Wangda Tan wrote:
>> > Uploaded sample file and signature.
>> >
>> >
>> >
>> > On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan > > <mailto:wheele...@gmail.com>> wrote:
>> >
>> > Actually, among the hundreds of failed messages, the "No public key"
>> > issues still occurred several times:
>> >
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45)
>> > was not able to be located on http://gpg-keyserver.de/. Upload
>> > your public key and try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45)
>> > was not able to be located on
>> > http://pool.sks-keyservers.net:11371. Upload your public key
>> and
>> > try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45)
>> > was not able to be located on http://pgp.mit.edu:11371. Upload
>> > your public key and try the operation again.
>> >
>> > Once the close operation returned, I will upload sample files which
>> > may help troubleshoot the issue.
>> >
>> > Thanks,
>> >
>> > On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan > > <mailto:wheele...@gmail.com>> wrote:
>> >
>> > Thanks David for the quick response!
>> >
>> > I just retried, now the "No public key" issue is gone. However,
>> > the issue:
>> >
>> > failureMessage  Failed to validate the pgp signature of
>> >
>>  
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
>> > check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> >
>>  
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
>> > check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> >
>>  
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
>> > check the logs.
>> >
>> >
>> > Still exists and repeated hundreds of times. Do you know how to
>> > access the logs mentioned by above log?
>> >
>> > Best,
>> > Wangda
>> >
>> > On Sat, Jan 12, 2019 at 8:37 PM David Nalley > > <mailto:da...@gnsa.us>> wrote:
>> >
>> > On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan
>> > mailto:wheele...@gmail.com>> wrote:
>> > >
>> > > Hi Devs,
>> > >
>> > > I'm currently rolling Hadoop 3.1.2 release candidate,
>> > however, I saw an issue when I try to close repo in Nexus.
>> > >
>> > > Logs of
>> https://repository.apache.org/#stagingRepositories
>> > (orgapachehadoop-1183) shows hundreds of lines of the
>> > following error:
>> > >
>> > > failureMessage  No public key: Key with id:
>> > (b

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Uploaded sample file and signature.



On Sat, Jan 12, 2019 at 9:18 PM Wangda Tan  wrote:

> Actually, among the hundreds of failed messages, the "No public key"
> issues still occurred several times:
>
> failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://gpg-keyserver.de/. Upload your public key
> and try the operation again.
> failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pool.sks-keyservers.net:11371. Upload your
> public key and try the operation again.
> failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pgp.mit.edu:11371. Upload your public key
> and try the operation again.
>
> Once the close operation returned, I will upload sample files which may
> help troubleshoot the issue.
>
> Thanks,
>
> On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan  wrote:
>
>> Thanks David for the quick response!
>>
>> I just retried, now the "No public key" issue is gone. However,  the
>> issue:
>>
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
>> check the logs.
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
>> check the logs.
>> failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
>> check the logs.
>>
>>
>> Still exists and repeated hundreds of times. Do you know how to access
>> the logs mentioned by above log?
>>
>> Best,
>> Wangda
>>
>> On Sat, Jan 12, 2019 at 8:37 PM David Nalley  wrote:
>>
>>> On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan  wrote:
>>> >
>>> > Hi Devs,
>>> >
>>> > I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw
>>> an issue when I try to close repo in Nexus.
>>> >
>>> > Logs of https://repository.apache.org/#stagingRepositories
>>> (orgapachehadoop-1183) shows hundreds of lines of the following error:
>>> >
>>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>>> able to be located on http://gpg-keyserver.de/. Upload your public key
>>> and try the operation again.
>>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>>> able to be located on http://pool.sks-keyservers.net:11371. Upload your
>>> public key and try the operation again.
>>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>>> able to be located on http://pgp.mit.edu:11371. Upload your public key
>>> and try the operation again.
>>> > ...
>>> > failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
>>> check the logs.
>>> > failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
>>> check the logs.
>>> > failureMessage  Failed to validate the pgp signature of
>>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
>>> check the logs.
>>> >
>>> >
>>> > This is the same key I used before (and finished two releases), the
>>> same environment I used before.
>>> >
>>> > I have tried more than 10 times in the last two days, no luck. And
>>> closing the repo takes almost one hour (Regular time is less than 1 min)
>>> and always fail at the last.
>>> >
>>> > I used following commands to validate key exists on key servers
>>> >
>>> > gpg --keyserver pgp.mit.edu --recv-keys 57300D45
>>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>>> > gpg: key B3FA653D57300D45: "Wangda tan " not
>>> changed
>>> > gpg: Total number processed: 1
>>> > gpg:  unchanged: 1
>>> >
>>> > gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
>>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>>> > gpg: key B3FA653D57300D45: "Wangda tan " not
>>> changed
>>> > gpg: Total number processed: 1
>>> > gpg:  unchanged: 1
>>> >
>>>
>>> Both of these report that your key was not found.
>>> I took the key from the KEYS file and uploaded it to both of those
>>> servers.
>>>
>>> You might try the release again and see if this resolves the issue.
>>>
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Actually, among the hundreds of failed messages, the "No public key" issues
still occurred several times:

failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://gpg-keyserver.de/. Upload your public key and try
the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pool.sks-keyservers.net:11371. Upload your public
key and try the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pgp.mit.edu:11371. Upload your public key and try
the operation again.

Once the close operation returned, I will upload sample files which may
help troubleshoot the issue.

Thanks,

On Sat, Jan 12, 2019 at 9:04 PM Wangda Tan  wrote:

> Thanks David for the quick response!
>
> I just retried, now the "No public key" issue is gone. However,  the
> issue:
>
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
> check the logs.
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
> check the logs.
> failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
> check the logs.
>
>
> Still exists and repeated hundreds of times. Do you know how to access the
> logs mentioned by above log?
>
> Best,
> Wangda
>
> On Sat, Jan 12, 2019 at 8:37 PM David Nalley  wrote:
>
>> On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan  wrote:
>> >
>> > Hi Devs,
>> >
>> > I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw an
>> issue when I try to close repo in Nexus.
>> >
>> > Logs of https://repository.apache.org/#stagingRepositories
>> (orgapachehadoop-1183) shows hundreds of lines of the following error:
>> >
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>> able to be located on http://gpg-keyserver.de/. Upload your public key
>> and try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>> able to be located on http://pool.sks-keyservers.net:11371. Upload your
>> public key and try the operation again.
>> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
>> able to be located on http://pgp.mit.edu:11371. Upload your public key
>> and try the operation again.
>> > ...
>> > failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
>> check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
>> check the logs.
>> > failureMessage  Failed to validate the pgp signature of
>> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
>> check the logs.
>> >
>> >
>> > This is the same key I used before (and finished two releases), the
>> same environment I used before.
>> >
>> > I have tried more than 10 times in the last two days, no luck. And
>> closing the repo takes almost one hour (Regular time is less than 1 min)
>> and always fail at the last.
>> >
>> > I used following commands to validate key exists on key servers
>> >
>> > gpg --keyserver pgp.mit.edu --recv-keys 57300D45
>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
>> > gpg: Total number processed: 1
>> > gpg:  unchanged: 1
>> >
>> > gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
>> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
>> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
>> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
>> > gpg: Total number processed: 1
>> > gpg:  unchanged: 1
>> >
>>
>> Both of these report that your key was not found.
>> I took the key from the KEYS file and uploaded it to both of those
>> servers.
>>
>> You might try the release again and see if this resolves the issue.
>>
>

Re: [Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Thanks David for the quick response!

I just retried, now the "No public key" issue is gone. However,  the issue:

failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2-test-sources.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.1.2/hadoop-mapreduce-client-jobclient-3.1.2.pom',
check the logs.


Still exists and repeated hundreds of times. Do you know how to access the
logs mentioned by above log?

Best,
Wangda

On Sat, Jan 12, 2019 at 8:37 PM David Nalley  wrote:

> On Sat, Jan 12, 2019 at 9:09 PM Wangda Tan  wrote:
> >
> > Hi Devs,
> >
> > I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw an
> issue when I try to close repo in Nexus.
> >
> > Logs of https://repository.apache.org/#stagingRepositories
> (orgapachehadoop-1183) shows hundreds of lines of the following error:
> >
> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://gpg-keyserver.de/. Upload your public key
> and try the operation again.
> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pool.sks-keyservers.net:11371. Upload your
> public key and try the operation again.
> > failureMessage  No public key: Key with id: (b3fa653d57300d45) was not
> able to be located on http://pgp.mit.edu:11371. Upload your public key
> and try the operation again.
> > ...
> > failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
> check the logs.
> > failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
> check the logs.
> > failureMessage  Failed to validate the pgp signature of
> '/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
> check the logs.
> >
> >
> > This is the same key I used before (and finished two releases), the same
> environment I used before.
> >
> > I have tried more than 10 times in the last two days, no luck. And
> closing the repo takes almost one hour (Regular time is less than 1 min)
> and always fail at the last.
> >
> > I used following commands to validate key exists on key servers
> >
> > gpg --keyserver pgp.mit.edu --recv-keys 57300D45
> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
> > gpg: Total number processed: 1
> > gpg:  unchanged: 1
> >
> > gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
> > gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
> > gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
> > gpg: key B3FA653D57300D45: "Wangda tan " not changed
> > gpg: Total number processed: 1
> > gpg:  unchanged: 1
> >
>
> Both of these report that your key was not found.
> I took the key from the KEYS file and uploaded it to both of those servers.
>
> You might try the release again and see if this resolves the issue.
>

[Urgent] Question about Nexus repo and Hadoop release

[Wangda Tan]

Hi Devs,

I'm currently rolling Hadoop 3.1.2 release candidate, however, I saw an
issue when I try to close repo in Nexus.

Logs of https://repository.apache.org/#stagingRepositories
(orgapachehadoop-1183) shows hundreds of lines of the following error:

failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://gpg-keyserver.de/. Upload your public key and try
the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pool.sks-keyservers.net:11371. Upload your public
key and try the operation again.
failureMessage  No public key: Key with id: (b3fa653d57300d45) was not able
to be located on http://pgp.mit.edu:11371. Upload your public key and try
the operation again.
...
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-tests.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-test-sources.jar',
check the logs.
failureMessage  Failed to validate the pgp signature of
'/org/apache/hadoop/hadoop-yarn-registry/3.1.2/hadoop-yarn-registry-3.1.2-sources.jar',
check the logs.


This is the same key I used before (and finished two releases), the same
environment I used before.

I have tried more than 10 times in the last two days, no luck. And closing
the repo *takes almost one hour* (Regular time is less than 1 min) and
always fail at the last.

I used following commands to validate key exists on key servers

gpg --keyserver pgp.mit.edu --recv-keys 57300D45
gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
gpg: key B3FA653D57300D45: "Wangda tan " not changed
gpg: Total number processed: 1
gpg:  unchanged: 1

gpg --keyserver pool.sks-keyservers.net --recv-keys B3FA653D57300D45
gpg: WARNING: unsafe permissions on homedir '/Users/wtan/.gnupg'
gpg: key B3FA653D57300D45: 1 signature not checked due to a missing key
gpg: key B3FA653D57300D45: "Wangda tan " not changed
gpg: Total number processed: 1
gpg:  unchanged: 1

Did I miss anything? I also checked Nexus errors page, it is filled by logs
(https://repository.apache.org/service/local/feeds/errorWarning):

2019-01-13 02:04:34 WARN  [etcherImpl-task] -
com.sonatype.central.secure.nexus.plugin.internal.AuthtokenFetcherImpl -
Failed to fetch authtoken: org.apache.http.conn.ConnectTimeoutException:
Connect to secure.central.sonatype.com:443 [
secure.central.sonatype.com/207.223.241.90] failed: connect timed out.


Not sure if it is related to error I saw. Also adding Apache Infra user
email list to rule out Nexus issues.

Hope to get your help sooner if possible.

Thanks,
Wangda .

[NOTICE] branch-3.1.2 created for 3.1.2 release

[Wangda Tan]

Hi devs,

I just created branch-3.1.2 for 3.1.2 release and pushed to remote.
branch-3.1 is still open for new commits required for next 3.1.x release
(which will be 3.1.3). In the meantime, branch-3.1.2 is closed for new
commits unless further noticed. I'm working on RC0 of 3.1.2 release now.

Best,
Wangda

Re: Apache Hadoop 3.1.2 release plan

[Wangda Tan]

Hi all,

Since all blockers have done, I plan to start rolling RC for 3.1.2
tomorrow, please let me know if I missed anything.

Thanks,
Wangda

On Wed, Oct 24, 2018 at 7:30 PM Vinod Kumar Vavilapalli 
wrote:

> 231 fixed JIRAs is already quite a bunch!
>
> I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them
> as blocker.
>
> Why not just release now as soon as there are no blockers?
>
> Thanks
> +Vinod
>
> > On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> >
> > Hi, All
> >
> > We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> > improve the quality of the release, I plan to release 3.1.2
> > by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> > and other enhancements. So far there are 231 JIRAs [1] have fix
> > version marked to 3.1.2
> >
> > I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> >
> > Please feel free to share your insights.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
> Map/Reduce")
> > AND fixVersion = 3.1.2
>
>

[jira] [Created] (MAPREDUCE-7172) Wildcard functionality of -libjar is broken when jars are located in same remote FS

[Wangda Tan (JIRA)]

Wangda Tan created MAPREDUCE-7172:
-

 Summary: Wildcard functionality of -libjar is broken when jars are 
located in same remote FS
 Key: MAPREDUCE-7172
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7172
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Wangda Tan


We recently found that when -libjar specified jars on the same remote FS, jars 
will not be properly added to classpath. 

The reason is MAPREDUCE-6719 added the wildcard functionality, but the follow 
logic assumes files are all placed under job's submission directory. (Inside 
JobResourceUploader)
{code:java}
if (useWildcard && !foundFragment) {
  // Add the whole directory to the cache using a wild card
  Path libJarsDirWildcard =
  jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
  DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
}{code}
However, in the same method, specified resources will be only uploaded when two 
FSes are different, see copyRemoteFiles:
{code:java}
if (FileUtil.compareFs(remoteFs, jtFs)) {
  return originalPath;
} {code}
Workaround of this issue is pass:

mapreduce.client.libjars.wildcard = false.

When the MR job got launched. 

Example commandline to reproduce this issue is: 
{code:java}
hadoop jar abc.jar org.ABC -libjars 
"wasb://host/path1/jar1,wasb://host/path2/jar2..."{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: New committer: Botong Huang

[Wangda Tan]

Congrats!

Best,
Wangda

On Wed, Nov 21, 2018 at 4:23 PM Srinivas Reddy 
wrote:

> Congratulations Botong !!!
>
> -
> Srinivas
>
> - Typed on tiny keys. pls ignore typos.{mobile app}
>
> On Thu 22 Nov, 2018, 03:27 Chang Qiang Cao 
> > Congrats Botong!
> >
> > On Wed, Nov 21, 2018 at 2:15 PM Subru Krishnan  wrote:
> >
> > > The Project Management Committee (PMC) for Apache Hadoophas invited
> > > Botong Huang to become a committer and we are pleased to announce that
> > > he has accepted.
> > > Being a committer enables easier contribution to theproject since
> > > there is no need to go via the patchsubmission process. This should
> > > enable better productivity.Being a PMC member enables assistance with
> > > the managementand to guide the direction of the project.
> > >
> > > Congrats and welcome aboard.
> > >
> > > -Subru
> > >
> >
>

Re: Apache Hadoop 3.1.2 release plan

[Wangda Tan]

Just cleaned up all tickets and updated tickets which have target version
== 3.1.2 to 3.1.3.

I will roll an RC next Monday, if you have any tickets need to be released
in 3.1.2, please let me know.

Thanks,
Wangda

On Wed, Oct 24, 2018 at 7:30 PM Vinod Kumar Vavilapalli 
wrote:

> 231 fixed JIRAs is already quite a bunch!
>
> I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them
> as blocker.
>
> Why not just release now as soon as there are no blockers?
>
> Thanks
> +Vinod
>
> > On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> >
> > Hi, All
> >
> > We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> > improve the quality of the release, I plan to release 3.1.2
> > by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> > and other enhancements. So far there are 231 JIRAs [1] have fix
> > version marked to 3.1.2
> >
> > I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> >
> > Please feel free to share your insights.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
> Map/Reduce")
> > AND fixVersion = 3.1.2
>
>

Apache Hadoop 3.1.2 release plan

[Wangda Tan]

Hi, All

We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
improve the quality of the release, I plan to release 3.1.2
by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
and other enhancements. So far there are 231 JIRAs [1] have fix
version marked to 3.1.2

I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
AND fixVersion = 3.1.2

Re: [IMPORTANT] Apache JIRA doesn't properly show sub-tickets from parent

[Wangda Tan]

Just checked the JIRA again, now the issue has gone. I think it might be
caused by some intermittent JIRA system issue.

Thanks,
Wangda

On Mon, Oct 1, 2018 at 10:52 AM Wangda Tan  wrote:

> Hi all devs,
>
> Today I found many subtickets doesn't show properly under parent.
>
> For example, YARN-6875 is the parent of YARN-7072. But from YARN-6875's
> sub ticket, YARN-7072 isn't shown here.
>
> I found many sub-tickets are gone from parent JIRA, such as YARN-6223
> (there were  ~20+ sub tickets), YARN-2492 (There were ~70-80 sub tickets).
>
> Are there any changes recently to Apache JIRA?
>
> Thanks,
> Wangda
>

[IMPORTANT] Apache JIRA doesn't properly show sub-tickets from parent

[Wangda Tan]

Hi all devs,

Today I found many subtickets doesn't show properly under parent.

For example, YARN-6875 is the parent of YARN-7072. But from YARN-6875's sub
ticket, YARN-7072 isn't shown here.

I found many sub-tickets are gone from parent JIRA, such as YARN-6223
(there were  ~20+ sub tickets), YARN-2492 (There were ~70-80 sub tickets).

Are there any changes recently to Apache JIRA?

Thanks,
Wangda

Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409

[Wangda Tan]

+1 for the merge, it gonna be a great addition to 3.2.0 release. Thanks to
everybody for pushing this feature to complete.

Best,
Wangda

On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt 
wrote:

> +1 for merge. Fetaure would be a good addition to 3.2 release.
>
> --
> Bibin A Chundatt
> M: +91-9742095715
> E: bibin.chund...@huawei.com<mailto:bibin.chund...@huawei.com>
> 2012实验室-印研IT BU分部
> 2012 Laboratories-IT BU Branch Dept.
> From:Naganarasimha Garla
> To:common-...@hadoop.apache.org,Hdfs-dev,yarn-...@hadoop.apache.org,
> mapreduce-dev@hadoop.apache.org,
> Date:2018-08-29 20:00:44
> Subject:[Discuss] Merge discussion for Node attribute support feature
> YARN-3409
>
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Wangda Tan]

+1, thanks for working on this, Marton!

Best,
Wangda

On Fri, Aug 31, 2018 at 11:24 AM Arpit Agarwal 
wrote:

> +1
>
> Thanks for initiating this Marton.
>
>
> On 8/31/18, 1:07 AM, "Elek, Marton"  wrote:
>
> Bumping this thread at last time.
>
> I have the following proposal:
>
> 1. I will request a new git repository hadoop-site.git and import the
> new site to there (which has exactly the same content as the existing
> site).
>
> 2. I will ask infra to use the new repository as the source of
> hadoop.apache.org
>
> 3. I will sync manually all of the changes in the next two months back
> to the svn site from the git (release announcements, new committers)
>
> IN CASE OF ANY PROBLEM we can switch back to the svn without any
> problem.
>
> If no-one objects within three days, I'll assume lazy consensus and
> start with this plan. Please comment if you have objections.
>
> Again: it allows immediate fallback at any time as svn repo will be
> kept
> as is (+ I will keep it up-to-date in the next 2 months)
>
> Thanks,
> Marton
>
>
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
> >
> > Thank you very much to bump up this thread.
> >
> >
> > About [2]: (Just for the clarification) the content of the proposed
> > website is exactly the same as the old one.
> >
> > About [1]. I believe that the "mvn site" is perfect for the
> > documentation but for website creation there are more simple and
> > powerful tools.
> >
> > Hugo has more simple compared to jekyll. Just one binary, without
> > dependencies, works everywhere (mac, linux, windows)
> >
> > Hugo has much more powerful compared to "mvn site". Easier to
> create/use
> > more modern layout/theme, and easier to handle the content (for
> example
> > new release announcements could be generated as part of the release
> > process)
> >
> > I think it's very low risk to try out a new approach for the site
> (and
> > easy to rollback in case of problems)
> >
> > Marton
> >
> > ps: I just updated the patch/preview site with the recent releases:
> >
> > ***
> > * http://hadoop.anzix.net *
> > ***
> >
> > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> >> Got pinged about this offline.
> >>
> >> Thanks for keeping at it, Marton!
> >>
> >> I think there are two road-blocks here
> >>   (1) Is the mechanism using which the website is built good enough
> -
> >> mvn-site / hugo etc?
> >>   (2) Is the new website good enough?
> >>
> >> For (1), I just think we need more committer attention and get
> >> feedback rapidly and get it in.
> >>
> >> For (2), how about we do it in a different way in the interest of
> >> progress?
> >>   - We create a hadoop.apache.org/new-site/ where this new site
> goes.
> >>   - We then modify the existing web-site to say that there is a new
> >> site/experience that folks can click on a link and navigate to
> >>   - As this new website matures and gets feedback & fixes, we
> finally
> >> pull the plug at a later point of time when we think we are good to
> go.
> >>
> >> Thoughts?
> >>
> >> +Vinod
> >>
> >>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I would like to bump this thread up.
> >>>
> >>> TLDR; There is a proposed version of a new hadoop site which is
> >>> available from here: https://elek.github.io/hadoop-site-proposal/
> and
> >>> https://issues.apache.org/jira/browse/HADOOP-14163
> >>>
> >>> Please let me know what you think about it.
> >>>
> >>>
> >>> Longer version:
> >>>
> >>> This thread started long time ago to use a more modern hadoop site:
> >>>
> >>> Goals were:
> >>>
> >>> 1. To make it easier to manage it (the release entries could be
> >>> created by a script as part of the release process)
> >>> 2. To use a better look-and-feel
> >>> 3. Move it out from svn to git
> >>>
> >>> I proposed to:
> >>>
> >>> 1. Move the existing site to git and generate it with hugo (which
> is
> >>> a single, standalone binary)
> >>> 2. Move both the rendered and source branches to git.
> >>> 3. (Create a jenkins job to generate the site automatically)
> >>>
> >>> NOTE: this is just about forrest based hadoop.apache.org, NOT
> about
> >>> the documentation which is generated by mvn-site (as before)
> >>>
> >>>
> >>> I got multiple valuable feedback and I improved the proposed site
> >>> according to the comments. Allen had some concerns about the used
> >>> technologies (hugo vs. mvn-site) and I answered all the questions
> why
> >>> I think mvn-site is the best for documentation and hugo is best
> for
> >>> 

[ANNOUNCE] Apache Hadoop 3.1.1 release

[Wangda Tan]

It gives me great pleasure to announce that the Apache Hadoop community has
voted to release Apache Hadoop 3.1.1.

Hadoop 3.1.1 is the first stable maintenance release for the year 2018 in
the Hadoop-3.1 line and brings a number of enhancements.

IMPORTANT NOTES

3.1.1 is the first stable release of 3.1 line which is production-ready.

The Hadoop community fixed 435 JIRAs [1] in total as part of the 3.1.1
release. Of these fixes:

   -

   60 in Hadoop Common
   -

   139 in HDFS
   -

   223 in YARN
   -

   13 in MapReduce

--

Apache Hadoop 3.1.1 contains a number of significant features and
enhancements. A few of them are noted below.


   -

   ENTRY_POINT support for Docker containers.
   -

   Restart policy support for YARN native services.
   -

   Capacity Scheduler: Intra-queue preemption for fairness ordering policy.
   -

   Stabilization works for schedulers, YARN service, docker support, etc.


Please see the Hadoop 3.1.1 CHANGES
<http://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/release/3.1.1/CHANGES.3.1.1.html>
for the detailed list of issues resolved. The release news is posted on the
Apache Hadoop website too, you can go to the downloads section.
<http://hadoop.apache.org/releases.html#Download>

--

Many thanks to everyone who contributed to the release, and everyone in the
Apache Hadoop community! The release is a result of direct and indirect
efforts from many contributors, listed below are the those who contributed
directly by submitting patches and reporting issues.

Abhishek Modi, Ajay Kumar, Akhil PB, Akira Ajisaka, Allen Wittenauer,
Anbang Hu, Andrew Wang, Arpit Agarwal, Atul Sikaria, BELUGA BEHR, Bharat
Viswanadham, Bibin A Chundatt, Billie Rinaldi, Bilwa S T, Botong Huang,
Brahma Reddy Battula, Brook Zhou, CR Hota, Chandni Singh, Chao Sun, Charan
Hebri, Chen Liang, Chetna Chaudhari, Chun Chen, Daniel Templeton, Davide
 Vergari, Dennis Huo, Dibyendu Karmakar, Ekanth Sethuramalingam, Eric
Badger, Eric Yang, Erik Krogen, Esfandiar Manii, Ewan Higgs, Gabor Bota,
Gang Li, Gang Xie, Genmao Yu, Gergely Novák, Gergo Repas, Giovanni Matteo
Fumarola, Gour Saha, Greg Senia, Haibo Yan, Hanisha Koneru, Hsin-Liang
Huang, Hu Ziqian, Istvan Fajth, Jack Bearden, Jason Lowe, Jeff Zhang, Jian
He, Jianchao Jia, Jiandan Yang , Jim Brennan, Jinglun, John Zhuge, Joseph
Fourny, K G Bakthavachalam, Karthik Palanisamy, Kihwal Lee, Kitti Nanasi,
Konstantin Shvachko, Lei (Eddy) Xu, LiXin Ge, Lokesh Jain, Lukas Majercak,
Miklos Szegedi, Mukul Kumar Singh, Namit Maheshwari, Nanda kumar, Nilotpal
Nandi, Pavel Avgustinov, Prabhu Joseph, Prasanth Jayachandran, Robert
Kanter, Rohith Sharma K S, Rushabh S Shah, Sailesh Patel, Sammi Chen, Sean
Mackrory, Sergey Shelukhin, Shane Kumpf, Shashikant Banerjee, Siyao Meng,
Sreenath Somarajapuram, Steve Loughran, Suma Shivaprasad, Sumana Sathish,
Sunil Govindan, Surendra Singh Lilhore, Szilard Nemeth, Takanobu Asanuma,
Tao Jie, Tao Yang, Ted Yu, Thomas Graves, Thomas Marquardt, Todd Lipcon,
Vinod Kumar Vavilapalli, Wangda Tan, Wei Yan, Wei-Chiu Chuang, Weiwei Yang,
Wilfred Spiegelenburg, Xiao Chen, Xiao Liang, Xintong Song, Xuan Gong, Yang
Wang, Yesha Vora, Yiqun Lin, Yiran Wu, Yongjun Zhang, Yuanbo Liu, Zian
Chen, Zoltan Haindrich, Zsolt Venczel, Zuoming Zhang, fang zhenyi, john
lilley, jwhitter, kyungwan nam, liaoyuxiangqin, liuhongtong, lujie, skrho,
yanghuafeng, yimeng, Íñigo Goiri.

Wangda Tan

[1] JIRA query: project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
Fixed AND fixVersion = 3.1.1 ORDER BY key ASC, updated ASC, created DESC,
priority DESC

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Wangda Tan]

Includes these additional votings we got 5 binding votings, and 12
non-binding votings, we have many others like Bibin/Akhil gave offline
suggestions and testing results. Thanks everyone for your help!

I've done most of staging/pushing works for the 3.1.1 release. Given Apache
mirrors will take some time to finish syncing tasks, I plan to send out the
release announcement by Thu noon PDT time. If there anything wanna to
highlight for the announcement of 3.1.1, please let me know by Thu 10 AM
PDT.

Thanks again for your help of the release!

Best,
Wangda


On Tue, Aug 7, 2018 at 7:57 PM Chandni Singh 
wrote:

> Thanks Wangda!
>
> +1 (non-binding)
>
> Tested the following:
> - Built from source and ran a single node cluster
> - Ran the example pi job
> - Launched yarn service sleep example
> - Verified upgrade of yarn service sleep
>
> Thanks,
> Chandni
>
>
> On Tue, Aug 7, 2018 at 7:02 PM Suma Shivaprasad <
> sumasai.shivapra...@gmail.com> wrote:
>
>> Thanks Wangda!
>>
>> +1 (non-binding)
>>
>> Tested the following:
>>   - Built from source
>>   - Setup single node cluster
>>   - Tested Dynamic queues
>>   - Tested MR and DS with default, docker runtime
>>   - Tested Yarn Services with various restart policies
>>
>> Thanks
>> Suma
>>
>>
>> On Tue, Aug 7, 2018 at 2:45 PM Eric Payne > .invalid>
>> wrote:
>>
>> > Thanks Wangda for creating this release.
>> >
>> > +1 (binding)
>> > Tested:
>> > - Built from source
>> > - Deployed to 6-node, multi-tennant, unsecured pseudo cluster with
>> > hierarchical queue structure (CS)
>> > - Refreshed queue (CS) properties
>> > - Intra-queue preemption (CS)
>> > - inter-queue preemption (CS)
>> > - User weights (CS)
>> >
>> > Issues:
>> > - Inter-queue preemption seems to be preempting unnecessarily (flapping)
>> > when the queue balancing feature is enabled. This does not seem to be
>> > specific to this release.
>> > - The preemption-to-balance-queue-after-satisfied.enabled property seems
>> > to always be enabled, but again, that is not specific to this release.
>> >
>> >
>> > Eric
>> >
>> >
>> > On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan <
>> > wheele...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> >
>> > Hi folks,
>> >
>> > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
>> here:
>> >
>> > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>> >
>> > The RC tag in git is release-3.1.1-RC0:
>> > https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>> >
>> > The maven artifacts are available via repository.apache.org at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>> >
>> > You can find my public key at
>> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>> >
>> > This vote will run 5 days from now.
>> >
>> > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>> >
>> > I have done testing with a pseudo cluster and distributed shell job. My
>> +1
>> > to start.
>> >
>> > Best,
>> > Wangda Tan
>> >
>> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
>> > ORDER BY priority DESC
>> >
>> > -
>> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Wangda Tan]

Thanks everyone for voting!

With 4 binding votes, 10 non-binding votes and no veto. The vote passed,
I'm going to work on staging the release.

Best,
Wangda


On Tue, Aug 7, 2018 at 12:33 PM Nandakumar Vadivelu <
nvadiv...@hortonworks.com> wrote:

> Thanks for all the hard work Wangda!
> +1
>
> * Verified pgp signatures
> * Verified MD5 checksums
> * Verified both source and binary releases
> * Installed pseudo distributed cluster
> * Verified hdfs filesystem commands
> * Verified hdfs admin commands
> * Verified hdfs snapshot commands
> * Basic sanity check on NN UI
>
> Issue found
> * In "Browse the file system" under "Utilities" tab in NN UI, "Head the
> file" and "Tail the file" are not working.
>
> -Nanda
>
> On 8/3/18, 12:14 AM, "Wangda Tan"  wrote:
>
> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available
> here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job.
> My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.1)
> ORDER BY priority DESC
>
>
>

[VOTE] Release Apache Hadoop 3.1.1 - RC0

[Wangda Tan]

Hi folks,

I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.1-RC0/

The RC tag in git is release-3.1.1-RC0:
https://github.com/apache/hadoop/commits/release-3.1.1-RC0

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1139/

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

This vote will run 5 days from now.

3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.

I have done testing with a pseudo cluster and distributed shell job. My +1
to start.

Best,
Wangda Tan

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
ORDER BY priority DESC

Re: Apache Hadoop 3.1.1 release plan

[Wangda Tan]

Hi all,

I'm working on RC for 3.1.1 now, if you have any commits to 3.1.x line,
please commit them to branch-3.1.

If it is a blocker for 3.1.1, please let me know so we can decide if we
want to roll another RC or not.

Thanks,
Wangda


On Thu, Jul 19, 2018 at 2:47 PM Wangda Tan  wrote:

> Hi all,
>
> After several blockers of 3.1.1 landed, I think we're pretty close to a
> clean 3.1.1 branch ready for RC.
>
> By far we have two blockers targeted 3.1.1 [1], and there're 420 tickets
> have fix version = 3.1.1 [2]
>
> As we previously communicated for 3.1.1 release date (May 01), we have
> delayed for more than two months, which I want to get 3.1.1 released as
> soon as possible. I just cut branch-3.1.1 for blockers only. Branch-3.1
> will be open for all bug fixes.
>
> I'm going to create RC0 by end of tomorrow or the last blocker get
> resolved, whichever is the later. Please let me know if there are any other
> blockers need to get in 3.1.1.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.1 ORDER
> BY priority DESC
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>
> Thanks,
> Wangda
>
>
> On Thu, May 10, 2018 at 6:40 PM Weiwei Yang 
> wrote:
>
>> Hi Wangda
>>
>> I would propose to have https://issues.apache.org/jira/browse/YARN-8015 
>> included
>> in 3.1.1.
>>
>> Once this is done, we get both intra and inter placement constraint
>> covered so users could start to explore this feature. Otherwise the
>> functionality is pretty limited. It has been Patch Available for a while, I
>> just promoted it targeting to 3.1.1. Hope that makes sense.
>>
>> Thanks!
>>
>> --
>> Weiwei
>>
>> On 11 May 2018, 9:02 AM +0800, Wangda Tan , wrote:
>>
>> Hi all,
>>
>> As we previously proposed RC time (May 1st), we want to release 3.1.1
>> sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0,
>> and
>> there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
>> posted comments to these open criticals/blockers ticket owners asking
>> about
>> statuses.
>>
>> If everybody agrees, I propose start code freeze of branch-3.1 from Sat
>> PDT
>> time this week, only blockers/criticals can be committed to branch-3.1. To
>> avoid the burden of committers, I want to delay cutting branch-3.1.1 as
>> late as possible. If you have any major/minors (For severe issues please
>> update priorities) tickets want to go to 3.1.1, please reply this email
>> thread and we can look at them and make a call together.
>>
>> Please feel free to share your comments and suggestions.
>>
>> Thanks,
>> Wangda
>>
>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
>> AND status = Resolved AND fixVersion = 3.1.1
>> [2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
>> priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
>> Version/s" = 3.1.1 ORDER BY priority DESC
>>
>>
>> On Thu, May 10, 2018 at 5:48 PM, Wangda Tan  wrote:
>>
>> Thanks Brahma/Sunil,
>>
>> For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
>> target version.
>> For YARN-8236, it is a severe issue and I think it is close to finish.
>>
>>
>>
>> On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:
>>
>>
>> Thanks Brahma.
>> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>>
>> - Sunil
>>
>>
>> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
>> addition to 3.1 line release for improving quality.
>>
>> Looks only following two are pending which are in review state. Hope you
>> are monitoring these two.
>>
>> https://issues.apache.org/jira/browse/YARN-8265
>> https://issues.apache.org/jira/browse/YARN-8236
>>
>>
>>
>> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
>> branch-3.1
>>
>>
>> -Original Message-----
>> From: Wangda Tan [mailto:wheele...@gmail.com]
>> Sent: 19 April 2018 17:49
>> To: Hadoop Common ;
>> mapreduce-dev@hadoop.apache.org; Hdfs-dev ;
>> yarn-...@hadoop.apache.org
>> Subject: Apache Hadoop 3.1.1 release plan
>>
>> Hi, All
>>
>> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
>> quality of the release, we plan to release 3.1.1 at May 06. The focus of
>> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
>> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>>
>> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>>
>> Please feel free to share your insights.
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>> Map/Reduce") AND fixVersion = 3.1.1
>>
>>
>>
>>

Re: Apache Hadoop 3.1.1 release plan

[Wangda Tan]

Hi all,

After several blockers of 3.1.1 landed, I think we're pretty close to a
clean 3.1.1 branch ready for RC.

By far we have two blockers targeted 3.1.1 [1], and there're 420 tickets
have fix version = 3.1.1 [2]

As we previously communicated for 3.1.1 release date (May 01), we have
delayed for more than two months, which I want to get 3.1.1 released as
soon as possible. I just cut branch-3.1.1 for blockers only. Branch-3.1
will be open for all bug fixes.

I'm going to create RC0 by end of tomorrow or the last blocker get
resolved, whichever is the later. Please let me know if there are any other
blockers need to get in 3.1.1.

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.1 ORDER
BY priority DESC
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
ORDER BY priority DESC

Thanks,
Wangda


On Thu, May 10, 2018 at 6:40 PM Weiwei Yang  wrote:

> Hi Wangda
>
> I would propose to have https://issues.apache.org/jira/browse/YARN-8015 
> included
> in 3.1.1.
>
> Once this is done, we get both intra and inter placement constraint
> covered so users could start to explore this feature. Otherwise the
> functionality is pretty limited. It has been Patch Available for a while, I
> just promoted it targeting to 3.1.1. Hope that makes sense.
>
> Thanks!
>
> --
> Weiwei
>
> On 11 May 2018, 9:02 AM +0800, Wangda Tan , wrote:
>
> Hi all,
>
> As we previously proposed RC time (May 1st), we want to release 3.1.1
> sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0, and
> there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
> posted comments to these open criticals/blockers ticket owners asking about
> statuses.
>
> If everybody agrees, I propose start code freeze of branch-3.1 from Sat PDT
> time this week, only blockers/criticals can be committed to branch-3.1. To
> avoid the burden of committers, I want to delay cutting branch-3.1.1 as
> late as possible. If you have any major/minors (For severe issues please
> update priorities) tickets want to go to 3.1.1, please reply this email
> thread and we can look at them and make a call together.
>
> Please feel free to share your comments and suggestions.
>
> Thanks,
> Wangda
>
> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
> AND status = Resolved AND fixVersion = 3.1.1
> [2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
> priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" = 3.1.1 ORDER BY priority DESC
>
>
> On Thu, May 10, 2018 at 5:48 PM, Wangda Tan  wrote:
>
> Thanks Brahma/Sunil,
>
> For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
> target version.
> For YARN-8236, it is a severe issue and I think it is close to finish.
>
>
>
> On Thu, May 10, 2018 at 3:08 AM, Sunil G  wrote:
>
>
> Thanks Brahma.
> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>
> - Sunil
>
>
> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com> wrote:
>
> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
> addition to 3.1 line release for improving quality.
>
> Looks only following two are pending which are in review state. Hope you
> are monitoring these two.
>
> https://issues.apache.org/jira/browse/YARN-8265
> https://issues.apache.org/jira/browse/YARN-8236
>
>
>
> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
> branch-3.1
>
>
> -Original Message-
> From: Wangda Tan [mailto:wheele...@gmail.com]
> Sent: 19 April 2018 17:49
> To: Hadoop Common ;
> mapreduce-dev@hadoop.apache.org; Hdfs-dev ;
> yarn-...@hadoop.apache.org
> Subject: Apache Hadoop 3.1.1 release plan
>
> Hi, All
>
> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
> quality of the release, we plan to release 3.1.1 at May 06. The focus of
> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>
> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>
> Please feel free to share your insights.
>
> Thanks,
> Wangda Tan
>
> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
> Map/Reduce") AND fixVersion = 3.1.1
>
>
>
>

Re: Hadoop 3.2 Release Plan proposal

[Wangda Tan]

Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.

To concerns from Steve,

It is a good idea to keep the door open to get important changes / features
in before cutoff. I would prefer to keep the proposed release date to make
sure things can happen earlier instead of last minute and we all know that
releases are always get delayed :). I'm also fine if we want get another
several weeks time.

Regarding of 3.3 release, I would suggest doing that before thanksgiving.
Do you think is it good or too early / late?

Eric,

The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get merged in
time, we probably not need the YARN-8220.

Sunil,

Could u update https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
with proposed plan as well? We can fill feature list first before getting
consensus of time.

Thanks,
Wangda

On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri 
wrote:

> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran 
> wrote:
>
> >
> >
> > On 16 Jul 2018, at 23:45, Sunil G  > sun...@apache.org>> wrote:
> >
> > I would also would like to take this opportunity to come up with a
> detailed
> > plan.
> >
> > - Feature freeze date : all features should be merged by August 10, 2018.
> >
> >
> >
> > 
>
> >
> > Please let me know if I missed any features targeted to 3.2 per this
> >
> >
> > Well there these big todo lists for S3 & S3Guard.
> >
> > https://issues.apache.org/jira/browse/HADOOP-15226
> > https://issues.apache.org/jira/browse/HADOOP-15220
> >
> >
> > There's a bigger bit of work coming on for Azure Datalake Gen 2
> > https://issues.apache.org/jira/browse/HADOOP-15407
> >
> > I don't think this is quite ready yet, I've been doing work on it, but if
> > we have a 3 week deadline, I'm going to expect some timely reviews on
> > https://issues.apache.org/jira/browse/HADOOP-15546
> >
> > I've uprated that to a blocker feature; will review the S3 & S3Guard
> JIRAs
> > to see which of those are blocking. Then there are some pressing "guave,
> > java 9 prep"
> >
> >
>  I can help with this part if you like.
>
>
>
> >
> >
> >
> > timeline. I would like to volunteer myself as release manager of 3.2.0
> > release.
> >
> >
> > well volunteered!
> >
> >
> >
> Yes, thank you for stepping up.
>
>
> >
> > I think this raises a good q: what timetable should we have for the 3.2.
> &
> > 3.3 releases; if we do want a faster cadence, then having the outline
> time
> > from the 3.2 to the 3.3 release means that there's less concern about
> > things not making the 3.2 dealine
> >
> > -Steve
> >
> >
> Good idea to mitigate the short deadline.
>
> -AF
>

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Wangda Tan]

+1

On Thu, Jul 5, 2018 at 2:37 PM Subru Krishnan  wrote:

> Folks,
>
> There was a merge commit accidentally pushed to trunk, you can find the
> details in the mail thread [1].
>
> I have raised an INFRA ticket [2] to reset/force push to clean up trunk.
>
> Can we have a quick vote for INFRA sign-off to proceed as this is blocking
> all commits?
>
> Thanks,
> Subru
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
> [2] https://issues.apache.org/jira/browse/INFRA-16727
>

Re: Merge branch commit in trunk by mistake

[Wangda Tan]

Adding back hdfs/common/mr-dev again to cc list.

Here's the last merge revert commit:
https://github.com/apache/hadoop/commit/39ad98903a5f042573b97a2e5438bc57af7cc7a1


On Thu, Jul 5, 2018 at 2:17 PM Wangda Tan  wrote:

> It looks like the latest revert is not correct, many of commits get
> reverted.
>
> Dealing with merge commit revert is different from reverting a normal
> commit: https://www.christianengvall.se/undo-pushed-merge-git/
>
> We have to do force reset, now it is a complete mess in trunk.
>
>
>
> On Thu, Jul 5, 2018 at 2:10 PM Vinod Kumar Vavilapalli 
> wrote:
>
>> What is broken due to this merge commit?
>>
>> +Vinod
>>
>> > On Jul 5, 2018, at 2:03 PM, Arun Suresh  wrote:
>> >
>> > I agree with Sean, to be honest.. it is disruptive.
>> > Also, we have to kind of lock down the repo till it is completed..
>> >
>> > I recommend we be careful and try not to get into this situation again..
>> >
>> > -1 on force pushing..
>> >
>> > Cheers
>> > -Arun
>> >
>> > On Thu, Jul 5, 2018, 1:55 PM Sean Busbey  wrote:
>> >
>> >> If we need a vote, please have a thread with either DISCUSS or
>> >> preferably VOTE in the subject so folks are more likely to see it.
>> >>
>> >> that said, I'm -1 (non-binding). force pushes are extremely
>> >> disruptive. there's no way to know who's updated their local git repo
>> >> to include these changes in the last few hours. if a merge commit is
>> >> so disruptive that we need to subject folks to the inconvenience of a
>> >> force push then we should have more tooling in place to avoid them
>> >> (like client side git hooks for all committers).
>> >>
>> >> On Thu, Jul 5, 2018 at 3:36 PM, Wangda Tan 
>> wrote:
>> >>> +1 for force reset the branch.
>> >>>
>> >>> On Thu, Jul 5, 2018 at 12:14 PM Subru Krishnan 
>> wrote:
>> >>>
>> >>>> Looking at the merge commit, I feel it's better to reset/force push
>> >>>> especially since this is still the latest commit on trunk.
>> >>>>
>> >>>> I have raised an INFRA ticket requesting the same:
>> >>>> https://issues.apache.org/jira/browse/INFRA-16727
>> >>>>
>> >>>> -S
>> >>>>
>> >>>> On Thu, Jul 5, 2018 at 11:45 AM, Sean Busbey
>> >> 
>> >>>> wrote:
>> >>>>
>> >>>>> FYI, no images make it through ASF mailing lists. I presume the
>> image
>> >> was
>> >>>>> of the git history? If that's correct, here's what that looks like
>> in
>> >> a
>> >>>>> paste:
>> >>>>>
>> >>>>> https://paste.apache.org/eRix
>> >>>>>
>> >>>>> There are no force pushes on trunk, so backing the change out would
>> >>>> require
>> >>>>> the PMC asking INFRA to unblock force pushes for a period of time.
>> >>>>>
>> >>>>> Probably the merge commit isn't a big enough deal to do that. There
>> >> was a
>> >>>>> merge commit ~5 months ago for when YARN-6592 merged into trunk.
>> >>>>>
>> >>>>> So I'd say just try to avoid doing it in the future?
>> >>>>>
>> >>>>> -busbey
>> >>>>>
>> >>>>> On Thu, Jul 5, 2018 at 1:31 PM, Giovanni Matteo Fumarola <
>> >>>>> giovanni.fumar...@gmail.com> wrote:
>> >>>>>
>> >>>>>> Hi folks,
>> >>>>>>
>> >>>>>> After I pushed something on trunk a merge commit showed up in the
>> >>>>> history. *My
>> >>>>>> bad*.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Since it was one of my first patches, I run a few tests on my
>> >> machine
>> >>>>>> before checked in.
>> >>>>>> While I was running all the tests, someone else checked in. I
>> >> correctly
>> >>>>>> pulled all the new changes.
>> >>>>>>
>> >>>>>> Even before I did the "git push" there was no merge commit in my
>> >>>> history.
>> >>>>>>
>> >>>>>> Can someone help me reverting this change?
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>> Giovanni
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> busbey
>> >>>>>
>> >>>>
>> >>
>> >>
>> >>
>> >> --
>> >> busbey
>> >>
>>
>>

Re: Merge branch commit in trunk by mistake

[Wangda Tan]

+ hdfs-dev/common-dev/mapreduce-dev

On Thu, Jul 5, 2018 at 2:09 PM Sunil G  wrote:

> I just see that this is reverted.
>
> commit 39ad98903a5f042573b97a2e5438bc57af7cc7a1 (origin/trunk, origin/HEAD)
> Author: Anu Engineer 
> Date:   Thu Jul 5 12:22:18 2018 -0700
>
> Revert "Merge branch 'trunk' of
> https://git-wip-us.apache.org/repos/asf/hadoop into trunk"
>
> This reverts commit c163d1797ade0f47d35b4a44381b8ef1dfec5b60, reversing
> changes made to 0d9804dcef2eab5ebf84667d9ca49bb035d9a731.
>
> commit c163d1797ade0f47d35b4a44381b8ef1dfec5b60
> Merge: 0d9804dcef2 99febe7fd50
> Author: Giovanni Matteo Fumarola 
> Date:   Thu Jul 5 10:55:05 2018 -0700
>
> Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/hadoop
> into trunk
>
>
> - Sunil
>
> On Thu, Jul 5, 2018 at 2:04 PM Arun Suresh  wrote:
>
> > I agree with Sean, to be honest.. it is disruptive.
> > Also, we have to kind of lock down the repo till it is completed..
> >
> > I recommend we be careful and try not to get into this situation again..
> >
> > -1 on force pushing..
> >
> > Cheers
> > -Arun
> >
> > On Thu, Jul 5, 2018, 1:55 PM Sean Busbey  wrote:
> >
> > > If we need a vote, please have a thread with either DISCUSS or
> > > preferably VOTE in the subject so folks are more likely to see it.
> > >
> > > that said, I'm -1 (non-binding). force pushes are extremely
> > > disruptive. there's no way to know who's updated their local git repo
> > > to include these changes in the last few hours. if a merge commit is
> > > so disruptive that we need to subject folks to the inconvenience of a
> > > force push then we should have more tooling in place to avoid them
> > > (like client side git hooks for all committers).
> > >
> > > On Thu, Jul 5, 2018 at 3:36 PM, Wangda Tan 
> wrote:
> > > > +1 for force reset the branch.
> > > >
> > > > On Thu, Jul 5, 2018 at 12:14 PM Subru Krishnan 
> > wrote:
> > > >
> > > >> Looking at the merge commit, I feel it's better to reset/force push
> > > >> especially since this is still the latest commit on trunk.
> > > >>
> > > >> I have raised an INFRA ticket requesting the same:
> > > >> https://issues.apache.org/jira/browse/INFRA-16727
> > > >>
> > > >> -S
> > > >>
> > > >> On Thu, Jul 5, 2018 at 11:45 AM, Sean Busbey
> > > 
> > > >> wrote:
> > > >>
> > > >> > FYI, no images make it through ASF mailing lists. I presume the
> > image
> > > was
> > > >> > of the git history? If that's correct, here's what that looks like
> > in
> > > a
> > > >> > paste:
> > > >> >
> > > >> > https://paste.apache.org/eRix
> > > >> >
> > > >> > There are no force pushes on trunk, so backing the change out
> would
> > > >> require
> > > >> > the PMC asking INFRA to unblock force pushes for a period of time.
> > > >> >
> > > >> > Probably the merge commit isn't a big enough deal to do that.
> There
> > > was a
> > > >> > merge commit ~5 months ago for when YARN-6592 merged into trunk.
> > > >> >
> > > >> > So I'd say just try to avoid doing it in the future?
> > > >> >
> > > >> > -busbey
> > > >> >
> > > >> > On Thu, Jul 5, 2018 at 1:31 PM, Giovanni Matteo Fumarola <
> > > >> > giovanni.fumar...@gmail.com> wrote:
> > > >> >
> > > >> > > Hi folks,
> > > >> > >
> > > >> > > After I pushed something on trunk a merge commit showed up in
> the
> > > >> > history. *My
> > > >> > > bad*.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Since it was one of my first patches, I run a few tests on my
> > > machine
> > > >> > > before checked in.
> > > >> > > While I was running all the tests, someone else checked in. I
> > > correctly
> > > >> > > pulled all the new changes.
> > > >> > >
> > > >> > > Even before I did the "git push" there was no merge commit in my
> > > >> history.
> > > >> > >
> > > >> > > Can someone help me reverting this change?
> > > >> > >
> > > >> > > Thanks
> > > >> > > Giovanni
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > busbey
> > > >> >
> > > >>
> > >
> > >
> > >
> > > --
> > > busbey
> > >
> >
>

[jira] [Created] (MAPREDUCE-7101) Revisit behavior of JHS scan file behavior

[Wangda Tan (JIRA)]

Wangda Tan created MAPREDUCE-7101:
-

 Summary: Revisit behavior of JHS scan file behavior
 Key: MAPREDUCE-7101
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7101
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Wangda Tan


Currently, the JHS scan directory if the modification of *directory* changed: 

{code} 
public synchronized void scanIfNeeded(FileStatus fs) {
  long newModTime = fs.getModificationTime();
  if (modTime != newModTime) {
<... omitted some logics ...>
// reset scanTime before scanning happens
scanTime = System.currentTimeMillis();
Path p = fs.getPath();
try {
  scanIntermediateDirectory(p);
{code}

This logic relies on an assumption that, the directory's modification time will 
be updated if a file got placed under the directory.

However, the semantic of directory's modification time is not consistent in 
different FS implementations. For example, MAPREDUCE-6680 fixed some issues of 
truncated modification time. And HADOOP-12837 mentioned on S3, the directory's 
modification time is always 0.

I think we need to revisit behavior of this logic to make it to more robustly 
work on different file systems.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

[Wangda Tan]

+1 (binding)

- Build from source.
- Ran sanity jobs successfully.

Thanks,
Wangda

On Mon, May 14, 2018 at 5:44 AM, Sunil G  wrote:

> +1 (binding)
>
> 1. Build package from src
> 2. Ran few MR jobs and verified checked App Priority cases
> 3. Node Label basic functions are ok.
>
> Thanks
> Sunil
>
>
> On Tue, May 8, 2018 at 11:11 PM 俊平堵  wrote:
>
> > Hi all,
> >  I've created the first release candidate (RC0) for Apache Hadoop
> > 2.8.4. This is our next maint release to follow up 2.8.3. It includes 77
> > important fixes and improvements.
> >
> > The RC artifacts are available at:
> > http://home.apache.org/~junping_du/hadoop-2.8.4-RC0
> >
> > The RC tag in git is: release-2.8.4-RC0
> >
> > The maven artifacts are available via repository.apache.org<
> > http://repository.apache.org> at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1118
> >
> > Please try the release and vote; the vote will run for the usual 5
> > working days, ending on 5/14/2018 PST time.
> >
> > Thanks,
> >
> > Junping
> >
>

Re: Apache Hadoop 3.1.1 release plan

[Wangda Tan]

Hi all,

As we previously proposed RC time (May 1st), we want to release 3.1.1
sooner if possible. As of now, 3.1.1 has 187 fixes [1] on top of 3.1.0, and
there're 10 open blockers/criticals which target to 3.1.1 [2]. I just
posted comments to these open criticals/blockers ticket owners asking about
statuses.

If everybody agrees, I propose start code freeze of branch-3.1 from Sat PDT
time this week, only blockers/criticals can be committed to branch-3.1. To
avoid the burden of committers, I want to delay cutting branch-3.1.1 as
late as possible. If you have any major/minors (For severe issues please
update priorities) tickets want to go to 3.1.1, please reply this email
thread and we can look at them and make a call together.

Please feel free to share your comments and suggestions.

Thanks,
Wangda

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
AND status = Resolved AND  fixVersion = 3.1.1
[2] project in (YARN, HADOOP, MAPREDUCE, "Hadoop Development Tools") AND
priority in (Blocker, Critical) AND resolution = Unresolved AND "Target
Version/s" = 3.1.1 ORDER BY priority DESC


On Thu, May 10, 2018 at 5:48 PM, Wangda Tan <wheele...@gmail.com> wrote:

> Thanks Brahma/Sunil,
>
> For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
> target version.
> For YARN-8236, it is a severe issue and I think it is close to finish.
>
>
>
> On Thu, May 10, 2018 at 3:08 AM, Sunil G <sun...@apache.org> wrote:
>
>>
>> Thanks Brahma.
>> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>>
>> - Sunil
>>
>>
>> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
>> brahmareddy.batt...@huawei.com> wrote:
>>
>>> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
>>> addition to 3.1 line release for improving quality.
>>>
>>> Looks only following two are pending which are in review state. Hope you
>>> are monitoring these two.
>>>
>>> https://issues.apache.org/jira/browse/YARN-8265
>>> https://issues.apache.org/jira/browse/YARN-8236
>>>
>>>
>>>
>>> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
>>> branch-3.1
>>>
>>>
>>> -Original Message-
>>> From: Wangda Tan [mailto:wheele...@gmail.com]
>>> Sent: 19 April 2018 17:49
>>> To: Hadoop Common <common-...@hadoop.apache.org>;
>>> mapreduce-dev@hadoop.apache.org; Hdfs-dev <hdfs-...@hadoop.apache.org>;
>>> yarn-...@hadoop.apache.org
>>> Subject: Apache Hadoop 3.1.1 release plan
>>>
>>> Hi, All
>>>
>>> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
>>> quality of the release, we plan to release 3.1.1 at May 06. The focus of
>>> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
>>> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>>>
>>> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>>>
>>> Please feel free to share your insights.
>>>
>>> Thanks,
>>> Wangda Tan
>>>
>>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>>> Map/Reduce") AND fixVersion = 3.1.1
>>>
>>
>

Re: Apache Hadoop 3.1.1 release plan

[Wangda Tan]

Thanks Brahma/Sunil,

For YARN-8265, it is a too big change for 3.1.1, I just removed 3.1.1 from
target version.
For YARN-8236, it is a severe issue and I think it is close to finish.



On Thu, May 10, 2018 at 3:08 AM, Sunil G <sun...@apache.org> wrote:

>
> Thanks Brahma.
> Yes, Billie is reviewing YARN-8265 and I am helping in YARN-8236.
>
> - Sunil
>
>
> On Thu, May 10, 2018 at 2:25 PM Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com> wrote:
>
>> Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better
>> addition to 3.1 line release for improving quality.
>>
>> Looks only following two are pending which are in review state. Hope you
>> are monitoring these two.
>>
>> https://issues.apache.org/jira/browse/YARN-8265
>> https://issues.apache.org/jira/browse/YARN-8236
>>
>>
>>
>> Note : https://issues.apache.org/jira/browse/YARN-8247==> committed
>> branch-3.1
>>
>>
>> -Original Message-
>> From: Wangda Tan [mailto:wheele...@gmail.com]
>> Sent: 19 April 2018 17:49
>> To: Hadoop Common <common-...@hadoop.apache.org>;
>> mapreduce-dev@hadoop.apache.org; Hdfs-dev <hdfs-...@hadoop.apache.org>;
>> yarn-...@hadoop.apache.org
>> Subject: Apache Hadoop 3.1.1 release plan
>>
>> Hi, All
>>
>> We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the
>> quality of the release, we plan to release 3.1.1 at May 06. The focus of
>> 3.1.1 will be fixing blockers / critical bugs and other enhancements. So
>> far there are 100 JIRAs [1] have fix version marked to 3.1.1.
>>
>> We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.
>>
>> Please feel free to share your insights.
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop
>> Map/Reduce") AND fixVersion = 3.1.1
>>
>

Apache Hadoop 3.1.1 release plan

[Wangda Tan]

Hi, All

We have released Apache Hadoop 3.1.0 on Apr 06. To further
improve the quality of the release, we plan to release 3.1.1
at May 06. The focus of 3.1.1 will be fixing blockers / critical bugs
and other enhancements. So far there are 100 JIRAs [1] have fix
version marked to 3.1.1.

We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day.

Please feel free to share your insights.

Thanks,
Wangda Tan

[1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
AND fixVersion = 3.1.1

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks @Konstantin for reporting this issue, I will post comments on the
JIRA (HADOOP-15205)

- Wangda

On Tue, Apr 10, 2018 at 12:08 PM, Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> A note to release managers. As discussed in
> https://issues.apache.org/jira/browse/HADOOP-15205
> We are producing release artifacts without sources jars. See e.g.
> https://repository.apache.org/content/repositories/releases/
> org/apache/hadoop/hadoop-common/3.1.0/
> I believe this has something to do with maven deployment stage,
> potentially maven-source-plugin.
> This is similar for all releases now, and I believe it should be fixed.
>
> Thanks,
> --Konstantin
>
> On Fri, Apr 6, 2018 at 2:01 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
>> Thanks guys for the additional votes! I just sent out announcement email.
>>
>> Best,
>> Wangda
>>
>> On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <junping...@apache.org> wrote:
>>
>> > Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
>> > based on:
>> >
>> > - Verified signatures
>> >
>> > - Verified checksums for source and binary artifacts
>> >
>> > - Built from source
>> >
>> > - Deployed a single node cluster
>> >
>> > - Verified web UIs, include Namenode, RM, etc.
>> >
>> > * Tried shell commands of HDFS and YARN
>> >
>> > * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>> >
>> >
>> > Thanks,
>> >
>> >
>> > Junping
>> >
>> >
>> >
>> > Wangda Tan <wheele...@gmail.com>于2018年3月30日 周五下午12:15写道：
>> >
>> >> Hi folks,
>> >>
>> >> Thanks to the many who helped with this release since Dec 2017 [1].
>> We've
>> >> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>> >>
>> >> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>> >>
>> >> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> >> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>> >>
>> >> The maven artifacts are available via repository.apache.org at
>> >> https://repository.apache.org/content/repositories/orgapache
>> hadoop-1090/
>> >> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>> >>
>> >> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> >> include the first class GPU/FPGA support on YARN, Native services,
>> Support
>> >> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> >> block replicas to be provided by an external storage system, etc.
>> >>
>> >> For 3.1.0 RC0 vote discussion, please see [3].
>> >>
>> >> We’d like to use this as a starting release for 3.1.x [1], depending on
>> >> how
>> >> it goes, get it stabilized and potentially use a 3.1.1 in several
>> weeks as
>> >> the stable release.
>> >>
>> >> We have done testing with a pseudo cluster:
>> >> - Ran distributed job.
>> >> - GPU scheduling/isolation.
>> >> - Placement constraints (intra-application anti-affinity) by using
>> >> distributed shell.
>> >>
>> >> My +1 to start.
>> >>
>> >> Best,
>> >> Wangda/Vinod
>> >>
>> >> [1]
>> >> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> >> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> >> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>> (3.1.0)
>> >> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
>> BY
>> >> fixVersion ASC
>> >> [3]
>> >> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> >> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>> >>
>> >
>>
>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks guys for the additional votes! I just sent out announcement email.

Best,
Wangda

On Fri, Apr 6, 2018 at 2:32 AM, 俊平堵 <junping...@apache.org> wrote:

> Thanks Wangda for the great work! Sorry for my late coming +1 (binding),
> based on:
>
> - Verified signatures
>
> - Verified checksums for source and binary artifacts
>
> - Built from source
>
> - Deployed a single node cluster
>
> - Verified web UIs, include Namenode, RM, etc.
>
> * Tried shell commands of HDFS and YARN
>
> * Ran sample MR jobs, include PI, Sleep, Terasort, etc.
>
>
> Thanks,
>
>
> Junping
>
>
>
> Wangda Tan <wheele...@gmail.com>于2018年3月30日 周五下午12:15写道：
>
>> Hi folks,
>>
>> Thanks to the many who helped with this release since Dec 2017 [1]. We've
>> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>>
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>>
>> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
>> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
>> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>>
>> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
>> include the first class GPU/FPGA support on YARN, Native services, Support
>> rich placement constraints in YARN, S3-related enhancements, allow HDFS
>> block replicas to be provided by an external storage system, etc.
>>
>> For 3.1.0 RC0 vote discussion, please see [3].
>>
>> We’d like to use this as a starting release for 3.1.x [1], depending on
>> how
>> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
>> the stable release.
>>
>> We have done testing with a pseudo cluster:
>> - Ran distributed job.
>> - GPU scheduling/isolation.
>> - Placement constraints (intra-application anti-affinity) by using
>> distributed shell.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda/Vinod
>>
>> [1]
>> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
>> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
>> fixVersion ASC
>> [3]
>> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
>> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.

I will work on the staging and releases.

Best,
Wangda Tan


On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vino...@apache.org
> wrote:

> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> 
> --
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
>-- Start NN, DN, RM, NM, JHS, Timeline Service
>-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
>-- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan <wheele...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com>
> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop Common <common-...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vino...@apache.org>
> Subject: 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

Hi Vinod / Arpit,

I checked following versions:
- 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:

Jars in maven repo [1] are *always* different from jars in the binary
tarball [2]: (I only checked hadoop-yarn-api-version.jar)

(Following numbers are sizes of the jar)
2.6.5:
- Jar in Maven: 1896185
- Jar in tarball: 1891485

2.7.5:
- Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
- Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
(Even size is same, md5 is different)

2.8.3:
- Jar in Maven: 2451433
- Jar in tarball: 2438975

2.9.0:
- Jar in Maven: 2791477
- Jar in tarball: 289

3.0.1:
- Jar in Maven: 2852604
- Jar in tarball: 2851373

I guess the differences come from our release process.

Thanks,
Wangda

[1] Maven jars are downloaded from
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
/hadoop-yarn-api-.jar
[2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/


On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
wrote:

> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
>
>> Thanks Wangda, I see the shaded jars now.
>>
>> Are the repo jars required to be the same as the binary release? They
>> don’t match right now, probably they got rebuilt.
>>
>> +1 (binding), modulo that remaining question.
>>
>> * Verified signatures
>> * Verified checksums for source and binary artefacts
>> * Sanity checked jars on r.a.o.
>> * Built from source
>> * Deployed to 3 node secure cluster with NameNode HA
>> * Verified HDFS web UIs
>> * Tried out HDFS shell commands
>> * Ran sample MapReduce jobs
>>
>> Thanks!
>>
>>
>> --
>> From: Wangda Tan <wheele...@gmail.com>
>> Date: Monday, April 2, 2018 at 9:25 PM
>> To: Arpit Agarwal <aagar...@hortonworks.com>
>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
>> hdfs-...@hadoop.apache.org>, Hadoop Common <common-...@hadoop.apache.org>,
>> "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
>> Vinod Kumar Vavilapalli <vino...@apache.org>
>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>
>> As pointed by Arpit, the previously deployed shared jars are incorrect.
>> Just redeployed jars and staged. @Arpit, could you please check the updated
>> Maven repo? https://repository.apache.org/content/repositories/
>> orgapachehadoop-1092
>>
>> Since the jars inside binary tarballs are correct (
>> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
>> need roll another RC, just update Maven repo should be sufficient.
>>
>> Best,
>> Wangda
>>
>>
>> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
>> wrote:
>> Hi Arpit,
>>
>> Thanks for pointing out this.
>>
>> I just removed all .md5 files from artifacts. I found md5 checksums still
>> exist in .mds files and I didn't remove them from .mds file because it is
>> generated by create-release script and Apache guidance is "should not"
>> instead of "must not". Please let me know if you think they need to be
>> removed as well.
>>
>> - Wangda
>>
>>
>>
>> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal > aagar...@hortonworks.com> wrote:
>> Thanks for putting together this RC, Wangda.
>>
>> The guidance from Apache is to omit MD5s, specifically:
>>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>>
>> https://www.apache.org/dev/release-distribution#sigs-and-sums
>>
>>
>>
>>
>> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com>
>> wrote:
>>
>> Hi Gera,
>>
>> It's my bad, I thought 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Wangda Tan]

HI Arpit,

I think it won't match if we do rebuild. It should be fine as far as
they're signed, correct? I don't see any policy doesn't allow this.

Thanks,
Wangda


On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
wrote:

> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> ----------
> From: Wangda Tan <wheele...@gmail.com>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com>
> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-...@hadoop.apache.org>, Hadoop Common <common-...@hadoop.apache.org>,
> "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
> Vinod Kumar Vavilapalli <vino...@apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/
> repositories/orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"
> instead of "must not". Please let me know if you think they need to be
> removed as well.
>
> - Wangda
>
>
>
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal  aagar...@hortonworks.com> wrote:
> Thanks for putting together this RC, Wangda.
>
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
>
> https://www.apache.org/dev/release-distribution#sigs-and-sums
>
>
>
>
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
>
> Hi Gera,
>
> It's my bad, I thought only src/bin tarball is enough.
>
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
>
> Please let me know if you have any other comments.
>
> Thanks,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com>
> wrote:
>
>
> Thanks, Wangda!
>
> There are many more artifacts in previous votes, e.g., see
> http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
> site tarball is missing.
>
> On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:
>
>
> Thanks Wangda for initiating the release.
>
> I tested this RC built from source file.
>
>
>   - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> UI.
>   - Below feature sanity is done
>  - Application priority
>  - Application timeout
>  - Intra Queue preemption with priority based
>  - DS based affinity tests to verify placement constraints.
>   - Tested basic NodeLabel scenarios.
>  - Added couple of labels to few of nodes and behavior is coming
>  correct.
>  - Verified old UI  and new YARN UI for labels.
>  - Submitted apps to labelled cluster and it works fine.
>  - Also performed few cli commands related to nodelabel.
>   - Test basic HA cases and seems correct.
>   - Tested new YARN UI . All pages are getting loaded correctly.
>
>
> - Sunil
>
> On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com>
> wrote:
>
>
> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1].
> We've
>
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
>