Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree [discussion -> lazy vote]

[Vinod Kumar Vavilapalli]

On Mon, Sep 23, 2019 at 4:02 PM Elek, Marton  wrote:

>  As the overall feedback was positive (in fact many of the answers were
> simple +1 votes) I don't think the thread should be repeated under
> [VOTE] subject. Therefore I call it for a lazy consensus.
>

Let's please not do this in the future.

These are large enough changes that we'd like to get formal votes both for
immediate visibility (VOTE threads are more in your face) as well as for
record-keeping in posterity.

Thanks
+Vinod

Re: VOTE PASSED - Hadoop-3.1.3-RC0 Re: [VOTE] Release Hadoop-3.1.3-RC0

[Vinod Kumar Vavilapalli]

Zhankun et.al,

Did we close down the rest of the release process for 3.1.3?

I don't yet see this on the site, so asking.

If it's not done yet and you are running into issues, please let me know
and I can help with the corresponding release management tasks.

Thanks
+Vinod

On Tue, Sep 24, 2019 at 2:16 PM Zhankun Tang  wrote:

> Hi,
>
> Thank you all who spent time to verify and vote 3.1.3 release!
> I‘m pleased to announce that the vote for 3.1.3 RC0 passed! I'll push the
> release bits and send out an announcement for 3.1.3 soon.
>
> Summary of votes for Hadoop-3.1.3-RC0:
>
> 4 binding +1s, from Rohith Sharma K S, Sunil Govindan, Weiwei Yang, Eric
> Payne
>
> 3 non-binding +1s, from Xun Liu, Zac Zhou, Zhankun Tang
>
> 1 non-binding with +0s from Masatake Iwasaki
>
> and *no -1s*.
>
> BR,
> Zhankun
>
> On Tue, 24 Sep 2019 at 15:40, zac yuan  wrote:
>
> > Looking forward to 3.1.3 release. Thanks Zhankun for your great effort~
> >
> > +1 (non-binding)
> >
> > Zac Zhou
> >
> > Xun Liu  于2019年9月23日周一 上午11:49写道：
> >
> > > +1
> > >
> > > Thanks Zhankun for all of your hard work on this release.
> > >
> > >
> > > > On Sep 20, 2019, at 5:34 AM, epa...@apache.org wrote:
> > > >
> > > >
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks Zhankun for all of your hard work on this release.
> > > >
> > > > I downloaded and built the source and ran it on an insecure
> multi-node
> > > pseudo cluster.
> > > >
> > > > I performed various YARN manual tests, including creating custom
> > > resources, creating queue submission ACLs, and queue refreshes.
> > > >
> > > > One concern is that preemption does not seem to be working when only
> > the
> > > custom resources are over the queue capacity, but I don't think this is
> > > something introduced with this release.
> > > >
> > > > -Eric
> > > >
> > > >
> > > >
> > > > On Thursday, September 12, 2019, 3:04:44 AM CDT, Zhankun Tang <
> > > zt...@apache.org> wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi folks,
> > > >
> > > > Thanks to everyone's help on this release. Special thanks to Rohith,
> > > > Wei-Chiu, Akira, Sunil, Wangda!
> > > >
> > > > I have created a release candidate (RC0) for Apache Hadoop 3.1.3.
> > > >
> > > > The RC release artifacts are available at:
> > > > http://home.apache.org/~ztang/hadoop-3.1.3-RC0/
> > > >
> > > > The maven artifacts are staged at:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1228/
> > > >
> > > > The RC tag in git is here:
> > > > https://github.com/apache/hadoop/tree/release-3.1.3-RC0
> > > >
> > > > And my public key is at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.*
> > > >
> > > > For the testing, I have run several Spark and distributed shell jobs
> in
> > > my
> > > > pseudo cluster.
> > > >
> > > > My +1 (non-binding) to start.
> > > >
> > > > BR,
> > > > Zhankun
> > > >
> > > > On Wed, 4 Sep 2019 at 15:56, zhankun tang 
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Thanks for everyone helping in resolving all the blockers targeting
> > > Hadoop
> > > >> 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers
> > > issues
> > > >> to 3.1.4.
> > > >>
> > > >> I'll cut the branch today and call a release vote soon. Thanks!
> > > >>
> > > >>
> > > >> [1]. https://s.apache.org/5hj5i
> > > >>
> > > >> BR,
> > > >> Zhankun
> > > >>
> > > >>
> > > >> On Wed, 21 Aug 2019 at 12:38, Zhankun Tang 
> wrote:
> > > >>
> > > >>> Hi folks,
> > > >>>
> > > >>> We have Apache Hadoop 3.1.2 released on Feb 2019.
> > > >>>
> > > >>> It's been more than 6 months passed and there're
> > > >>>
> > > >>> 246 fixes[1]. 2 blocker and 4 critical Issues [2]
> > > >>>
> > > >>> (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker)
> > > >>>
> > > >>>
> > > >>> I propose my plan to do a maintenance release of 3.1.3 in the next
> > few
> > > >>> (one or two) weeks.
> > > >>>
> > > >>> Hadoop 3.1.3 release plan:
> > > >>>
> > > >>> Code Freezing Date: *25th August 2019 PDT*
> > > >>>
> > > >>> Release Date: *31th August 2019 PDT*
> > > >>>
> > > >>>
> > > >>> Please feel free to share your insights on this. Thanks!
> > > >>>
> > > >>>
> > > >>> [1] https://s.apache.org/zw8l5
> > > >>>
> > > >>> [2] https://s.apache.org/fjol5
> > > >>>
> > > >>>
> > > >>> BR,
> > > >>>
> > > >>> Zhankun
> > > >>>
> > > >>
> > > >
> > > > -
> > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: Does VOTE necessary to create a child repo?

[Vinod Kumar Vavilapalli]

Moving the thread to the dev lists.

Thanks
+Vinod

> On Sep 23, 2019, at 11:43 PM, Vinayakumar B  wrote:
> 
> Thanks Marton,
> 
> Current created 'hadoop-thirdparty' repo is empty right now.
> Whether to use that repo  for shaded artifact or not will be monitored in
> HADOOP-13363 umbrella jira. Please feel free to join the discussion.
> 
> There is no existing codebase is being moved out of hadoop repo. So I think
> right now we are good to go.
> 
> -Vinay
> 
> On Mon, Sep 23, 2019 at 11:38 PM Marton Elek  wrote:
> 
>> 
>> I am not sure if it's defined when is a vote required.
>> 
>> https://www.apache.org/foundation/voting.html
>> 
>> Personally I think it's a big enough change to send a notification to the
>> dev lists with a 'lazy consensus'  closure
>> 
>> Marton
>> 
>> On 2019/09/23 17:46:37, Vinayakumar B  wrote:
>>> Hi,
>>> 
>>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be more in
>> future)
>>> will be kept as a shaded artifact in a separate repo, which will be
>>> referred as dependency in hadoop modules.  This approach avoids shading
>> of
>>> every submodule during build.
>>> 
>>> So question is does any VOTE required before asking to create a git repo?
>>> 
>>> On selfserve platform https://gitbox.apache.org/setup/newrepo.html
>>> I can access see that, requester should be PMC.
>>> 
>>> Wanted to confirm here first.
>>> 
>>> -Vinay
>>> 
>> 
>> -
>> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: private-h...@hadoop.apache.org
>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.2.1 release

[Vinod Kumar Vavilapalli]

Done: https://twitter.com/hadoop/status/1176787511865008128.

If you have tweetdeck, any of the PMC members can do this.

BTW, it looks we haven't published any releases since Nov 2018. Let's get back 
to doing this going forward!

Thanks
+Vinod

> On Sep 25, 2019, at 2:44 PM, Rohith Sharma K S  
> wrote:
> 
> Updated twitter message:
> 
> ``
> Apache Hadoop 3.2.1 is released: https://s.apache.org/96r4h
> 
> Announcement: https://s.apache.org/jhnpe
> Overview: https://s.apache.org/tht6a
> Changes: https://s.apache.org/pd6of
> Release notes: https://s.apache.org/ta50b
> 
> Thanks to our community of developers, operators, and users.
> 
> 
> -Rohith Sharma K S
> 
> 
> On Wed, 25 Sep 2019 at 14:15, Sunil Govindan  wrote:
> 
>> Here the link of Overview URL is old.
>> We should ideally use https://hadoop.apache.org/release/3.2.1.html
>> 
>> Thanks
>> Sunil
>> 
>> On Wed, Sep 25, 2019 at 2:10 PM Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>> 
>>> Can someone help to post this in twitter account?
>>> 
>>> Apache Hadoop 3.2.1 is released: https://s.apache.org/mzdb6
>>> Overview: https://s.apache.org/tht6a
>>> Changes: https://s.apache.org/pd6of
>>> Release notes: https://s.apache.org/ta50b
>>> 
>>> Thanks to our community of developers, operators, and users.
>>> 
>>> -Rohith Sharma K S
>>> 
>>> On Wed, 25 Sep 2019 at 13:44, Rohith Sharma K S <
>>> rohithsharm...@apache.org> wrote:
>>> 
 Hi all,
 
It gives us great pleasure to announce that the Apache Hadoop
 community has
 voted to release Apache Hadoop 3.2.1.
 
 Apache Hadoop 3.2.1 is the stable release of Apache Hadoop 3.2 line,
 which
 includes 493 fixes since Hadoop 3.2.0 release:
 
 - For major changes included in Hadoop 3.2 line, please refer Hadoop
 3.2.1 main page[1].
 - For more details about fixes in 3.2.1 release, please read
 CHANGELOG[2] and RELEASENOTES[3].
 
 The release news is posted on the Hadoop website too, you can go to the
 downloads section directly[4].
 
 Thank you all for contributing to the Apache Hadoop!
 
 Cheers,
 Rohith Sharma K S
 
 
 [1] https://hadoop.apache.org/docs/r3.2.1/index.html
 [2]
 https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/CHANGELOG.3.2.1.html
 [3]
 https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/RELEASENOTES.3.2.1.html
 [4] https://hadoop.apache.org
 
>>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Vinod Kumar Vavilapalli]

Looks to me that the advantages of this additional step are only incremental 
given that you've already decoupled releases and dependencies.

Do you see a Submarine like split-also-into-a-TLP for Ozone? If not now, 
sometime further down the line? If so, why not do both at the same time? I felt 
the same way with Submarine, but couldn't follow up in time.

Thanks
+Vinod

> On Sep 18, 2019, at 4:04 AM, Wangda Tan  wrote:
> 
> +1 (binding).
> 
> From my experiences of Submarine project, I think moving to a separate repo
> helps.
> 
> - Wangda
> 
> On Tue, Sep 17, 2019 at 11:41 AM Subru Krishnan  wrote:
> 
>> +1 (binding).
>> 
>> IIUC, there will not be an Ozone module in trunk anymore as that was my
>> only concern from the original discussion thread? IMHO, this should be the
>> default approach for new modules.
>> 
>> On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
>> slamendo...@bloomberg.net> wrote:
>> 
>>> +1
>>> 
>>> From: e...@apache.org At: 09/17/19 05:48:32To:
>> hdfs-...@hadoop.apache.org,
>>> mapreduce-dev@hadoop.apache.org,  common-...@hadoop.apache.org,
>>> yarn-...@hadoop.apache.org
>>> Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
>>> source tree
>>> 
>>> 
>>> TLDR; I propose to move Ozone related code out from Hadoop trunk and
>>> store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
>>> 
>>> 
>>> When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
>>> be part of the source tree but with separated release cadence, mainly
>>> because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
>>> 
>>> During the last Ozone releases this dependency is removed to provide
>>> more stable releases. Instead of using the latest trunk/SNAPSHOT build
>>> from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
>>> 
>>> As we have no more strict dependency between Hadoop trunk SNAPSHOT and
>>> Ozone trunk I propose to separate the two code base from each other with
>>> creating a new Hadoop git repository (apache/hadoop-ozone.git):
>>> 
>>> With moving Ozone to a separated git repository:
>>> 
>>>  * It would be easier to contribute and understand the build (as of now
>>> we always need `-f pom.ozone.xml` as a Maven parameter)
>>>  * It would be possible to adjust build process without breaking
>>> Hadoop/Ozone builds.
>>>  * It would be possible to use different Readme/.asf.yaml/github
>>> template for the Hadoop Ozone and core Hadoop. (For example the current
>>> github template [2] has a link to the contribution guideline [3]. Ozone
>>> has an extended version [4] from this guideline with additional
>>> information.)
>>>  * Testing would be more safe as it won't be possible to change core
>>> Hadoop and Hadoop Ozone in the same patch.
>>>  * It would be easier to cut branches for Hadoop releases (based on the
>>> original consensus, Ozone should be removed from all the release
>>> branches after creating relase branches from trunk)
>>> 
>>> 
>>> What do you think?
>>> 
>>> Thanks,
>>> Marton
>>> 
>>> [1]:
>>> 
>>> 
>> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
>>> c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
>>> [2]:
>>> 
>>> 
>> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
>>> [3]:
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
>>> [4]:
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
>>> 
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>> 
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this 
enthusiasm turns into building a real, independent but more importantly a 
sustainable community.

Given that there were two official releases off the Apache Hadoop project, I 
doubt if you'd need to go through the incubator process. Instead you can 
directly propose a new TLP at ASF board. The last few times this happened was 
with ORC, and long before that with Hive, HBase etc. Can somebody who have 
cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and 
so need a committer +1? You can be more gently by formally doing a vote once a 
process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, 
you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can 
exercise their will to join this new project as PMC/Committers respectively 
without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu  于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣  于2019年7月22日周一 上午9:38写道：
>>> 
 Hi
 
 I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
 a major in electrical engineering. Our teaching teams and students use
 bigoop submarine for big data analysis and automation control of
>>> electrical
 equipment.
 
 Many thanks to the hadoop community for providing us with machine
>>> learning
 tools like submarine.
 
 I wish hadoop submarine is getting better and better.
 
 
 ==
 赵欣
 东南大学电气工程学院
 
 -
 
 Zhao XIN
 
 School of Electrical Engineering
 
 ==
 2019-07-18
 
 
 *From:* Xun Liu 
 *Date:* 2019-07-18 09:46
 *To:* xinzhao 
 *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
 project?
 
 
 -- Forwarded message -
 发件人： dashuiguailu...@gmail.com 
 Date: 2019年7月17日周三 下午3:17
 Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
 To: Szilard Nemeth , runlin zhang <
 runlin...@gmail.com>
 Cc: Xun Liu , common-dev <
>>> common-...@hadoop.apache.org>,
 yarn-dev , hdfs-dev <
 hdfs-...@hadoop.apache.org>, mapreduce-dev <
 mapreduce-dev@hadoop.apache.org>, submarine-dev <
 submarine-...@hadoop.apache.org>
 
 
 +1 ，Good idea, we are very much looking forward to it.
 
 --
 dashuiguailu...@gmail.com
 
 
 *From:* Szilard Nemeth 
 *Date:* 2019-07-17 14:55
 *To:* runlin zhang 
 *CC:* Xun Liu ; Hadoop Common
 ; yarn-dev ;
 Hdfs-dev ; mapreduce-dev
 ; submarine-dev
 
 *Subject:* Re: Any thoughts making Submarine a separate Apache project?
 +1, this is a very great idea.
 As Hadoop repository has already grown huge and contains many
>> projects, I
 think in general it's a good idea to separate projects in the early
>>> phase.
 
 
 On Wed, Jul 17, 2019, 08:50 runlin zhang  wrote:
 
> +1 ，That will be great ！
> 
>> 在 2019年7月10日，下午3:34，Xun Liu  写道：
>> 
>> Hi all,
>> 
>> This is Xun Liu contributing to the Submarine 

Re: [DISCUSS] Prefer JUnit5 for new tests

[Vinod Kumar Vavilapalli]

+1 if it is indeed possible to have both.

Thanks
+Vinod

> On Jul 26, 2019, at 1:56 PM, Akira Ajisaka  wrote:
> 
> Hi folks,
> 
> Now we are slowly migrating from JUnit4 to JUnit5.
> https://issues.apache.org/jira/browse/HADOOP-14693
> 
> However, as Steve commented [1], if we are going to migrate the
> existing tests, the backporting cost will become too expensive.
> Therefore, I'd like to recommend using JUnit5 for new tests before
> migrating the existing tests. Using junit-vintage-engine, we can mix
> JUnit4 and JUnit5 APIs in the same module, so writing new tests in
> JUnit5 is relatively easy.
> 
> Any thoughts?
> 
> [1] 
> https://issues.apache.org/jira/browse/HADOOP-16318?focusedCommentId=16890955=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16890955
> 
> -Akira
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

[Vinod Kumar Vavilapalli]

Makes sense, +1.

Thanks
+Vinod

> On Jul 17, 2019, at 11:37 AM, Elek, Marton  wrote:
> 
> Hi,
> 
> Github UI (ui!) helps to merge Pull Requests to the proposed branch.
> There are three different ways to do it [1]:
> 
> 1. Keep all the different commits from the PR branch and create one
> additional merge commit ("Create a merge commit")
> 
> 2. Squash all the commits and commit the change as one patch ("Squash
> and merge")
> 
> 3. Keep all the different commits from the PR branch but rebase, merge
> commit will be missing ("Rebase and merge")
> 
> 
> 
> As only the option 2 is compatible with the existing development
> practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
> consensus vote: If no objections withing 3 days, I will ask INFRA to
> disable the options 1 and 3 to make the process less error prone.
> 
> Please let me know, what do you think,
> 
> Thanks a lot
> Marton
> 
> ps: Personally I prefer to merge from local as it enables to sign the
> commits and do a final build before push. But this is a different story,
> this proposal is only about removing the options which are obviously
> risky...
> 
> ps2: You can always do any kind of merge / commits from CLI, for example
> to merge a feature branch together with keeping the history.
> 
> [1]:
> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Fwd: Incorrect NOTICE files in TLP releases

[Vinod Kumar Vavilapalli]

A bit of an old email, but want to make sure this isn't missed.

Has anyone looked into this concern?

Ref https://issues.apache.org/jira/browse/ROL-2138 
.

Thanks
+Vinod

> Begin forwarded message:
> 
> From: sebb 
> Subject: Incorrect NOTICE files in TLP releases
> Date: June 21, 2019 at 2:34:17 AM GMT+5:30
> To: "bo...@apache.org Board" 
> Reply-To: bo...@apache.org
> 
> To whom it may concern:
> 
> I had occasion to download the source for Roller, and happened to look
> at the NOTICE file.
> It does not conform to ASF policies, so I raised ROL-2138.
> 
> One of the replies asked how to manage different N files for binary
> and source releases, and pointed out that Hadoop and Karaf don't
> appear to have multiple copies of the files.
> 
> So I had a look at Hadoop and Karaf; their NOTICE files are also
> non-standard, and it looks like Kafka does not have a NOTICE file in
> the source bundle.
> 
> I suspect these are not the only projects with non-conformant NOTICE files.
> The LICENSE files are also likely to be incorrect (I have not checked).
> 
> Sebb.

Re: Task Counters

[Vinod Kumar Vavilapalli]

When you say "counters updated by AM", do you mean
 (a) the Job Counters or
 (b) counters written by a custom AM that extends MR AM?

If it's (a), the only way is for the Task to use JobClient API to get those 
counters. For this to work in secure clusters, you will have to get YARN RM 
delegation-tokens and put them into the Job.

HTH

+Vinod

> On May 5, 2019, at 5:16 PM, Kha  wrote:
> 
> Hi all,
> 
> I understand that task counters are updated by tasks, and they can be seen
> by the AM only. The counters information go in one direction (Tasks --> AM)
> 
> My question: Is there any way where I can let tasks see some task counters
> which are updated by the AM (Tasks <--> AM)?
> 
> Thanks,


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Result] [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Vinod Kumar Vavilapalli]

Agree, it isn't productive this way.

I can't seem to find it, but was there a DISCUSS thread for this branch-merge? 
I usually recommend addressing issues on a DISCUSS thread instead of fighting 
things over a VOTE.

+Vinod

> On Dec 13, 2018, at 10:09 AM, Konstantin Shvachko  
> wrote:
> 
> This vote failed due to Daryn Sharp's veto.
> The concern is being addressed by HDFS-13873. I will start a new vote once
> this is committed.
> 
> Note for Daryn. Your non-responsive handling of the veto makes a bad
> precedence and is a bad example of communication on the lists from a
> respected member of this community. Please check your availability for
> followup discussions if you choose to get involved with important decisions.
> 
> On Fri, Dec 7, 2018 at 4:10 PM Konstantin Shvachko 
> wrote:
> 
>> Hi Daryn,
>> 
>> Wanted to backup Chen's earlier response to your concerns about rotating
>> calls in the call queue.
>> Our design
>> 1. targets directly the livelock problem by rejecting calls on the
>> Observer that are not likely to be responded in timely matter: HDFS-13873.
>> 2. The call queue rotation is only done on Observers, and never on the
>> active NN, so it stays free of attacks like you suggest.
>> 
>> If this is a satisfactory mitigation for the problem could you please
>> reconsider your -1, so that people could continue voting on this thread.
>> 
>> Thanks,
>> --Konst
>> 
>> On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:
>> 
>>> -1 pending additional info.  After a cursory scan, I have serious
>>> concerns regarding the design.  This seems like a feature that should have
>>> been purely implemented in hdfs w/o touching the common IPC layer.
>>> 
>>> The biggest issue in the alignment context.  It's purpose appears to be
>>> for allowing handlers to reinsert calls back into the call queue.  That's
>>> completely unacceptable.  A buggy or malicious client can easily cause
>>> livelock in the IPC layer with handlers only looping on calls that never
>>> satisfy the condition.  Why is this not implemented via RetriableExceptions?
>>> 
>>> On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang 
>>> wrote:
>>> 
 Great work guys.
 
 Wonder if we can elaborate what's impact of not having #2 fixed, and why
 #2
 is not needed for the feature to complete?
 2. Need to fix automatic failover with ZKFC. Currently it does not
 doesn't
 know about ObserverNodes trying to convert them to SBNs.
 
 Thanks.
 --Yongjun
 
 
 On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko  
 wrote:
 
> Hi Hadoop developers,
> 
> I would like to propose to merge to trunk the feature branch
 HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale
 read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC
 workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
> 
> The main functionality has been implemented see sub-tasks of
 HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
> 
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new
 feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not
 doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
> 
> I attached a unified patch to the umbrella jira for the review and
 Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec
 12.
> 
> Thanks,
> --Konstantin
> 
 
>>> 
>>> 
>>> --
>>> 
>>> Daryn
>>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Move to gitbox

[Vinod Kumar Vavilapalli]

We need to write up the stages of the process and put a concrete timeline first.

Is it an atomic move? Or will both repos live at the same time? Do we need a 
stop-the-world period and messaging sent out?

When do the git-wip repos get deleted and/or stop getting updates?

Things I can think of that need tracking
 - Wiki pages?
 - Jenkins jobs
 - Email notification setup for the commits?
 - Policies - like no-mainline-branch-deletion support

+Vinod

> On Dec 13, 2018, at 7:57 AM, Elek, Marton  wrote:
> 
> 
> 
> On 12/12/18 12:27 PM, Akira Ajisaka wrote:
>> Thank you for your positive feedback! I'll file a jira to INFRA in this 
>> weekend.
>> 
>>> If I understood well the only bigger task here is to update all the jenkins 
>>> jobs. (I am happy to help/contribute what I can do)
> 
>> Thank you Elek for the information. Do you have the privilege to
>> update the Jenkins jobs?
>> 
> I have, but I am more familiar with the Ozone jenkins jobs. I created a
> jira (HADOOP-16003) to discuss / record the changes / where it can be
> discussed or commented by anybody with more expertise.
> 
> Marton
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.1.2 release plan

[Vinod Kumar Vavilapalli]

231 fixed JIRAs is already quite a bunch!

I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them as 
blocker.

Why not just release now as soon as there are no blockers?

Thanks
+Vinod

> On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> 
> Hi, All
> 
> We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> improve the quality of the release, I plan to release 3.1.2
> by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> and other enhancements. So far there are 231 JIRAs [1] have fix
> version marked to 3.1.2
> 
> I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> 
> Please feel free to share your insights.
> 
> Thanks,
> Wangda Tan
> 
> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
> AND fixVersion = 3.1.2


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop Ozone 0.2.1-alpha release

[Vinod Kumar Vavilapalli]

Congratulations on the release!

Just seeing this, why is this its own website with its own downloads section, 
look & feel etc instead of being part of our larger site? 

+Vinod


> On Oct 1, 2018, at 5:24 PM, Elek, Marton  wrote:
> 
> 
> It gives me great pleasure to announce that the Apache Hadoop community has 
> voted to release Apache Hadoop Ozone 0.2.1-alpha.
> 
> Apache Hadoop Ozone is an object store for Hadoop built using Hadoop 
> Distributed Data Store.
> 
> For more information and to download, please check
> 
> https://hadoop.apache.org/ozone
> 
> Note: This release is alpha quality, it's not recommended to use in 
> production.
> 
> Many thanks to everyone who contributed to the release, and everyone in the 
> Apache Hadoop community! The release is a result of work from many 
> contributors. Thank you for all of them.
> 
> On behalf of the Hadoop community,
> Márton Elek
> 
> 
> ps: Hadoop Ozone and HDDS are released separately from the main Hadoop 
> releases, this release doesn't include new Hadoop Yarn/Mapreduce/Hdfs 
> versions.
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Vinod Kumar Vavilapalli]

Is there no way to host the new site and the old site concurrently? And link 
back & forth?

+Vinod


> On Aug 31, 2018, at 1:07 AM, Elek, Marton  wrote:
> 
> Bumping this thread at last time.
> 
> I have the following proposal:
> 
> 1. I will request a new git repository hadoop-site.git and import the new 
> site to there (which has exactly the same content as the existing site).
> 
> 2. I will ask infra to use the new repository as the source of 
> hadoop.apache.org
> 
> 3. I will sync manually all of the changes in the next two months back to the 
> svn site from the git (release announcements, new committers)
> 
> IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
> 
> If no-one objects within three days, I'll assume lazy consensus and start 
> with this plan. Please comment if you have objections.
> 
> Again: it allows immediate fallback at any time as svn repo will be kept as 
> is (+ I will keep it up-to-date in the next 2 months)
> 
> Thanks,
> Marton
> 
> 
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
>> Thank you very much to bump up this thread.
>> About [2]: (Just for the clarification) the content of the proposed website 
>> is exactly the same as the old one.
>> About [1]. I believe that the "mvn site" is perfect for the documentation 
>> but for website creation there are more simple and powerful tools.
>> Hugo has more simple compared to jekyll. Just one binary, without 
>> dependencies, works everywhere (mac, linux, windows)
>> Hugo has much more powerful compared to "mvn site". Easier to create/use 
>> more modern layout/theme, and easier to handle the content (for example new 
>> release announcements could be generated as part of the release process)
>> I think it's very low risk to try out a new approach for the site (and easy 
>> to rollback in case of problems)
>> Marton
>> ps: I just updated the patch/preview site with the recent releases:
>> ***
>> * http://hadoop.anzix.net *
>> ***
>> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
>>> Got pinged about this offline.
>>> 
>>> Thanks for keeping at it, Marton!
>>> 
>>> I think there are two road-blocks here
>>>   (1) Is the mechanism using which the website is built good enough - 
>>> mvn-site / hugo etc?
>>>   (2) Is the new website good enough?
>>> 
>>> For (1), I just think we need more committer attention and get feedback 
>>> rapidly and get it in.
>>> 
>>> For (2), how about we do it in a different way in the interest of progress?
>>>   - We create a hadoop.apache.org/new-site/ where this new site goes.
>>>   - We then modify the existing web-site to say that there is a new 
>>> site/experience that folks can click on a link and navigate to
>>>   - As this new website matures and gets feedback & fixes, we finally pull 
>>> the plug at a later point of time when we think we are good to go.
>>> 
>>> Thoughts?
>>> 
>>> +Vinod
>>> 
>>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I would like to bump this thread up.
>>>> 
>>>> TLDR; There is a proposed version of a new hadoop site which is available 
>>>> from here: https://elek.github.io/hadoop-site-proposal/ and 
>>>> https://issues.apache.org/jira/browse/HADOOP-14163
>>>> 
>>>> Please let me know what you think about it.
>>>> 
>>>> 
>>>> Longer version:
>>>> 
>>>> This thread started long time ago to use a more modern hadoop site:
>>>> 
>>>> Goals were:
>>>> 
>>>> 1. To make it easier to manage it (the release entries could be created by 
>>>> a script as part of the release process)
>>>> 2. To use a better look-and-feel
>>>> 3. Move it out from svn to git
>>>> 
>>>> I proposed to:
>>>> 
>>>> 1. Move the existing site to git and generate it with hugo (which is a 
>>>> single, standalone binary)
>>>> 2. Move both the rendered and source branches to git.
>>>> 3. (Create a jenkins job to generate the site automatically)
>>>> 
>>>> NOTE: this is just about forrest based hadoop.apache.org, NOT about the 
>>>> documentation which is generated by mvn-site (as before)
>>>> 
>>>> 
>>>> I got multiple valuable feedback and I improved 

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Vinod Kumar Vavilapalli]

Yongjun,

Looks like you didn't add the links to 3.0.3 binary release on the 
http://hadoop.apache.org/releases.html page.

I just did it, FYI: 
https://svn.apache.org/viewvc?view=revision=1837967 


Thanks
+Vinod


> On May 31, 2018, at 10:48 PM, Yongjun Zhang  wrote:
> 
> Greetings all,
> 
> I've created the first release candidate (RC0) for Apache Hadoop
> 3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
> about 249
> important fixes and improvements, among which there are 8 blockers. See
> https://issues.apache.org/jira/issues/?filter=12343997
> 
> The RC artifacts are available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
> 
> The maven artifacts are available via
> https://repository.apache.org/content/repositories/orgapachehadoop-1126
> 
> Please try the release and vote; the vote will run for the usual 5 working
> days, ending on 06/07/2018 PST time. Would really appreciate your
> participation here.
> 
> I bumped into quite some issues along the way, many thanks to quite a few
> people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.
> 
> Thanks,
> 
> --Yongjun

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Vinod Kumar Vavilapalli]

+1

Thanks
+Vinod

> On Jul 6, 2018, at 11:12 AM, Sunil G  wrote:
> 
> I just checked.  YARN-7556 and YARN-7451 can be cherry-picked.
> I cherry-picked in my local and compiled. Things are good.
> 
> I can push this now  which will restore trunk to its original.
> I can do this if there are no objection.
> 
> - Sunil
> 
> On Fri, Jul 6, 2018 at 11:10 AM Arpit Agarwal  <mailto:aagar...@hortonworks.com>>
> wrote:
> 
>> afaict YARN-8435 is still in trunk. YARN-7556 and YARN-7451 are not.
>> 
>> 
>> From: Giovanni Matteo Fumarola 
>> Date: Friday, July 6, 2018 at 10:59 AM
>> To: Vinod Kumar Vavilapalli 
>> Cc: Anu Engineer , Arpit Agarwal <
>> aagar...@hortonworks.com>, "su...@apache.org" , "
>> yarn-...@hadoop.apache.org" , "
>> hdfs-...@hadoop.apache.org" , "
>> common-...@hadoop.apache.org" , "
>> mapreduce-dev@hadoop.apache.org" 
>> Subject: Re: [VOTE] reset/force push to clean up inadvertent merge commit
>> pushed to trunk
>> 
>> Everything seems ok except the 3 commits: YARN-8435, YARN-7556, YARN-7451
>> are not anymore in trunk due to the revert.
>> 
>> Haibo/Robert if you can recommit your patches I will commit mine
>> subsequently to preserve the original order.
>> 
>> (My apology for the mess I did with the merge commit)
>> 
>> On Fri, Jul 6, 2018 at 10:42 AM, Vinod Kumar Vavilapalli <
>> vino...@apache.org <mailto:vino...@apache.org><mailto:vino...@apache.org 
>> <mailto:vino...@apache.org>>> wrote:
>> I will add that the branch also successfully compiles.
>> 
>> Let's just move forward as is, unblock commits and just fix things if
>> anything is broken.
>> 
>> +Vinod
>> 
>>> On Jul 6, 2018, at 10:30 AM, Anu Engineer >> <mailto:aengin...@hortonworks.com>
>> <mailto:aengin...@hortonworks.com <mailto:aengin...@hortonworks.com>>> wrote:
>>> 
>>> Hi All,
>>> 
>>> [ Thanks to Arpit for working offline and verifying that branch is
>> indeed good.]
>>> 
>>> I want to summarize what I know of this issue and also solicit other
>> points of view.
>>> 
>>> We reverted the commit(c163d1797) from the branch, as soon as we noticed
>> it. That is, we have made no other commits after the merge commit.
>>> 
>>> We used the following command to revert
>>> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
>>> 
>>> Giovanni's branch had three commits + merge, The JIRAs he had were
>> YARN-7451, YARN-7556, YARN-8435.
>>> 
>>> The issue seems to be the revert of merge has some diffs. I am not a
>> YARN developer, so the only problem is to look at the revert and see if
>> there were any spurious edits in Giovanni's original commit + merge.
>>> If there are none, we don't need a reset/force push.  But if we find an
>> issue I am more than willing to go the force commit route.
>>> 
>>> The revert takes the trunk back to the point of the first commit from
>> Giovanni which is YARN-8435. His branch was also rewriting the order of
>> commits which we have lost due to the revert.
>>> 
>>> Based on what I know so far, I am -1 on the force push.
>>> 
>>> In other words, I am trying to understand why we need the force push. I
>> have left a similar comment in JIRA (
>> https://issues.apache.org/jira/browse/INFRA-16727 
>> <https://issues.apache.org/jira/browse/INFRA-16727>) too.
>>> 
>>> 
>>> Thanks
>>> Anu
>>> 
>>> 
>>> On 7/6/18, 10:24 AM, "Arpit Agarwal" >> <mailto:aagar...@hortonworks.com>> aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>> wrote:
>>> 
>>>   -1 for the force push. Nothing is broken in trunk. The history looks
>> ugly for two commits and we can live with it.
>>> 
>>>   The revert restored the branch to Giovanni's intent. i.e. only
>> YARN-8435 is applied. Verified there is no delta between hashes 0d9804d and
>> 39ad989 (HEAD).
>>> 
>>>   39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch
>> 't...
>>>   c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of
>> https://git- <https://git-/>...
>>>   99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to
>> veri...
>>>   1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler
>> configurat...
>>>   0d9804d 2018-07-05 gifuma@apa o │

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Vinod Kumar Vavilapalli]

I will add that the branch also successfully compiles.

Let's just move forward as is, unblock commits and just fix things if anything 
is broken.

+Vinod

> On Jul 6, 2018, at 10:30 AM, Anu Engineer  wrote:
> 
> Hi All,
> 
> [ Thanks to Arpit for working offline and verifying that branch is indeed 
> good.]
> 
> I want to summarize what I know of this issue and also solicit other points 
> of view.
> 
> We reverted the commit(c163d1797) from the branch, as soon as we noticed it. 
> That is, we have made no other commits after the merge commit.
> 
> We used the following command to revert 
> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
> 
> Giovanni's branch had three commits + merge, The JIRAs he had were YARN-7451, 
> YARN-7556, YARN-8435.
> 
> The issue seems to be the revert of merge has some diffs. I am not a YARN 
> developer, so the only problem is to look at the revert and see if there were 
> any spurious edits in Giovanni's original commit + merge. 
> If there are none, we don't need a reset/force push.  But if we find an issue 
> I am more than willing to go the force commit route.
> 
> The revert takes the trunk back to the point of the first commit from 
> Giovanni which is YARN-8435. His branch was also rewriting the order of 
> commits which we have lost due to the revert.
> 
> Based on what I know so far, I am -1 on the force push.
> 
> In other words, I am trying to understand why we need the force push. I have 
> left a similar comment in JIRA 
> (https://issues.apache.org/jira/browse/INFRA-16727) too.
> 
> 
> Thanks
> Anu
> 
> 
> On 7/6/18, 10:24 AM, "Arpit Agarwal"  wrote:
> 
>-1 for the force push. Nothing is broken in trunk. The history looks ugly 
> for two commits and we can live with it.
> 
>The revert restored the branch to Giovanni's intent. i.e. only YARN-8435 
> is applied. Verified there is no delta between hashes 0d9804d and 39ad989 
> (HEAD).
> 
>39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch 't...
>c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of https://git-...
>99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to veri...
>1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler configurat...
>0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same cli...
>71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce NodeStateManager...
> 
>Regards,
>Arpit
> 
> 
>On 7/5/18, 2:37 PM, "Subru Krishnan"  wrote:
> 
>Folks,
> 
>There was a merge commit accidentally pushed to trunk, you can find the
>details in the mail thread [1].
> 
>I have raised an INFRA ticket [2] to reset/force push to clean up 
> trunk.
> 
>Can we have a quick vote for INFRA sign-off to proceed as this is 
> blocking
>all commits?
> 
>Thanks,
>Subru
> 
>[1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
>[2] https://issues.apache.org/jira/browse/INFRA-16727
> 
> 
> 
>-
>To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7097) MapReduce JHS should honor yarn.resourcemanager.display.per-user-apps

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created MAPREDUCE-7097:
--

 Summary: MapReduce JHS should honor 
yarn.resourcemanager.display.per-user-apps
 Key: MAPREDUCE-7097
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7097
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli


When this config is on, MR JHS should filter the app list based on 
authenticated user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

That is a great observation. And I missed your previous email about the shaded 
vs unshaded jars already getting fixed.

I guess we are good to go.

--

Looking at the RC. Went through my usual check-list. Here's my summary.

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form 
"hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B 
B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, 
"/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 
9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

+1 binding.

Thanks
+Vinod

> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> Hi Vinod / Arpit,
> 
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> 
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> 
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
> 
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
> 
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
> 
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
> 
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
> 
> I guess the differences come from our release process.
> 
> Thanks,
> Wangda
> 
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
> 
> 
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
> wrote:
> 
>> We vote on the source code. The binaries are convenience artifacts.
>> 
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>> 
>> Thanks
>> +Vinod
>> 
>> 
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>> 
>> HI Arpit,
>> 
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>> 
>> Thanks,
>> Wangda
>> 
>> 
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
>> wrote:
>> 
>>> Thanks Wangda, I see the shaded jars now.
>>> 
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>> 
>>> +1 (binding), modulo that remaining question.
>>> 
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>> 
>>> Thanks!
>>> 
>>> 
>>> ----------
>>> From: Wangda Tan <wheele...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aagar...@hortonworks.com>
>>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-...@hadoop.apache.org>, Hadoop Common <common-...@hadoop.apache.org>,
>>> "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>> 
>>> As point

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as the 
binaries to be consistent and correct. And then (b) Give a couple more days for 
folks who tested on the binaries to reverify - I count one such clear vote as 
of now.

Thanks
+Vinod

> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> HI Arpit,
> 
> I think it won't match if we do rebuild. It should be fine as far as they're 
> signed, correct? I don't see any policy doesn't allow this. 
> 
> Thanks,
> Wangda
> 
> 
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com 
> <mailto:aagar...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
> 
> Are the repo jars required to be the same as the binary release? They don’t 
> match right now, probably they got rebuilt.
> 
> +1 (binding), modulo that remaining question.
> 
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
> 
> Thanks!
> 
> 
> --
> From: Wangda Tan <wheele...@gmail.com <mailto:wheele...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>
> Cc: Gera Shegalov <ger...@gmail.com <mailto:ger...@gmail.com>>, Sunil G 
> <sun...@apache.org <mailto:sun...@apache.org>>, "yarn-...@hadoop.apache.org 
> <mailto:yarn-...@hadoop.apache.org>" <yarn-...@hadoop.apache.org 
> <mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev <hdfs-...@hadoop.apache.org 
> <mailto:hdfs-...@hadoop.apache.org>>, Hadoop Common 
> <common-...@hadoop.apache.org <mailto:common-...@hadoop.apache.org>>, 
> "mapreduce-dev@hadoop.apache.org <mailto:mapreduce-dev@hadoop.apache.org>" 
> <mapreduce-dev@hadoop.apache.org <mailto:mapreduce-dev@hadoop.apache.org>>, 
> Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> 
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
> redeployed jars and staged. @Arpit, could you please check the updated Maven 
> repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
> 
> Since the jars inside binary tarballs are correct 
> (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need 
> roll another RC, just update Maven repo should be sufficient. 
> 
> Best,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> Hi Arpit,
> 
> Thanks for pointing out this.
> 
> I just removed all .md5 files from artifacts. I found md5 checksums still 
> exist in .mds files and I didn't remove them from .mds file because it is 
> generated by create-release script and Apache guidance is "should not" 
> instead of "must not". Please let me know if you think they need to be 
> removed as well. 
> 
> - Wangda
> 
> 
> 
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
> <mailto:aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
> 
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> 
> https://www.apache.org/dev/release-distribution#sigs-and-sums 
> <https://www.apache.org/dev/release-distribution#sigs-and-sums>
> 
>  
> 
> 
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> 
> Hi Gera,
> 
> It's my bad, I thought only src/bin tarball is enough.
> 
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> 
> Please let me know if you have any other comments.
> 
> Thanks,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com 
> <mailto:ger...@gmail.com>> wrote:
> 
> 
> Thanks, Wangda!
> 
> There are many more artifacts in previou

[jira] [Reopened] (MAPREDUCE-6823) FileOutputFormat to support configurable PathOutputCommitter factory

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened MAPREDUCE-6823:


I'm making a guess that this is a dup of HADOOP-13786 like the others I just 
closed as dups.

Reopening and closing this as a dup. [~ste...@apache.org], please revert back 
if this is incorrect.

> FileOutputFormat to support configurable PathOutputCommitter factory
> 
>
> Key: MAPREDUCE-6823
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6823
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 3.0.0-alpha2
> Environment: Targeting S3 as the output of work
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-13786-HADOOP-13345-001.patch, 
> MAPREDUCE-6823-002.patch, MAPREDUCE-6823-002.patch, MAPREDUCE-6823-004.patch
>
>
> In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which 
> can talk direct to the S3A Filesystem for more efficient operations, better 
> failure modes, and, most critically, as part of HADOOP-13345, atomic commit 
> of output. The normal committer relies on directory rename() being atomic for 
> this; for S3 we don't have that luxury.
> To support a custom committer, we need to be able to tell FileOutputFormat 
> (and implicitly, all subclasses which don't have their own custom committer), 
> to use our new {{S3AOutputCommitter}}.
> I propose: 
> # {{FileOutputFormat}} takes a factory to create committers.
> # The factory to take a URI and {{TaskAttemptContext}} and return a committer
> # the default implementation always returns a {{FileOutputCommitter}}
> # A configuration option allows a new factory to be named
> # An {{S3AOutputCommitterFactory}} to return a  {{FileOutputCommitter}} or 
> new {{S3AOutputCommitter}} depending upon the URI of the destination.
> Note that MRv1 already supports configurable committers; this is only the V2 
> API



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6823) FileOutputFormat to support configurable PathOutputCommitter factory

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved MAPREDUCE-6823.

   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> FileOutputFormat to support configurable PathOutputCommitter factory
> 
>
> Key: MAPREDUCE-6823
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6823
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 3.0.0-alpha2
> Environment: Targeting S3 as the output of work
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-13786-HADOOP-13345-001.patch, 
> MAPREDUCE-6823-002.patch, MAPREDUCE-6823-002.patch, MAPREDUCE-6823-004.patch
>
>
> In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which 
> can talk direct to the S3A Filesystem for more efficient operations, better 
> failure modes, and, most critically, as part of HADOOP-13345, atomic commit 
> of output. The normal committer relies on directory rename() being atomic for 
> this; for S3 we don't have that luxury.
> To support a custom committer, we need to be able to tell FileOutputFormat 
> (and implicitly, all subclasses which don't have their own custom committer), 
> to use our new {{S3AOutputCommitter}}.
> I propose: 
> # {{FileOutputFormat}} takes a factory to create committers.
> # The factory to take a URI and {{TaskAttemptContext}} and return a committer
> # the default implementation always returns a {{FileOutputCommitter}}
> # A configuration option allows a new factory to be named
> # An {{S3AOutputCommitterFactory}} to return a  {{FileOutputCommitter}} or 
> new {{S3AOutputCommitter}} depending upon the URI of the destination.
> Note that MRv1 already supports configurable committers; this is only the V2 
> API



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

Thanks for reporting this.

It's straight forward. Done, commit pushed.

Thanks
+Vinod

> On Mar 19, 2018, at 2:54 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote:
> 
> I just pulled the latest from branch-3.1 and noticed that now that it was
> reset to trunk, the version in the pom.xml files is 3.2.0-SNAPSHOT instead
> of 3.1.0-SNAPSHOT. Is somebody going to submit a new commit to change this
> version back to 3.1.0-SNAPSHOT in this branch?
> 
> Thank you,
> Jonathan
> 
> On Mon, Mar 19, 2018 at 11:43 AM Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
> 
>> Thanks Wangda.
>> 
>> 
>> On 3/19/18, 11:38 AM, "Wangda Tan" <wheele...@gmail.com> wrote:
>> 
>>Done JIRA fix version update:
>> 
>>Moved all JIRAs with fixVersion = 3.2.0 to 3.1.0 except following few
>> fixes
>>(which committed after 49c747ab187d0650143205ba57ca19607ec4c6bd)
>> 
>>YARN-8002. Support NOT_SELF and ALL namespace types for allocation
>> tag.
>>(Weiwe
>>i Yang via wangda)
>> 
>>HADOOP-15262. AliyunOSS: move files under a directory in parallel
>> when
>>rename a directory. Contributed by Jinhu Wu.
>> 
>>MAPREDUCE-7066. TestQueue fails on Java9
>> 
>>YARN-8028. Support authorizeUserAccessToQueue in RMWebServices.
>>Contributed by Wangda Tan.
>> 
>>YARN-8040. [UI2] New YARN UI webapp does not respect current
>> pathname
>>for REST api. Contributed by Sunil G.
>> 
>>Thanks,
>>Wangda
>> 
>>On Mon, Mar 19, 2018 at 11:12 AM, Wangda Tan <wheele...@gmail.com>
>> wrote:
>> 
>>> Thanks Akira for the additional vote,
>>> 
>>> With help from Apache Infra Team (Daniel Takamori), we just reset
>>> branch-3.1 to trunk (SHA: 49c747ab187d0650143205ba57ca19607ec4c6bd).
>> Will
>>> update JIRA fix version shortly.
>>> 
>>> - Wangda
>>> 
>>> On Sun, Mar 18, 2018 at 6:10 PM, Akira Ajisaka <
>> ajisa...@oss.nttdata.co.jp
>>>> wrote:
>>> 
>>>> +1 for resetting branch-3.1.
>>>> 
>>>> Thanks,
>>>> Akira
>>>> 
>>>> 
>>>> On 2018/03/18 12:51, Wangda Tan wrote:
>>>> 
>>>>> Thanks for sharing your thoughts.
>>>>> 
>>>>> We have done build and single node cluster deploy / test for the
>> latest
>>>>> trunk code (commit: 49c747ab187d0650143205ba57ca19607ec4c6bd).
>> Since
>>>>> there
>>>>> are no objections, so I will go ahead to do the branch replace.
>>>>> 
>>>>> Since we don't have force push permission to release branches. I
>> just
>>>>> filed
>>>>> https://issues.apache.org/jira/browse/INFRA-16204 to get help from
>>>>> Apache
>>>>> infra team.
>>>>> 
>>>>> Please hold any commits to branch-3.1, will keep this email thread
>>>>> posted.
>>>>> 
>>>>> Best,
>>>>> Wangda
>>>>> 
>>>>> On Wed, Mar 14, 2018 at 3:14 PM, Vinod Kumar Vavilapalli <
>>>>> vino...@apache.org
>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>> 
>>>>> I see one new feature:
>> https://issues.apache.org/jira/browse/YARN-7626:
>>>>>> Allow regular expression matching in container-executor.cfg for
>> devices
>>>>>> and
>>>>>> named docker volumes mount.
>>>>>> 
>>>>>> There are 21 sub-tasks. There are three feature-type JIRAs in
>> those -
>>>>>> https://issues.apache.org/jira/browse/YARN-7972,
>>>>>> https://issues.apache.org/jira/browse/YARN-7891 and
>>>>>> https://issues.apache.org/jira/browse/YARN-5015. These should be
>> okay -
>>>>>> not major disrupting features.
>>>>>> 
>>>>>> Everything else is either a bug-fix or an improvement so we
>> should be
>>>>>> good.
>>>>>> 
>>>>>> From the list, it doesn't look like resetting will destabilize
>> 3.1, +1
>>>>>> for
>>>>>> doing this.
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>> On Mar

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

I see one new feature: https://issues.apache.org/jira/browse/YARN-7626: Allow 
regular expression matching in container-executor.cfg for devices and named 
docker volumes mount.

There are 21 sub-tasks. There are three feature-type JIRAs in those - 
https://issues.apache.org/jira/browse/YARN-7972, 
https://issues.apache.org/jira/browse/YARN-7891 and 
https://issues.apache.org/jira/browse/YARN-5015. These should be okay - not 
major disrupting features.

Everything else is either a bug-fix or an improvement so we should be good.

From the list, it doesn't look like resetting will destabilize 3.1, +1 for 
doing this.

Thanks
+Vinod

> On Mar 14, 2018, at 1:54 PM, Wangda Tan  wrote:
> 
> Hi mapreduce/yarn/common/hdfs-devs, 
> 
> As of now, we have all blockers done for 3.1.0 release [1]. The release is 
> running behind schedule due to a few security-related issues. Because of this 
> and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is already 
> diverging. There're 64 commits in trunk but not in branch-3.1. [2]
> 
> I took a quick scan of them, most of them are good fixes which we should 
> bring to 3.1.0 as well. And this can also reduce differences between 3.2.0 
> and 3.1.0 release for less maintenance burden in the future.
> 
> Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days and cut 
> RC after that.
> 
> Thoughts?
> 
> - Wangda
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, 
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0 ORDER BY 
> priority DESC
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) AND 
> fixVersion not in (3.1.0)


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Yes, JIRAs will be filed, the wiki-page idea from YARN meetup is to record all 
combinations of testing that need to be done and correspondingly capture all 
the testing that someone in the community has already done and record it for 
future perusal.

From what you are saying, I guess we haven't advertised to the public yet on 
rolling upgrades, but in our meetups etc so far, you have been saying that 
rolling upgrades is supported - so I assumed we did put it in our messaging.

The important question is if we are or are not allowed to make potentially 
incompatible changes to fix bugs in the process of supporting 2.x to 3.x 
upgrades whether rolling or not.

+Vinod

> On Dec 13, 2017, at 1:05 PM, Andrew Wang  wrote:
> 
> I'm hoping we can address YARN-7588 and any remaining rolling upgrade issues 
> in 3.0.x maintenance releases. Beyond a wiki page, it would be really great 
> to get JIRAs filed and targeted for tracking as soon as possible.
> 
> Vinod, what do you think we need to do regarding caveating rolling upgrade 
> support? We haven't advertised rolling upgrade support between major releases 
> outside of dev lists and JIRA. As a new major release, our compat guidelines 
> allow us to break compatibility, so I don't think it's expected by users.
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Good stuff Andrew, and thanks everyone!

+Vinod

> On Dec 13, 2017, at 1:05 PM, Andrew Wang  wrote:
> 
> To close this out, the vote passes successfully with 13 binding +1s, 5 
> non-binding +1s, and no -1s. Thanks everyone for voting! I'll work on staging.
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

I was waiting for Daniel to post the minutes from YARN meetup to talk about 
this. Anyways, in that discussion, we identified a bunch of key upgrade related 
scenarios that no-one seems to have validated - atleast from the representation 
in the YARN meetup. I'm going to create a wiki-page listing all these scenarios.

But back to the bug that Junping raised. At this point, we don't have a clear 
path towards running 2.x applications on 3.0.0 clusters. So, our claim of 
rolling-upgrades already working is not accurate.

One of the two options that Junping proposed should be pursued before we close 
the release. I'm in favor of calling out rolling-upgrade support be with-drawn 
or caveated and push for progress instead of blocking the release.

Thanks
+Vinod

> On Dec 12, 2017, at 5:44 PM, Junping Du  wrote:
> 
> Thanks Andrew for pushing new RC for 3.0.0. I was out last week, just get 
> chance to validate new RC now.
> 
> Basically, I found two critical issues with the same rolling upgrade scenario 
> as where HADOOP-15059 get found previously:
> HDFS-12920, we changed value format for some hdfs configurations that old 
> version MR client doesn't understand when fetching these configurations. Some 
> quick workarounds are to add old value (without time unit) in hdfs-site.xml 
> to override new default values but will generate many annoying warnings. I 
> provided my fix suggestions on the JIRA already for more discussion.
> The other one is YARN-7646. After we workaround HDFS-12920, will hit the 
> issue that old version MR AppMaster cannot communicate with new version of 
> YARN RM - could be related to resource profile changes from YARN side but 
> root cause are still in investigation.
> 
> The first issue may not belong to a blocker given we can workaround this 
> without code change. I am not sure if we can workaround 2nd issue so far. If 
> not, we may have to fix this or compromise with withdrawing support of 
> rolling upgrade or calling it a stable release.
> 
> 
> Thanks,
> 
> Junping
> 
> 
> From: Robert Kanter 
> Sent: Tuesday, December 12, 2017 3:10 PM
> To: Arun Suresh
> Cc: Andrew Wang; Lei Xu; Wei-Chiu Chuang; Ajay Kumar; Xiao Chen; Aaron T. 
> Myers; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC1
> 
> +1 (binding)
> 
> + Downloaded the binary release
> + Deployed on a 3 node cluster on CentOS 7.3
> + Ran some MR jobs, clicked around the UI, etc
> + Ran some CLI commands (yarn logs, etc)
> 
> Good job everyone on Hadoop 3!
> 
> 
> - Robert
> 
> On Tue, Dec 12, 2017 at 1:56 PM, Arun Suresh  wrote:
> 
>> +1 (binding)
>> 
>> - Verified signatures of the source tarball.
>> - built from source - using the docker build environment.
>> - set up a pseudo-distributed test cluster.
>> - ran basic HDFS commands
>> - ran some basic MR jobs
>> 
>> Cheers
>> -Arun
>> 
>> On Tue, Dec 12, 2017 at 1:52 PM, Andrew Wang 
>> wrote:
>> 
>>> Hi everyone,
>>> 
>>> As a reminder, this vote closes tomorrow at 12:31pm, so please give it a
>>> whack if you have time. There are already enough binding +1s to pass this
>>> vote, but it'd be great to get additional validation.
>>> 
>>> Thanks to everyone who's voted thus far!
>>> 
>>> Best,
>>> Andrew
>>> 
>>> 
>>> 
>>> On Tue, Dec 12, 2017 at 11:08 AM, Lei Xu  wrote:
>>> 
 +1 (binding)
 
 * Verified src tarball and bin tarball, verified md5 of each.
 * Build source with -Pdist,native
 * Started a pseudo cluster
 * Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run hdfs
 dfs put/get/cat on "/" with XOR-2-1 policy.
 
 Thanks Andrew for this great effort!
 
 Best,
 
 
 On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang >> 
 wrote:
> Hi Wei-Chiu,
> 
> The patchprocess directory is left over from the create-release
>>> process,
> and it looks empty to me. We should still file a create-release JIRA
>> to
 fix
> this, but I think this is not a blocker. Would you agree?
> 
> Best,
> Andrew
> 
> On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang <
>> weic...@cloudera.com
 
> wrote:
> 
>> Hi Andrew, thanks the tremendous effort.
>> I found an empty "patchprocess" directory in the source tarball,
>> that
>>> is
>> not there if you clone from github. Any chance you might have some
 leftover
>> trash when you made the tarball?
>> Not wanting to nitpicking, but you might want to double check so we
 don't
>> ship anything private to you in public :)
>> 
>> 
>> 
>> On Tue, Dec 12, 2017 at 7:48 AM, Ajay Kumar <
>>> ajay.ku...@hortonworks.com
> 
>> wrote:
>> 
>>> +1 (non-binding)
>>> Thanks for driving 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Looked at RC1. Went through my usual check-list. Here's my summary.

+1 (binding) overall

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
- [Check] Generating dist tarballs from source tar-ball
- [Check] Validating the layout of the binary tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

Few issues as before found during testing, but shouldn't be blockers
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty. Will file a ticket.
 - resourcemanager-metrics.out is going into current directory instead of log 
directory. Will file a ticket.

One thing I want to make sure folks agree to. Allen filed a ticket to remove 
yarn-historyserver option per our previous discussion - 
https://issues.apache.org/jira/browse/YARN-7588 
. It isn't done - I am hoping 
this 'incompatible change' can be put in 3.0.1 and not 4.0.

Thanks
+Vinod


> On Dec 8, 2017, at 12:31 PM, Andrew Wang  wrote:
> 
> Hi all,
> 
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
> 
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
> 
> You can find the artifacts here:
> 
> http://home.apache.org/~wang/3.0.0-RC1/
> 
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
> 
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
> 
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

I couldn't find the release tag for RC1 either - is it just me or has the 
release-process changed?

+Vinod

> On Dec 10, 2017, at 4:31 PM, Sangjin Lee  wrote:
> 
> Hi Andrew,
> 
> Thanks much for your effort! Just to be clear, could you please state the
> git commit id of the RC1 we're voting for?
> 
> Sangjin
> 
> On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
> wrote:
> 
>> Hi all,
>> 
>> Let me start, as always, by thanking the efforts of all the contributors
>> who contributed to this release, especially those who jumped on the issues
>> found in RC0.
>> 
>> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
>> fixed JIRAs since the previous 3.0.0-beta1 release.
>> 
>> You can find the artifacts here:
>> 
>> http://home.apache.org/~wang/3.0.0-RC1/
>> 
>> I've done the traditional testing of building from the source tarball and
>> running a Pi job on a single node cluster. I also verified that the shaded
>> jars are not empty.
>> 
>> Found one issue that create-release (probably due to the mvn deploy change)
>> didn't sign the artifacts, but I fixed that by calling mvn one more time.
>> Available here:
>> 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
>> 
>> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
>> Pacific. My +1 to start.
>> 
>> Best,
>> Andrew
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Missed this response on the old thread, but closing the loop here..

The incompatibility conundrum with Dot-zeroes did indeed happen, in early 2.x 
releases - multiple times at that. And the downstream projects did raise 
concerns at these unfixable situations.

I wasn't advocating a new formalism, this was more of a lesson taken from real 
life experience that I wanted share with fellow RMs - as IMO the effort was 
worth the value for the releases where I used it.

If RMs of these more recent releases choose to not do this if it is perceived 
that a release won't run into those past issues at all, it's clearly their 
call. It's just that we are bound to potentially make the same mistakes and 
learn the same lesson all over again..

+Vinod

> On Nov 9, 2017, at 9:51 AM, Chris Douglas <cdoug...@apache.org> wrote:
> 
> The labor required for these release formalisms is exceeding their
> value. Our minor releases have more bugs than our patch releases (we
> hope), but every consumer should understand how software versioning
> works. Every device I own has bugs on major OS updates. That doesn't
> imply that every minor release is strictly less stable than a patch
> release, and users need to be warned off it.
> 
> In contrast, we should warn users about features that compromise
> invariants like security or durability, either by design or due to
> their early stage of development. We can't reasonably expect them to
> understand those tradeoffs, since they depend on internal details of
> Hadoop.
> 
> On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli
> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
>> When we tried option (b), we used to make .0 as a GA release, but downstream 
>> projects like Tez, Hive, Spark would come back and find an incompatible 
>> change - and now we were forced into a conundrum - is fixing this 
>> incompatible change itself an incompatibility?
> 
> Every project takes these case-by-case. Most of the time we'll
> accommodate the old semantics- and we try to be explicit where we
> promise compatibility- but this isn't a logic problem, it's a
> practical one. If it's an easy fix to an obscure API, we probably
> won't even hear about it.
> 
>> Long story short, I'd just add to your voting thread and release notes that 
>> 2.9.0 still needs to be tested downstream and so users may want to wait for 
>> subsequent point releases.
> 
> It's uncomfortable to have four active release branches, with 3.1
> coming in early 2018. We all benefit from the shared deployment
> experiences that harden these releases, and fragmentation creates
> incentives to compete for that attention. Rather than tacitly
> scuffling over waning interest in the 2.x series, I'd endorse your
> other thread encouraging consolidation around 3.x.
> 
> To that end, there is no policy or precedent that requires that new
> minor releases be labeled as "alpha". If there is cause to believe
> that 2.9.0 is not ready to release in the stable line, then we
> shouldn't release it. -C
> 
>>> On Nov 8, 2017, at 12:43 AM, Subru Krishnan <su...@apache.org> wrote:
>>> 
>>> We are canceling the RC due to the issue that Rohith/Sunil identified. The
>>> issue was difficult to track down as it only happens when you use IP for ZK
>>> (works fine with host names) and moreover if ZK and RM are co-located on
>>> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
>>> 
>>> Thanks to everyone for the extensive testing/validation. Hopefully cost to
>>> replicate with RC1 is much lower.
>>> 
>>> -Subru/Arun.
>>> 
>>> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos <kkarana...@gmail.com
>>>> wrote:
>>> 
>>>> +1 from me too.
>>>> 
>>>> Did the following:
>>>> 1) set up a 9-node cluster;
>>>> 2) ran some Gridmix jobs;
>>>> 3) ran (2) after enabling opportunistic containers (used a mix of
>>>> guaranteed and opportunistic containers for each job);
>>>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>>>> containers.
>>>> 
>>>> All the above worked with no issues.
>>>> 
>>>> Thanks for all the effort guys!
>>>> 
>>>> Konstantinos
>>>> 
>>>> 
>>>> 
>>>> Konstantinos
>>>> 
>>>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger <ebad...@oath.com.invalid>
>>>> wrote:
>>>> 
>>>>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
>>>>> 
>>>>&

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
>> work. Not just deprecated in favor of timelineserver as was advertised.
> 
>   This works for me in trunk and the bash code doesn’t appear to have 
> changed in a very long time.  Probably something local to your install.  (I 
> do notice that the deprecation message says “starting” which is awkward when 
> the stop command is given though.)  Also: is the deprecation message even 
> true at this point?


Sorry, I mischaracterized the problem.

The real issue is that I cannot use this command line when the MapReduce 
JobHistoryServer is already started on the same machine.

~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.
DEPRECATED: Use of this command to start the timeline server is deprecated.
Instead use the timelineserver command for it.
Starting the History Server anyway...
historyserver is running as process 86156.  Stop it first.

So, it looks like in shell-scripts, there can ever be only one daemon of a 
given name, irrespective of which daemon scripts are invoked.

We need to figure out two things here
 (a) The behavior of this command. Clearly, it will conflict with the MapReduce 
JHS - only one of them can be started on the same node.
 (b) We need to figure out if this V1 TimelineService should even be support 
given ATSv2.

@Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please comment 
on (b).

Thanks
+Vinod

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>> 
>> 
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>> 
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
> 
>   -1 on making yarn-ui always build.
> 
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:
>> Info :-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
>> not in the CLI node -list
>> 
> 
> Is this a blocker? Could we get a JIRA?


I just cross verified this with others - looks like an issue with my 
environment - not a blocker, ignore.

+Vinod
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

 - [Check] Successful recompilation from source tar-ball
 - [Check] Signature verification
 - [Check] Generating dist tarballs from source tar-ball
 - [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
 - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
 - One decommissioned node in YARN ResourceManager UI always appears to start 
with, even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI 
node -list

Minor
 - resourcemanager-metrics.out is going into current directory instead of log 
directory
 - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work. 
Not just deprecated in favor of timelineserver as was advertised.
 - Spurious warnings on CLI
17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

 - When did we stop putting CHANGES files into the source artifacts?
 - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see 
how this can be avoided.
 - Compatibility notes
-- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME is 
not automatically inherited. Correct behavior)
-- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Compilation passed for me. Using jdk1.8.0_40.jdk.

+Vinod

> On Nov 20, 2017, at 4:16 PM, Wei-Chiu Chuang <weic...@cloudera.com> wrote:
> 
> @vinod
> I followed your command but I could not reproduce your problem.
> 
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
> ommon/target/hadoop-common-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
> hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
> s/target/hadoop-hdfs-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
> hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
> 
> During compilation I found the following error with a Java 1.8.0_5 JDK:
> 
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven
> -compiler-plugin:3.1:testCompile (default-testCompile) on project
> hadoop-aws: Compilation failure: Compilation failure:
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> 
> And then I realized Ray filed HADOOP-14900
> <https://issues.apache.org/jira/browse/HADOOP-14900> for the same
> issue. This problem is not reproducible with a more recent JDK 8, such as
> 1.8.0_151
> Maybe it would be a good idea to name a list of JDKs that are known to be
> buggy. Can we get this documented somewhere? I don't consider it a blocker
> so a release note in a later release or a wiki entry should be good enough.
> 
> On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> 
>> Quick question.
>> 
>> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
>> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
>> voted on (hadoop-3.0.0-src.tar.gz).
>> 
>> The idea is to install HDFS, YARN, MR separately in separate
>> root-directories from the generated individual dist tarballs.
>> 
>> But now I see that HDFS and common dist tarballs are empty
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
>> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
>> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>> 
>> But YARN and MR are fine
>> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
>> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
>> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
>> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>> 
>> Is it just me? Or is this broken?
>> 
>> Thanks
>> +Vinod
>> 
>>> On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.w...@cloude

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Quick question.

I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being voted 
on (hadoop-3.0.0-src.tar.gz).

The idea is to install HDFS, YARN, MR separately in separate root-directories 
from the generated individual dist tarballs.

But now I see that HDFS and common dist tarballs are empty
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz - 
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

But YARN and MR are fine
-rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
-rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz

Is it just me? Or is this broken?

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

I'd definitely extend it for a few more days. I only see 3 binding +1s so far - 
not a great number to brag about on our first major release in years.

Also going to nudge folks into voting.

+Vinod

> On Nov 17, 2017, at 3:26 PM, Andrew Wang  wrote:
> 
> Hi Arpit,
> 
> I agree the timing is not great here, but extending it to meaningfully
> avoid the holidays would mean extending it an extra week (e.g. to the
> 29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
> really like to get the RC out before then.
> 
> In terms of downstream testing, we've done extensive integration testing
> with downstreams via the alphas and betas, and we have continuous
> integration running at Cloudera against branch-3.0. Because of this, I have
> more confidence in our integration for 3.0.0 than most Hadoop releases.
> 
> Is it meaningful to extend to say, the 21st, which provides for a full week
> of voting?
> 
> Best,
> Andrew
> 
> On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
> wrote:
> 
>> Hi Andrew,
>> 
>> Thank you for your hard work in getting us to this step. This is our first
>> major GA release in many years.
>> 
>> I feel a 5-day vote window ending over the weekend before thanksgiving may
>> not provide sufficient time to evaluate this RC especially for downstream
>> components.
>> 
>> Would you please consider extending the voting deadline until a few days
>> after the thanksgiving holiday? It would be a courtesy to our broader
>> community and I see no harm in giving everyone a few days to evaluate it
>> more thoroughly.
>> 
>> On a lighter note, your deadline is also 4 minutes short of the required 5
>> days. :)
>> 
>> Regards,
>> Arpit
>> 
>> 
>> 
>> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>> 
>>Hi folks,
>> 
>>Thanks as always to the many, many contributors who helped with this
>>release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>available here:
>> 
>>http://people.apache.org/~wang/3.0.0-RC0/
>> 
>>This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> 
>>3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>additions include the merge of YARN resource types, API-based
>> configuration
>>of the CapacityScheduler, and HDFS router-based federation.
>> 
>>I've done my traditional testing with a pseudo cluster and a Pi job.
>> My +1
>>to start.
>> 
>>Best,
>>Andrew
>> 
>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

A related point - I thought I mentioned this in one of the release preparation 
threads, but in any case.

Starting 2.7.0, for every .0 release, we've been adding a disclaimer (to the 
voting thread as well as the final release) that the first release can 
potentially go through additional fixes to incompatible changes (besides 
stabilization fixes). We should do this with 2.9.0 too.

This has some history - long before this, we tried two different things: (a) 
downstream projects consume an RC (b) downstream projects consume a release. 
Option (a) was tried many times but it was increasingly getting hard to manage 
this across all the projects that depend on Hadoop. When we tried option (b), 
we used to make .0 as a GA release, but downstream projects like Tez, Hive, 
Spark would come back and find an incompatible change - and now we were forced 
into a conundrum - is fixing this incompatible change itself an 
incompatibility? So to avoid this problem, we've started marking the first few 
releases as alpha eventually making a stable point release. Clearly, specific 
users can still use this in production as long as we the Hadoop community 
reserve the right to fix incompatibilities.

Long story short, I'd just add to your voting thread and release notes that 
2.9.0 still needs to be tested downstream and so users may want to wait for 
subsequent point releases.

Thanks
+Vinod

> On Nov 8, 2017, at 12:43 AM, Subru Krishnan  wrote:
> 
> We are canceling the RC due to the issue that Rohith/Sunil identified. The
> issue was difficult to track down as it only happens when you use IP for ZK
> (works fine with host names) and moreover if ZK and RM are co-located on
> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
> 
> Thanks to everyone for the extensive testing/validation. Hopefully cost to
> replicate with RC1 is much lower.
> 
> -Subru/Arun.
> 
> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos > wrote:
> 
>> +1 from me too.
>> 
>> Did the following:
>> 1) set up a 9-node cluster;
>> 2) ran some Gridmix jobs;
>> 3) ran (2) after enabling opportunistic containers (used a mix of
>> guaranteed and opportunistic containers for each job);
>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>> containers.
>> 
>> All the above worked with no issues.
>> 
>> Thanks for all the effort guys!
>> 
>> Konstantinos
>> 
>> 
>> 
>> Konstantinos
>> 
>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
>> wrote:
>> 
>>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
>>> 
>>> - Verified all hashes and checksums
>>> - Built from source on macOS 10.12.6, Java 1.8.0u65
>>> - Deployed a pseudo cluster
>>> - Ran some example jobs
>>> 
>>> Thanks,
>>> 
>>> Eric
>>> 
>>> On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan  wrote:
>>> 
 Sunil / Rohith,
 
 Could you check if your configs are same as Jonathan posted configs?
 https://issues.apache.org/jira/browse/YARN-7453?
>>> focusedCommentId=16242693&
 page=com.atlassian.jira.plugin.system.issuetabpanels:
 comment-tabpanel#comment-16242693
 
 And could you try if using Jonathan's configs can still reproduce the
 issue?
 
 Thanks,
 Wangda
 
 
 On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
>> wrote:
 
> Thanks for testing Rohith and Sunil
> 
> Can you please confirm if it is not a config issue at your end ?
> We (both Jonathan and myself) just tried testing this on a fresh
>>> cluster
> (both automatic and manual) and we are not able to reproduce this.
>> I've
> updated the YARN-7453 > jira/browse/YARN-7453
 
> JIRA
> with details of testing.
> 
> Cheers
> -Arun/Subru
> 
> On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
> rohithsharm...@apache.org
>> wrote:
> 
>> Thanks Sunil for confirmation. Btw, I have raised YARN-7453
>>  JIRA to track
>> this
>> issue.
>> 
>> - Rohith Sharma K S
>> 
>> On 7 November 2017 at 16:44, Sunil G  wrote:
>> 
>>> Hi Subru and Arun.
>>> 
>>> Thanks for driving 2.9 release. Great work!
>>> 
>>> I installed cluster built from source.
>>> - Ran few MR jobs with application priority enabled. Runs fine.
>>> - Accessed new UI and it also seems fine.
>>> 
>>> However I am also getting same issue as Rohith reported.
>>> - Started an HA cluster
>>> - Pushed RM to standby
>>> - Pushed back RM to active then seeing an exception.
>>> 
>>> org.apache.hadoop.ha.ServiceFailedException: RM could not
>>> transition
 to
>>> Active
>>>at
>>> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyE
>>> lectorBasedElectorServic
>>>

Re: [DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Thanks for your comments, Zheng. Replies inline.


> On the other hand, I've discussed with quite a few 3.0 potential users, it 
> looks like most of them are interested in the erasure coding feature and a 
> major scenario for that is to back up their large volume of data to save 
> storage cost. They might run analytics workload using Hive, Spark, Impala and 
> Kylin on the new cluster based on the version, but it's not a must at the 
> first time. They understand there might be some gaps so they'd migrate their 
> workloads incrementally. For the major analytics workload, we've performed 
> lots of benchmark and integration tests as well as other sides I believe, we 
> did find some issues but they should be fixed in downstream projects. I 
> thought the release of GA will accelerate the progress and expose the issues 
> if any. We couldn't wait for it being matured. There isn't perfectness.


3.0 is a GA release from the Apache Hadoop community. So, we cannot assume that 
all usages in the short term are *only* going to be for storage optimization 
features and only on dedicated clusters. We have to make sure that the 
workloads can be migrated right now and/or that existing clusters can be 
upgraded in-place. If not, we shouldn't be calling it GA.


> This sounds a good consideration. I'm thinking if I'm a Hadoop user, for 
> example, I'm using 2.7.4 or 2.8.2 or whatever 2.x version, would I first 
> upgrade to this bridging release then use the bridge support to upgrade to 
> 3.x version? I'm not sure. On the other hand, I might tend to look for some 
> guides or supports in 3.x docs about how to upgrade from 2.7 to 3.x. 



Arun Suresh also asked this same question earlier. I think this will really 
depend on what we discover as part of the migration and user-acceptance 
testing. If we don't find major issues, you are right, folks can jump directly 
from one of 2.7, 2.8 or 2.9 to 3.0.



> Frankly speaking, working on some bridging release not targeting any feature 
> isn't so attractive to me as a contributor. Overall, the final minor release 
> off branch-2 is good, we should also give 3.x more time to evolve and mature, 
> therefore it looks to me we would have to work on two release lines meanwhile 
> for some time. I'd like option C), and suggest we focus on the recent 
> releases.



Answering this question is also one of the goals of my starting this thread. 
Collectively we need to conclude if we are okay or not okay with no longer 
putting any new feature work in general on the 2.x line after 2.9.0 release and 
move over our focus into 3.0.


Thanks
+Vinod

> -Original Message-
> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
> Sent: Tuesday, November 07, 2017 9:43 AM
> To: Andrew Wang <andrew.w...@cloudera.com>
> Cc: Arun Suresh <asur...@apache.org>; common-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; Hdfs-dev <hdfs-...@hadoop.apache.org>; 
> mapreduce-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] A final minor release off branch-2?
> 
> The main goal of the bridging release is to ease transition on stuff that is 
> guaranteed to be broken.
> 
> Of the top of my head, one of the biggest areas is application compatibility. 
> When folks move from 2.x to 3.x, are their apps binary compatible? Source 
> compatible? Or need changes?
> 
> In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
> source compatible. This means relooking at the API compatibility in 3.x and 
> their impact of migrating applications. We will have to revist and 
> un-deprecate old APIs, un-delete old APIs and write documentation on how apps 
> can be migrated.
> 
> Most of this work will be in 3.x line. The bridging release on the other hand 
> will have deprecation for APIs that cannot be undeleted. This may be already 
> have been done in many places. But we need to make sure and fill gaps if any.
> 
> Other areas that I can recall from the old days
> - Config migration: Many configs are deprecated or deleted. We need 
> documentation to help folks to move. We also need deprecations in the 
> bridging release for configs that cannot be undeleted.
> - You mentioned rolling-upgrades: It will be good to exactly outline the type 
> of testing. For e.g., the rolling-upgrades orchestration order has direct 
> implication on the testing done.
> - Story for downgrades?
> - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
> Is it broken anywhere that we cannot fix? Do we need bridging features for 
> this work?
> 
> +Vinod
> 
>> On Nov 6, 2017, at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
>> 
>> What are the known gaps that need bridging between 2.x and 3.x?
>> 
>> From an HDFS perspective,

Re: [DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

The main goal of the bridging release is to ease transition on stuff that is 
guaranteed to be broken.

Of the top of my head, one of the biggest areas is application compatibility. 
When folks move from 2.x to 3.x, are their apps binary compatible? Source 
compatible? Or need changes?

In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
source compatible. This means relooking at the API compatibility in 3.x and 
their impact of migrating applications. We will have to revist and un-deprecate 
old APIs, un-delete old APIs and write documentation on how apps can be 
migrated.

Most of this work will be in 3.x line. The bridging release on the other hand 
will have deprecation for APIs that cannot be undeleted. This may be already 
have been done in many places. But we need to make sure and fill gaps if any.

Other areas that I can recall from the old days
 - Config migration: Many configs are deprecated or deleted. We need 
documentation to help folks to move. We also need deprecations in the bridging 
release for configs that cannot be undeleted.
 - You mentioned rolling-upgrades: It will be good to exactly outline the type 
of testing. For e.g., the rolling-upgrades orchestration order has direct 
implication on the testing done.
 - Story for downgrades?
 - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
Is it broken anywhere that we cannot fix? Do we need bridging features for this 
work?

+Vinod

> On Nov 6, 2017, at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
> 
> What are the known gaps that need bridging between 2.x and 3.x?
> 
> From an HDFS perspective, we've tested wire compat, rolling upgrade, and
> rollback.
> 
> From a YARN perspective, we've tested wire compat and rolling upgrade. Arun
> just mentioned an NM rollback issue that I'm not familiar with.
> 
> Anything else? External to this discussion, these should be documented as
> known issues for 3.0.
> 
> Best.
> Andrew
> 
> On Sun, Nov 5, 2017 at 1:46 PM, Arun Suresh <asur...@apache.org> wrote:
> 
>> Thanks for starting this discussion VInod.
>> 
>> I agree (C) is a bad idea.
>> I would prefer (A) given that ATM, branch-2 is still very close to
>> branch-2.9 - and it is a good time to make a collective decision to lock
>> down commits to branch-2.
>> 
>> I think we should also clearly define what the 'bridging' release should
>> be.
>> I assume it means the following:
>> * Any 2.x user wanting to move to 3.x must first upgrade to the bridging
>> release first and then upgrade to the 3.x release.
>> * With regard to state store upgrades (at least NM state stores) the
>> bridging state stores should be aware of all new 3.x keys so the implicit
>> assumption would be that a user can only rollback from the 3.x release to
>> the bridging release and not to the old 2.x release.
>> * Use the opportunity to clean up deprecated API ?
>> * Do we even want to consider a separate bridging release for 2.7, 2.8 an
>> 2.9 lines ?
>> 
>> Cheers
>> -Arun
>> 
>> On Fri, Nov 3, 2017 at 5:07 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org>
>> wrote:
>> 
>>> Hi all,
>>> 
>>> With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out
>>> (tx Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we
>> have
>>> a discussion on how we manage our developmental bandwidth between 2.x
>> line
>>> and 3.x lines.
>>> 
>>> Once 3.0 GA goes out, we will have two parallel and major release lines.
>>> The last time we were in this situation was back when we did 1.x -> 2.x
>>> jump.
>>> 
>>> The parallel releases implies overhead of decisions, branch-merges and
>>> back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1,
>>> 3.0.1 and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of
>>> these lines - for e.g 2.8, 2.9 - are going to be used for a while at a
>>> bunch of large sites! At the same time, our users won't migrate to 3.0 GA
>>> overnight - so we do have to support two parallel lines.
>>> 
>>> I propose we start thinking of the fate of branch-2. The idea is to have
>>> one final release that helps our users migrate from 2.x to 3.x. This
>>> includes any changes on the older line to bridge compatibility issues,
>>> upgrade issues, layout changes, tooling etc.
>>> 
>>> We have a few options I think
>>> (A)
>>>-- Make 2.9.x the last minor release off branch-2
>>>-- Have a maintenance release that bridges 2.9 to 3.x
>>>-- Continue to make more ma

Re: [VOTE] Merge yarn-native-services branch into trunk

[Vinod Kumar Vavilapalli]

Congratulations to all the contributors involved, this is a great step forward!

+Vinod

> On Nov 6, 2017, at 2:40 PM, Jian He <j...@hortonworks.com> wrote:
> 
> Okay, I just merged the branch to trunk (108 commits in total !)
> Again, thanks for all who contributed to this feature!
> 
> Jian
> 
> On Nov 6, 2017, at 1:26 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> 
> Here’s +1 from myself.
> The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.
> 
> Thanks for all who voted. I’ll merge to trunk by the end of today.
> 
> Jian
> 
> On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
> <billie.rina...@gmail.com<mailto:billie.rina...@gmail.com>> wrote:
> 
> +1 (binding)
> 
> On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi All,
> 
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation, 
> DNS, CLI modifications etc. We believe now the feature is in a much better 
> shape.
> 
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to 
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS service 
> to enable users to discover services deployed on YARN via standard DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the existing 
> system, and have no impact on existing system if disabled.
> 
> Special thanks to a team of folks who worked hard towards this: Billie 
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K 
> S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without 
> their ideas and hard work.
> Also thanks Allen for some review and verifications.
> 
> Thanks,
> Jian
> 
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
> 
> 
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Hi all,

With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out (tx 
Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we have a 
discussion on how we manage our developmental bandwidth between 2.x line and 
3.x lines.

Once 3.0 GA goes out, we will have two parallel and major release lines. The 
last time we were in this situation was back when we did 1.x -> 2.x jump.

The parallel releases implies overhead of decisions, branch-merges and 
back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1, 3.0.1 
and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of these lines 
- for e.g 2.8, 2.9 - are going to be used for a while at a bunch of large 
sites! At the same time, our users won't migrate to 3.0 GA overnight - so we do 
have to support two parallel lines.

I propose we start thinking of the fate of branch-2. The idea is to have one 
final release that helps our users migrate from 2.x to 3.x. This includes any 
changes on the older line to bridge compatibility issues, upgrade issues, 
layout changes, tooling etc.

We have a few options I think
 (A)
-- Make 2.9.x the last minor release off branch-2
-- Have a maintenance release that bridges 2.9 to 3.x
-- Continue to make more maintenance releases on 2.8 and 2.9 as necessary
-- All new features obviously only go into the 3.x line as no features can 
go into the maint line.

 (B)
-- Create a new 2.10 release which doesn't have any new features, but as a 
bridging release
-- Continue to make more maintenance releases on 2.8, 2.9 and 2.10 as 
necessary
-- All new features, other than the bridging changes, go into the 3.x line

 (C)
-- Continue making branch-2 releases and postpone this discussion for later

I'm leaning towards (A) or to a lesser extent (B). Willing to hear otherwise.

Now, this obviously doesn't mean blocking of any more minor releases on 
branch-2. Obviously, any interested committer / PMC can roll up his/her 
sleeves, create a release plan and release, but we all need to acknowledge that 
versions are not cheap and figure out how the community bandwidth is split 
overall.

Thanks
+Vinod
PS: The proposal is obviously not to force everyone to go in one direction but 
more of a nudging the community to figure out if we can focus a major part of 
of our bandwidth on one line. I had a similar concern when we were doing 2.8 
and 3.0 in parallel, but the impending possibility of spreading too thin is 
much worse IMO.
PPS: (C) is a bad choice. With 2.8 and 2.9 we are already seeing user adoption 
splintering between two lines. With 2.10, 2.11 etc coexisting with 3.0, 3.1 
etc, we will revisit the mad phase years ago when we had 0.20.x, 0.20-security 
coexisting with 0.21, 0.22 etc.

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Arun / Subru,

Thanks for the great work!

Few quick comments
 - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and their 
signatures and delete everything else? So that it's easy to pick up the 
important bits for the voters. For e.g, like this 
http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/ 

 - Can you put the generated CHANGES.html and releasenotes.html instead of the 
md files, for quicker perusal?

Thanks
+Vinod

> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
> 
> Hi folks,
> 
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
> 
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
> 
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
> 
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
> *
> 
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
> 
> Thanks,
> 
> Arun/Subru

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Vinod Kumar Vavilapalli]

Thanks for starting this thread, Wangda!

+1 for establishing a faster cadence now itself.

One word of caution though. The same I expressed while we were trying to do 
both 2.8 and 3.0 releases at the same time. Please try avoiding concurrent 
releases and splitting community bandwidth - it's not that we cannot do 
multiple releases in parallel, it's mainly that we will not be able to give our 
best on both.

Thanks
+Vinod

> On Sep 6, 2017, at 11:13 AM, Wangda Tan  wrote:
> 
> Hi all,
> 
> As we discussed on [1], there were proposals from Steve / Vinod etc to have
> a faster cadence of releases and to start thinking of a Hadoop 3.1 release
> earlier than March 2018 as is currently proposed.
> 
> I think this is a good idea. I'd like to start the process sooner, and
> establish timeline etc so that we can be ready when 3.0.0 GA is out. With
> this we can also establish faster cadence for future Hadoop 3.x releases.
> 
> To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
> 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
> 6.5 months from now).
> 
> I'd also want to take this opportunity to come up with a more elaborate
> release plan to avoid some of the confusion we had with 3.0 beta. General
> proposal for the timeline (per this other proposal [2])
> - Feature freeze date - all features should be merged by Dec 15, 2017.
> - Code freeze date - blockers/critical only, no more improvements and non
> blocker/critical bug-fixes: Jan 1, 2018.
> - Release date: Jan 15, 2018
> 
> Following is a list of features on my radar which could be candidates for a
> 3.1 release:
> - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan Hung)
> - YARN-5881, Add absolute resource configuration to CapacityScheduler.
> (Owner: Sunil)
> - YARN-5673, Container-executor rewrite for better security, extensibility
> and portability. (Owner Varun Vasudev)
> - YARN-6223, GPU isolation. (Owner: Wangda)
> 
> And from email [3] mentioned by Andrew, there’re several other HDFS
> features want to be released with 3.1 as well, assuming they fit the
> timelines:
> - Storage Policy Satisfier
> - HDFS tiered storage
> 
> Please let me know if I missed any features targeted to 3.1 per this
> timeline.
> 
> And I want to volunteer myself as release manager of 3.1.0 release. Please
> let me know if you have any suggestions/concerns.
> 
> Thanks,
> Wangda Tan
> 
> [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
> Branch+merges+and+3%2E0%2E0-beta1+scope
> [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
> merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
> 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
> [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
> s+and+3%2E0%2E0-beta1+scope


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

> From a release management perspective, it's *extremely* reasonable to block 
> the inclusion of new features a month from the planned release date. A 
> typical software development lifecycle includes weeks of feature freeze and 
> weeks of code freeze. It is no knock on any developer or any feature to say 
> that we should not include something in 3.0.0.


We have never followed the ‘typical' lifecycle that I am guessing you are 
referring to. If we are, you'll need to publish some of the following: a 
feature freeze date, blockers-criticals-only-from-now date, testing-finish 
date, documentation-finish date, final release date and so on.

What we do with Apache releases typically is instead we say ‘this' is roughly 
when we want to release, and roughly what features must land and let the rest 
figure out itself.

Neither is right or wrong. If we want to change the process, we should 
communicate as such.

Proposing a feature freeze date on the fly is only going to confuse people.


> I've been very open and clear about the goals, schedule, and scope of 3.0.0 
> over the last year plus. The point of the extended alpha process was to get 
> all our features in during alpha, and the alpha merge window has been open 
> for a year. I'm unmoved by arguments about how long a feature has been worked 
> on. None of these were not part of the original 3.0.0 scope, and our users 
> have been waiting even longer for big-ticket 3.0 items like JDK8 and HDFS EC 
> that were part of the discussed scope.


Except our schedule is so fluid (not due to the release management process to 
be fair) that it is hard for folks to plan their features. IIRC, our schedule 
was a GA release beginning of this year. Again, this is not a critique of 3.0 
release process - I have myself done enough releases to know that sticking to a 
date and herding the crowd has been an extremely hard job.

The only way out of this is get 3.0 out and stick to 1-2 month minor releases. 
May be we start that by planning a 3.1 right now and push one in January 
assuming 3.0 GA happens in November. Cadence is a habit.


> I see that two VOTEs have gone out since I was out. I still plan to follow 
> the proposal in my original email. This means I'll cut branch-3 and 
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open up 
> development for Hadoop 3.1.0 and 4.0.0.
> 
> I'm reaching out to the lead contributor of each of these features 
> individually to discuss. We need to close on this quickly, and email is too 
> low bandwidth at this stage.


My only concern in all of this is if some of these branch contributors decide 
that they don’t want and then proceed to having a parallel release and cause 
pains for everyone. This is what I tried avoiding parallel 2.9 and 2.10 offline 
and that’s what I am trying to do here. For now, I don’t have a horse in this 
race - that’s between you and the branch-contributors in question for now. If 
you can reach an agreement, we are all good for it.

+Vinod



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Agreed. I was very clearly not advocating for rushing in features. If you have 
followed my past emails, I have only strongly advocated features be worked in 
branches and get merged when they are in a reasonable state.

Each branch contributor group should look at their readiness and merge stuff in 
assuming that the branch reached a satisfactory state. That’s it.

From release management perspective, blocking features just because we are a 
month close to the deadline is not reasonable. Let the branch contributors 
rationalize, make this decision and the rest of us can support them in making 
the decision.

+Vinod

> At this point, there have been three planned alphas from September 2016 until 
> July 2017 to "get in features".  While a couple of upcoming features are "a 
> few weeks" away, I think all of us are aware how predictable software 
> development schedules can be.  I think we can also all agree that rushing 
> just to meet a release deadline isn't the best practice when it comes to 
> software development either.
> 
> Andrew has been very clear about his goals at each step and I think Wangda's 
> willingness to not rush in resource types was an appropriate response.  I'm 
> sympathetic to the goals of getting in a feature for 3.0, but it might be a 
> good idea for each project that is a "few weeks away" to seriously look at 
> the readiness compared to the features which have been testing for 6+ months 
> already.
> 
> -Ray

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Vinod Kumar Vavilapalli]

Such a great community effort - hats off, team!

Thanks
+Vinod

> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vrushalic2...@gmail.com> 
> wrote:
> 
> Hi folks,
> 
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
> 
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
> 
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
> 
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
> 
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
> 
> There are a total of 82 subtasks that were completed as part of this effort.
> 
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
> 
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> 
> Regards,
> Vrushali
> 
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Steve,

You can be strict & ruthless about the timelines. Anything that doesn’t get in 
by mid-September, as was originally planned, can move to the next release - 
whether it is feature work on branches or feature work on trunk.

The problem I see here is that code & branches being worked on for a year are 
now (apparently) close to being done and we are telling them to hold for 7 more 
months - this is not a reasonable ask..

If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ can 
volunteer. But this is how you get competing releases and split bandwidth.

As for compatibility / testing etc, it seems like there is a belief that the 
current ‘scoped’ features are all tested well in these areas and so adding more 
is going to hurt the release. There is no way this is the reality, trunk has so 
many features that have been landing for years, the only way we can 
collectively attempt towards making this stable is by getting as many parties 
together as possible, each verifying stuff that they need. Not by excluding 
specific features.

+Vinod

> This is one of those curse-of-cadence things: The higher your release 
> cadence, the less pressure to get "everything in". With a slower cadence, 
> more pressure to get stuff in, more pressure to hold up the release, slows 
> the cadence, gets even more stuff in, etc. etc.
> 
> - Andrew has been working on the release for months, we all need to 
> appreciate how much hard work that is and has been, especially for what is 
> going to be a major release.
> 
> - We know that things will be unstable in 3.0; Andrew's concern is about 
> making sure that the newest, unstablest (?) features can at least be bypassed 
> if there are problems. I we should also call out in the release notes what we 
> think are the unstable bits where people need to use caution (example: 
> S3Guard in "authoritative" mode)
> 
> - Anything related to wire compatibility has been problematic in the past; I 
> think it's essential that whatever packets get sent around are going to be 
> stable, so changes there need to be in, or at least the payloads set up ready 
> for the features. Same for new public APIs.
> 
> - As fpr the rest, I don't know. I think being strict about it and ruthless 
> in assessing the feature's stability & consequences of postponing the feature 
> until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
> with a 3.2 in the summer.
> 
> Then: start planning that 3.1 release. Maybe I should put my hand up as 
> release manager for that one. Then everyone would realise how amenable Andrew 
> is being today.
> 
> 
> One other thing: alongside the big branches, there's the eternal backlog of 
> small patches. We should organise spending a few days updating, reviewing & 
> merging them in
> 
> -Steve
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
> 

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Andrew,

Each of the branches below have been created more than a year ago (!) and have 
been consistently worked upon and are now finally seeing the light of the day. 
When they are "few weeks” away, pushing them out by 7 *more* months just 
doesn’t make sense.

While I deeply appreciate the push for the dates, we shouldn’t be putting 
moratorium on features like this till the proposed date is due. As a release 
manager, it’s good to say that we will release a specific version as soon as 
so-and-so features are in, but let’s not put exclusions like this.

I propose that, as we have done with every release so far, (and more 
specifically the way we did 2.x alphas, betas, GA back in the day), we float 
the date, let the individual branch contributors decide their fate. As long as 
of course, they meet the timelines and the usual branch merge bar, testing / 
compatibility / impact on rest of the code-base etc.

Anything short of that, we will just be incentivizing contributors away from 
doing work in branches and towards putting stuff directly into trunk.

+Vinod

> On Aug 18, 2017, at 6:22 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> As you might have seen, we've had a number of branch merges floated this
> past week targeted for 3.0.0-beta1, which is planned for about a month from
> now.
> 
> In total, I'm currently tracking these branches:
> 
> YARN-2915: YARN federation (recently merged)
> HADOOP-13345: S3Guard (currently being voted on)
> YARN-5355: TSv2 alpha2 ("few weeks")
> YARN-5079: Native services ("few weeks")
> YARN-3926: Resource profiles ("few weeks")
> 
> We should effectively be in code freeze (only blockers/criticals), so the
> volume of merge proposals at this point came as a surprise. Despite our
> best efforts as software engineers, big code movement always comes with
> risk.
> 
> Since we started the 3.0 release series close to a year ago, I'm also loath
> to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
> HDFS EC, and our users deserve a production-quality release with these
> features. We've also been good about the release cadence thus far in 3.x,
> so a 3.1 isn't that far out.
> 
> Here's my proposal:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 
> My rationale for inclusion and exclusion is as follows:
> 
> Inclusion:
> 
> * YARN federation has been run in production, does not touch existing code,
> adds no new APIs, and is off by default.
> * S3Guard has been run in production and is off by default.
> * The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
> committed to this for 3.0.0 GA. It's off by default and adds no impact.
> 
> Exclusion:
> 
> * The primary reason for exclusion is to maintain the planned release
> schedule. Native services and resource profiles are still a few weeks from
> being ready for merge.
> * A reminder that 3.1 is only another 3 months after 3.0 given our release
> cadence thus far. If there's demand, we could even do a 3.1 immediately
> following 3.0.
> 
> I'm happy to talk with the contributors of each of these features to
> understand their timelines and requirements, with the caveat that I'll be
> out through Wednesday next week. Please reach out.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Vinod Kumar Vavilapalli]

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
> 
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
> 
> Thanks,
> 
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
> 
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
> 
> Jason
> 
> 
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
> 
> 
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
> 
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
> 
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
> 
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
> 
>  Please share thoughts if you have any. Thanks!
> 
> Thanks,
> 
> Junping
> 
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
> 
> Hi all,
> 
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
> 
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
> 
> Thanks
> +Vinod
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

Thanks for taking 2.7.4 over Konstantin!

Regarding rolling RC next week, I still see that there are 4 blocker / critical 
tickets targeted for 2.7.4: 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4
 
.

We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 
 definitely was something 
that was deemed a blocker for 2.8.2, not sure about 2.7.4.

I’m ‘back’ - let me know if you need any help.

Thanks
+Vinod

> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko  wrote:
> 
> Hi everybody.
> 
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
> 
> I plan to build an RC next week. If there are no objection.
> 
> Thanks,
> --Konst
> 
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko 
> wrote:
> 
>> Hey guys.
>> 
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>> 
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>> 
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>> 
>> There is a plan to enable BigTop for 2.7.4 testing.
>> 
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>> 
>> Regards,
>> --Konstantin
>> 
>> 
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka 
>> wrote:
>> 
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>> 
>>> -Akira
>>> 
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>> 
 Couple of more JIRAs need to be back ported for 2.7.4 release. These will
 solve RM HA unstability issues.
 https://issues.apache.org/jira/browse/YARN-5333
 https://issues.apache.org/jira/browse/YARN-5988
 https://issues.apache.org/jira/browse/YARN-6304
 
 I will raise a JIRAs to back port it.
 
 @Akira , could  you help to add these JIRAs into wiki?
 
 Thanks & Regards
 Rohith Sharma K S
 
 On 29 May 2017 at 12:19, Akira Ajisaka  wrote:
 
 Created a page for 2.7.4 release.
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
> 
> If you want to edit this wiki, please ping me.
> 
> Regards,
> Akira
> 
> 
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
> 
> Hi Konstantin Shvachko
>> 
>> 
>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>> trunk in following link.??
>> 
>> 
>> https://cwiki.apache.org/confluence/display/HADOOP
>> 
>> 
>> 
>> From: Konstantin Shvachko 
>> Sent: Saturday, May 13, 2017 3:58 AM
>> To: Akira Ajisaka
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>> yarn-...@hadoop.apache.org
>> Subject: Re: About 2.7.4 Release
>> 
>> Latest update on the links and filters. Here is the correct link for
>> the
>> filter:
>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>> requestId=12340814
>> 
>> Also updated: https://s.apache.org/Dzg4
>> 
>> Had to do some Jira debugging. Sorry for confusion.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>> shv.had...@gmail.com>
>> wrote:
>> 
>> Hey Akira,
>> 
>>> 
>>> I didn't have private filters. Most probably Jira caches something.
>>> Your filter is in the right direction, but for some reason it lists
>>> only
>>> 22 issues, while mine has 29.
>>> It misses e.g. YARN-5543 >> a/browse/YARN-5543>
>>> .
>>> 
>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release 

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Vinod Kumar Vavilapalli]

Thanks for taking the mantle from me on 2.8.0 and some persistent work getting 
2.8.0 out the door, Junping!

Apologies for bringing this up late, but I’d like to add one comment.

We should repeat what we did for 2.7.0 and In line with our experience there, 
we should annotate this release as not ready for production use. See the 
releases page - 
http://hadoop.apache.org/releases.html#25+August%2C+2016%3A+Release+2.7.3+available
 for our messaging on 2.7.0.

The expectation is that more downstream projects pick up the bits, iron out any 
incompatibilities we might have missed, and production users then pick up a 
solid 2.8.1.

Thanks
+Vinod

> On Mar 14, 2017, at 1:41 AM, Junping Du  wrote:
> 
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
> 
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
> 
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
> 
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
> 
>  The RC tag in git is: release-2.8.0-RC2
> 
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
> 
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
> 
> Thanks,
> 
> Junping


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

I was planning to take this up, celebrating my return from my paternity leave 
of absence for quite a while.

Marton, let me know if you do want to take this up instead and we can work 
together.

Thanks
+Vinod

> On Mar 7, 2017, at 9:13 AM, Sangjin Lee  wrote:
> 
> If we have a volunteer for releasing 2.7.4, we should go full speed
> ahead. We still need a volunteer from a PMC member or a committer as some
> tasks may require certain privileges, but I don't think it precludes
> working with others to close down the release.

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Here’s my +1 to conclude the vote.

With 8 binding +1s, 9 non-binding +1s and no -1s, this vote passes.

I’ll push the RC forward to an official release.

Thanks to everyone who voted!

+Vinod

> On Aug 24, 2016, at 10:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> +1 Binding.
> 
> - Built and deploy a single node cluster from source code.
> - Ran some example jobs.
> 
> Thanks,
> Wangda
> 
> On Tue, Aug 23, 2016 at 12:35 PM, Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
> 
>> +1 (non-binding)
>> 
>> - Downloaded source and tar verified the checksum
>> - Installed pseudo cluster with Node Labels and ATSV1 enabled.
>> - Verified with running MR apps with and without labels.
>> - Verified ATSV1 UI.
>> 
>> Regards,
>> + Naga
>> 
>> On Mon, Aug 22, 2016 at 11:55 PM, Jason Lowe <jl...@yahoo-inc.com.invalid>
>> wrote:
>> 
>>> +1 (binding)
>>> - Verified signatures and digests- Successfully built from source with
>>> native support- Deployed a single-node cluster- Ran some sample jobs
>>> successfully
>>> 
>>> Jason
>>> 
>>>  From: Vinod Kumar Vavilapalli <vino...@apache.org>
>>> To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>;
>>> hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; "
>>> mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
>>> Cc: Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Sent: Wednesday, August 17, 2016 9:05 PM
>>> Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2
>>> 
>>> Hi all,
>>> 
>>> I've created a new release candidate RC2 for Apache Hadoop 2.7.3.
>>> 
>>> As discussed before, this is the next maintenance release to follow up
>>> 2.7.2.
>>> 
>>> The RC is available for validation at: http://home.apache.org/~
>>> vinodkv/hadoop-2.7.3-RC2/ <http://home.apache.org/~
>>> vinodkv/hadoop-2.7.3-RC2/>
>>> 
>>> The RC tag in git is: release-2.7.3-RC2
>>> 
>>> The maven artifacts are available via repository.apache.org <
>>> http://repository.apache.org/> at https://repository.apache.org/
>>> content/repositories/orgapachehadoop-1046 <https://repository.apache.
>>> org/content/repositories/orgapachehadoop-1046>
>>> 
>>> The release-notes are inside the tar-balls at location
>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>>> hosted this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
>>> releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
>>> releasenotes.html> for your quick perusal.
>>> 
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - few more issues with RC1 forced a RC2 [2]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
>>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
>>> bit. This release's related discussion thread is linked below: [3].
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> Thanks,
>>> Vinod
>>> 
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
>> https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/index.html#26106 <
>>> https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/index.html#26106
>>> 
>>> [2] [VOTE] Release Apache Hadoop 2.7.3 RC1:
>> https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/msg26336.html <
>> https://www.mail-archive.com/
>>> hdfs-...@hadoop.apache.org/msg26336.html>
>>> [3] 2.7.3 release plan: https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
>>> 6yv2fyrs4jlepmmr>
>>> 
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Too late for 2.7.3 - I want us to quickly get back to a regular cadence of 
releases.

Doesn’t look like a regression IAC. Let’s put it in the next release if needed.

Thanks
+Vinod

> On Aug 22, 2016, at 6:19 AM, Brahma Reddy Battula 
>  wrote:
> 
> Felt like HDFS-8388, should be in for 2.7.3

[VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Canceling the release vote for this and other issues reported.

+Vinod

> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <ajisa...@oss.nttdata.co.jp> 
> wrote:
> 
> -1 (binding)
> 
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not 
> reflected in the release note.
> 
> -Akira
> 
> On 8/17/16 13:29, Allen Wittenauer wrote:
>> 
>> 
>> -1
>> 
>> HDFS-9395 is an incompatible change:
>> 
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>> 
>> 
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>>> wrote:
>>> 
>>> Hi all,
>>> 
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>> 
>>> As discussed before, this is the next maintenance release to follow up 
>>> 2.7.2.
>>> 
>>> The RC is available for validation at: 
>>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>> 
>>> The RC tag in git is: release-2.7.3-RC1
>>> 
>>> The maven artifacts are available via repository.apache.org 
>>> <http://repository.apache.org/> at 
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>>> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>> 
>>> The release-notes are inside the tar-balls at location 
>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>>> your quick perusal.
>>> 
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a 
>>> bit. This release's related discussion thread is linked below: [2].
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> Thanks,
>>> Vinod
>>> 
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: 
>>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>> 
>> 
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

I always look at CHANGES.txt entries for incompatible-changes and this JIRA 
obviously wasn’t there.

Anyways, this shouldn’t be in any of branch-2.* as committers there clearly 
mentioned that this is an incompatible change.

I am reverting the patch from branch-2* .

Thanks
+Vinod

> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <a...@effectivemachines.com> 
> wrote:
> 
> 
> 
> -1
> 
> HDFS-9395 is an incompatible change:
> 
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
> 
> 
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>> wrote:
>> 
>> Hi all,
>> 
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>> 
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>> 
>> The RC tag in git is: release-2.7.3-RC1
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>> your quick perusal.
>> 
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
>> This release's related discussion thread is linked below: [2].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Thanks Steve, this is one area that isn’t very well release-tested usually!

+Vinod

> On Aug 16, 2016, at 2:25 AM, Steve Loughran  wrote:
> 
> I've just looked at the staged JARs and how they worked with downstream apps 
> —that being a key way that Hadoop artifacts are adopted.

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Thanks Marco. It was a Thursday late-night slip-up.

Fixed the dates and replaced the bits, so the voting can continue.

FYI, they aren’t binding though - as it all depends on how the release voting 
goes. One should usually only trust the release-date published on the website.

Thanks
+Vinod

> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzueh...@gmail.com 
> <mailto:mzueh...@gmail.com>> wrote:
> 
> Hi Vinod,
> 
> I'm not sure if this is relevant, but you changed the release date in the 
> CHANGES.txt 
> <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
>  files to 2016-09-19.
> I guess you have meant 2016-08-19.
> 
> See: 
> https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
>  
> <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
> 
> 
> Thanks,
> Marco
> 
> 
> 
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vino...@apache.org 
> <mailto:vino...@apache.org>>:
> Hi all,
> 
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
> 
> The RC tag in git is: release-2.7.3-RC1
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> <http://repository.apache.org/ 
> <http://repository.apache.org/>> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
> your quick perusal.
> 
> As you may have noted,
>  - few issues with RC0 forced a RC1 [1]
>  - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
> This release's related discussion thread is linked below: [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
> 

[VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 


The RC tag in git is: release-2.7.3-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

But, everyone please do continue your sanity checking on RC0 in case there are 
more issues to be fixed.

Thanks
+Vinod

> On Jul 26, 2016, at 12:11 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Thanks Daniel and Wei.
> 
> I think these are worth fixing, I’m withdrawing this RC. Will look at fixing 
> these issues and roll a new candidate with the fixes as soon as possible.
> 
> Thanks
> +Vinod
> 
>> On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang <weic...@apache.org 
>> <mailto:weic...@apache.org>> wrote:
>> 
>> I noticed two issues:
>> 
>> (1) I ran hadoop checknative, but it seems the binary tarball was not 
>> compiled with native library for Linux. On the contrary, the Hadoop built 
>> from source tarball with maven -Pnative can find the native libraries on the 
>> same host.
>> 
>> (2) I noticed that the release dates in CHANGES.txt in tag release-2.7.3-RC0 
>> are set to Release 2.7.3 - 2016-07-27.
>> However, the release dates in CHANGES.txt in the source and binary tar balls 
>> are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue though.
>> 
>> * Downloaded source and binary.
>> * Verified signature.
>> * Verified checksum.
>> * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
>> went fine.
>> * Ran hadoop checknative
>> 
>> On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah 
>> <rusha...@yahoo-inc.com.invalid <mailto:rusha...@yahoo-inc.com.invalid>> 
>> wrote:
>> Thanks Vinod for all the release work !
>> +1 (non-binding).
>> * Downloaded from source and built it.* Deployed a pseudo distributed 
>> cluster.
>> * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything works 
>> fine.
>> 
>> 
>> On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
>> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
>> 
>> 
>>  Hi all,
>> 
>> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>> 
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>> 
>> The RC tag in git is: release-2.7.3-RC0
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> <http://repository.apache.org/ 
>> <http://repository.apache.org/>> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/> 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/>>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html> 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
>> your quick perusal.
>> 
>> As you may have noted, a very long fix-cycle for the License & Notice issues 
>> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
>> by quite a bit. This release's related discussion thread is linked below: 
>> [1].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>> 
>>
>> 
> 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

Thanks Daniel and Wei.

I think these are worth fixing, I’m withdrawing this RC. Will look at fixing 
these issues and roll a new candidate with the fixes as soon as possible.

Thanks
+Vinod

> On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang <weic...@apache.org> wrote:
> 
> I noticed two issues:
> 
> (1) I ran hadoop checknative, but it seems the binary tarball was not 
> compiled with native library for Linux. On the contrary, the Hadoop built 
> from source tarball with maven -Pnative can find the native libraries on the 
> same host.
> 
> (2) I noticed that the release dates in CHANGES.txt in tag release-2.7.3-RC0 
> are set to Release 2.7.3 - 2016-07-27.
> However, the release dates in CHANGES.txt in the source and binary tar balls 
> are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue though.
> 
> * Downloaded source and binary.
> * Verified signature.
> * Verified checksum.
> * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
> went fine.
> * Ran hadoop checknative
> 
> On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah <rusha...@yahoo-inc.com.invalid 
> <mailto:rusha...@yahoo-inc.com.invalid>> wrote:
> Thanks Vinod for all the release work !
> +1 (non-binding).
> * Downloaded from source and built it.* Deployed a pseudo distributed cluster.
> * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything works 
> fine.
> 
> 
> On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
> 
> 
>  Hi all,
> 
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
> 
> The RC tag in git is: release-2.7.3-RC0
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> <http://repository.apache.org/ 
> <http://repository.apache.org/>> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/> 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/>>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html> 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
> your quick perusal.
> 
> As you may have noted, a very long fix-cycle for the License & Notice issues 
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
> by quite a bit. This release's related discussion thread is linked below: [1].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
> 
>
> 

[DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Vinod Kumar Vavilapalli]

Forking the thread to make sure it attracts enough eye-balls. The earlier one 
was about 3.0.0 specifically and I don’t think enough people were watching that.

I’ll try to summarize a bit.

# Today’s state of release numbering and ordering:
So far, all the releases we have done, we have followed a simple timeline 
ordering. By this I mean that we always lined up releases one after another. 
Due to this, it was relatively simple to follow the causal ordering of 
releases, and we could answer ordering questions like “when was this patch 
first committed”.

The first time this started getting hairy was with parallel 2.6.x and 2.7.x 
releases. We managed the confusion by ordering them one after another: 2.7.1 -> 
2.6.2 -> 2.6.3 -> 2.7.2 -> 2.6.4 -> 2.7.3. This worked okay with two concurrent 
releases. 

Yes, by doing this, we could no longer answer questions by simply looking 
at the release numbers. But Sangjin / Junping / myself who did these 2.x 
releases ordered them one after another to avoid further confusion to 
developers and still retain the ability to just go back, look at the history 
and quickly answer the question of "what happened before and what happened 
after”. It wasn’t perfect, but we did what we did to keep it under control.

# What’s coming
Obviously, with 2.8.0 and 3.0.0-alpha1 release trains starting, this 
confusion goes to a whole new level. We’ll have 2.6.x, 2.7.x, 2.8.x, 3.0.0.x 
(and 2.9.x if we chose to make one) and following the ordering becomes a 
nightmare. The related problems that were discussed in the original thread was 
about how we mark fix-versions for patches, and how we generate change-logs - 
the answers for both of these follow the solution to the root problem of 
concurrent releases.

# Proposals?
There were some discussions about semantic versioning in the thread form 
which I forked this one from, but it’s not clear to me how semantic versioning 
supports concurrent release trains.

IAC, it’s time we look at this afresh and if need be, make some new rules 
before we make more of these releases and make it that much harder to follow 
for everyone.

Thanks
+Vinod
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

+1

Thanks
+Vinod

> On Jul 26, 2016, at 7:39 AM, Wangda Tan  wrote:
> 
> lets try to use both jdiff and the new tool and compare results because this 
> is the first time with the new tool.
> 
> Appreciate your time to help us about this effort!

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He <j...@hortonworks.com> wrote:
> 
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
> 
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
> 
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
> 
> Jian
> 
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:
> 
> Sounds good to me :).
> 
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
> 
> Will keep this thread updated.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>> wrote:
> 
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
> 
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
> 
> Thanks,
> Sangjin
> 
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:
> 
> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> for
> every releases.
> 
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>> wrote:
> 
> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
> 
> But I see a lot of JIRAs with no patches or very stale patches. It
> would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
> 
> Thanks,
> Sangjin
> 
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> <ste...@hortonworks.com<mailto:ste...@hortonworks.com>
> 
> wrote:
> 
> 
> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org<mailto:vino...@apache.org>>
> wrote:
> 
> We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start
> with
> reviewing Patch available tickets, that’ll be great.
> 
> 
> 
> 
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
> 
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure
> instrumentation
> 
> https://issues.apache.org/jira/browse/HADOOP-13028
> 
> 
> 
> 
> 
> 
> 
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

Please don’t do this for 2.8.0 against 2.7.0 till we change our existing 
release ordering conventions.

Till now, I made sure 2.8.0 only has issues that are not in 2.7.3 - per the 
2.8.0-follows-2.7.3 convention. As part of the release process, I will fix the 
remaining tickets to follow this same convention. Again till we create new 
rules.

Thanks
+Vinod

> On Jul 25, 2016, at 11:02 AM, Andrew Wang  wrote:
> 
> If I don't hear otherwise, I'd like to go ahead and do the bulk fix version
> update on JIRA for 3.0.0-alpha1, diffing from 2.7.0. Will wait until EOD
> tomorrow in case there's more discussion. If any one is willing to help
> review my script and JIRA queries, that'd also be great.
> 
> I can also do the 2.8.0 bulk update (diffing from 2.7.0), since it should
> be a very similar process.
> 
> Best,
> Andrew
> 
> 
> On Fri, Jul 22, 2016 at 9:50 PM, Allen Wittenauer 
> wrote:
> 
>> 
>>> On Jul 22, 2016, at 7:16 PM, Andrew Wang 
>> wrote:
>>> 
>>> Does this mean you find our current system of listing a JIRA as being
>> fixed in both a 2.6.x and 2.7.x to be confusing?
>> 
>>Nope.  I'm only confused when there isn't a .0 release in the fix
>> line.  When I see 2.6.x and 2.7.x I know that it was back ported to those
>> branches.  If I don't see a .0, I figure it's either a mistake or something
>> that was already fixed by another change in that major/minor branch.  It's
>> almost always the former, however.
>> 
>>> FWIW, my usecase is normally not "what is the earliest release that has
>> this fix?" but rather "is this fix in this release?". If it's easy to query
>> the latter, you can also determine the former. Some kind of query tool
>> could help here.
>> 
>>It literally becomes a grep if people commit the release data into
>> the source tree, the release data is correct, etc:
>> 
>> $ mvn install site  -Preleasedocs -Pdocs -DskipTests
>> $ grep issueid
>> hadoop-common-project/hadoop-common/src/site/markdown/release/*/CHANGES*
>> 
>>We should probably update the release process to make sure that
>> *in progress* release data is also committed when a .0 is cut.  That's
>> likely missing. Another choice would be to modify the pom to that runs
>> releasedocmaker to use a range rather than single version, but that gets a
>> bit tricky with release dates, how big of a range, etc.  Not impossible,
>> just tricky.  Probably needs to be script that gets run as part of
>> create-release, maybe?
>> 
>>(In reality, I do this grep against my git repo that generates the
>> change log data automatically.  This way it is always up-to-date and not
>> dependent upon release data being committed.  But that same grep could be
>> done with a JQL query just as easily.)
>> 
>>> For the release notes, am I correct in interpreting this as:
>>> 
>>> * diff a.0.0 from the previous x.y.0 release
>>> * diff a.b.0  from the previous a.0.0 or a.b.0 release
>>> * diff a.b.c from the previous a.b.0 or a.b.c release
>> 
>>Pretty much yes.
>> 
>>> Ray pointed me at the changelogs of a few other enterprise software
>> products, and this strategy seems pretty common. I like it.
>> 
>>It's extremely common, to the point that putting every fix for
>> every release touched is, at least to me, weird and extremely
>> unconventional.
>> 
>>> I realize now that this means a lot more JIRAs will need the 2.8.0 fix
>> version, since they only have 2.6.x and 2.7.x.
>> 
>>Yup.
>> 
>>>  This makes the fix rules actually pretty easy:  the lowest a.b.0
>> release and all non-.0 releases.
>>> 
>>> I think this needs to be amended to handle the case of multiple major
>> release branches, since we could have something committed for both 2.9.0
>> and 3.1.0. So "lowest a.b.0 release within each major version"?
>> 
>>Yeah, switching to effectively trunk-based development makes the
>> rules harder.  It's one of the reasons why the two big enterprisey
>> companies I worked at prior to working on Hadoop didn't really do
>> trunk-based for the vast majority of projects.  They always cut a branch
>> (or equivalent for that SCM) to delineate a break.   Given the amount of
>> ex-Sun folks involved in the early days of Hadoop, our pre-existing
>> development processes very much reflect that culture.
>> 
>>> This was true previously (no releases from trunk, trunk is versioned
>> a.0.0), but now that trunk is essentially a minor release branch, its fix
>> version needs to be treated as such.
>> 
>>Yeah, I misspoke a bit when dealing with a head-of-tree model.
>> 3.0.0-alpha1 will generate different notes than 3.0.0-alpha2, obviously.
>> Every 3.0.0-(label) release is effectively a major version in that case.
>> 
>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

Actually, I wouldn’t trust this report as it stands today at all.

I quickly glanced the report, looking for what it highlights as incompatible. 
But the ones that I saw have private / visible for testing annotations. Other 
than acting as useless evidence for those lashing out on branch-2, this won’t 
do much good.

Whenever we start working towards switching to this tool, it should incorporate 
the same exclude-annotations logic that the jdiff code-path does today. Do you 
think that is possible?

Thanks
+Vinod

> On Jul 22, 2016, at 4:53 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
>> Do you want this check from some particular branch-2 release? It
>> matters since the releases along branch-2 have themselves had some
>> noise[2].
>> 
>> [1]: https://github.com/lvc/japi-compliance-checker 
>> <https://github.com/lvc/japi-compliance-checker> 
>> <https://github.com/lvc/japi-compliance-checker 
>> <https://github.com/lvc/japi-compliance-checker>>
>> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/ 
>> <http://abi-laboratory.pro/java/tracker/timeline/hadoop/> 
>> <http://abi-laboratory.pro/java/tracker/timeline/hadoop/ 
>> <http://abi-laboratory.pro/java/tracker/timeline/hadoop/>>
>> 
>> -- 
>> busbey

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

I’ve been using jdiff simply because of a lack of alternative.

If you’ve had experience with tool [1], if you think it serves our purpose, and 
if you can spare some time, that’ll be greatly appreciated. I can also pitch in 
with whatever help is needed.

I think we should pick one of 2.6.3 or 2.7.2 as baseline.

Thanks
+Vinod

> On Jul 21, 2016, at 2:41 PM, Sean Busbey  wrote:
> 
> I can come up with this, atleast for Source / Binary API compatibility,
> provided folks don't mind if I use the Java API Compliance Checker[1]
> instead of jdiff.
> 
> I'm already familiar with quickly using it, esp with Audience
> Annotations from my work in HBase.
> 
> Do you want this check from some particular branch-2 release? It
> matters since the releases along branch-2 have themselves had some
> noise[2].
> 
> [1]: https://github.com/lvc/japi-compliance-checker 
> 
> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/ 
> 
> 
> -- 
> busbey

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

> I really, really want a 3.0.0-alpha1 ASAP, since it's basically impossible 
> for downstreams to test incompat changes and new features without a release 
> artifact. I've been doing test builds, and branch-3.0.0-alpha1 is ready for 
> an RC besides possibly this fix version issue.

Not arguing against the need for an alpha release, the question is if it can 
wait till after 2.8 gets done. 

Orthogonally, do we have a report of the incompatible changes? Like the one I 
generated for some of the branch-2 releases using late jdiff work from Li Lu 
etc. We should do this and fix any inadvertant incompatibilities. Without 
seeing this list of incompatibilities, why even make an alpha release and force 
downstream components to discover issues what can be identified through running 
reports.

Similarly the list of features we are enabling in this alpha would be good - 
may be update the Roadmap wiki. Things like classpath-isolation which were part 
of the original 3.x roadmap are still not done.

> * Community bandwidth isn't zero-sum. This particularly applies to people 
> working on features that are only present in trunk, like EC, shell script 
> rewrite, etc.


A bunch of us are going to be busy with finishing 2.8.0. It isn’t zero-sum, but 
it predicates those of us involved with 2.8.0 from looking at it, even though 
we are very interested in doing so.


> Longer-term, I assume the 2.x line is not ending with 2.8. So we'd still have 
> the issue of things committed for 2.9.0 that will be appearing for the first 
> timein 3.0.0-alpha1. Assuming a script exists to fix up 2.9 JIRAs, it's only 
> incrementally more work to also fix up 2.8 and other unreleased versions too.


Obviously, I am not making the case that this issue won’t happen ever. In fact, 
this already happened with the parallel 2.6.x and 2.7.x releases. And we 
precisely avoided major confusion there by lining up 2.7.2 behind 2.6.3 etc.

+Vinod
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

The L & N fixes just went out, I’m working to push out 2.7.3 - running into a 
Nexus issue. Once that goes out, I’ll immediately do a 2.8.0.

Like I requested before in one of the 3.x threads, can we just line up 
3.0.0-alpha1 right behind 2.8.0?

That simplifies most of this confusion, we can avoid splitting the bandwidth 
from the community on fixing blockers / vetting these concurrent releases. 
Waiting a little more for 3.0.0 alpha to avoid most of this is worth it, IMO.

Thanks
+Vinod

> On Jul 21, 2016, at 11:34 AM, Andrew Wang  wrote:
> 
> Hi all,
> 
> Since we're planning to spin releases off of both branch-2 and trunk, the
> changelog for 3.0.0-alpha1 based on JIRA information isn't accurate. This
> is because historically, we've only set 2.x fix versions, and 2.8.0 and
> 2.9.0 and etc have not been released. So there's a whole bunch of changes
> which will show up for the first time in 3.0.0-alpha1.
> 
> I think I can write a script to (carefully) add 3.0.0-alpha1 to these
> JIRAs, but I figured I'd give a heads up here in case anyone felt
> differently. I can also update the HowToCommit page to match.
> 
> Thanks,
> Andrew


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6733) MapReduce JerseyTest tests failing with "java.net.BindException: Address already in use"

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created MAPREDUCE-6733:
--

 Summary: MapReduce JerseyTest tests failing with 
"java.net.BindException: Address already in use"
 Key: MAPREDUCE-6733
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6733
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Priority: Critical


Similar to YARN-2912 / YARN-3433, MR JerseyTests fail when port 9998 is in 
external use. We should fix the MR tests too similar to YARN-2912.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DICUSS] Upgrading Guice to 4.0(HADOOP-12064)

[Vinod Kumar Vavilapalli]

My strong expectation is that we’ll have a version of classpath isolation in 
our first release of 3.x. I’m planning to spending some cycles right away on 
this.

Assuming classpath isolation gets in, it is reasonable to bump up our 
dependencies like Jetty / Guice to the latest stable versions.

Thanks
+Vinod

> On Jun 27, 2016, at 6:01 AM, Tsuyoshi Ozawa  wrote:
> 
> Hi developers,
> 
> I will plan to upgrade Google Guice dependency on trunk. The change
> also includes asm and cglib upgrade.
> I checked following points:
> 
> * Both HDFS and YARN UIs work well.
> * All webIU-related tests pass as described on HADOOP-12064.
> * Ran mapreduce job, and it works well.
> 
> https://issues.apache.org/jira/browse/HADOOP-12064
> 
> Do you have any concern or opinion?  I would like to merge it to trunk
> on this Friday if you have no objections.
> 
> Best,
> - Tsuyoshi
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Looking to a Hadoop 3 release

[Vinod Kumar Vavilapalli]

Tx for your replies, Andrew.

>> For exit criteria, how about we time box it? My plan was to do monthly
> alphas through the summer, leading up to beta in late August / early Sep.
> At that point we freeze and stabilize for GA in Nov/Dec.


Time-boxing is a reasonable exit-criterion.


> In this case, does trunk-incompat essentially become the new trunk? Or are
> we treating trunk-incompat as a feature branch, which periodically merges
> changes from trunk?


It’s the later. Essentially
 - trunk-incompat = trunk + only incompatible changes, periodically kept 
up-to-date to trunk
 - trunk is always ready to ship
 - and no compatible code gets left behind

The reason for my proposal like this is to address the tension between “there 
is lot of compatible code in trunk that we are not shipping” and “don’t ship 
trunk, it has incompatibilities”. With this, we will not have (compatible) code 
not getting shipped to users.

Obviously, we can forget about all of my proposal completely if everyone puts 
in all compatible code into branch-2 / branch-3 or whatever the main releasable 
branch is. This didn’t work in practice, have seen this not happening 
prominently during 0.21, and now 3.x.

There is another related issue - "my feature is nearly ready, so I’ll just 
merge it into trunk as we don’t release that anyways, but not the current 
releasable branch - I’m lazy to fix the last few stability related issues”. 
With this, we will (should) get more disciplined, take feature stability on a 
branch seriously and merge a feature branch only when it is truly ready!

> For 3.x, my strawman was to release off trunk for the alphas, then branch a
> branch-3 for the beta and onwards.


Repeating above, I’m proposing continuing to make GA 3.x releases also off of 
trunk! This way only incompatible changes don’t get shipped to users - by 
design! Eventually, trunk-incompat will be latest 3.x GA + enough incompatible 
code to warrant a 4.x, 5.x etc.

+Vinod

Re: Looking to a Hadoop 3 release

[Vinod Kumar Vavilapalli]

Nope.

I’m proposing making a new 3.x release (as has been discussed in this thread) 
off today’s trunk (instead of creating a fresh branch-3) and create a new 
trunk-incompt where incompatible changes that we don’t want in 3.x go.

This is mainly to avoid repeating the “we are not releasing 3.x off trunk” 
issue when we start thinking about 4.x or any such major release in the future.

We’ll do 2.8.x independently and later figure out if 2.9 is needed or not.

+Vinod

> On Apr 22, 2016, at 5:59 PM, Allen Wittenauer <a...@apache.org> wrote:
> 
> 
>> On Apr 22, 2016, at 5:38 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>> wrote:
>> 
>> On an unrelated note, offline I was pitching to a bunch of contributors 
>> another idea to deal with rotting trunk post 3.x: *Make 3.x releases off of 
>> trunk directly*.
>> 
>> What this gains us is that
>> - Trunk is always nearly stable or nearly ready for releases
>> - We no longer have some code lying around in some branch (today’s trunk) 
>> that is not releasable because it gets mixed with other undesirable and 
>> incompatible changes.
>> - This needs to be coupled with more discipline on individual features - 
>> medium to to large features are always worked upon in branches and get 
>> merged into trunk (and a nearing release!) when they are ready
>> - All incompatible changes go into some sort of a trunk-incompat branch and 
>> stay there till we accumulate enough of those to warrant another major 
>> release.
>> 
>> Thoughts?
> 
>   Unless I’m missing something, all this proposal does is (using today’s 
> branch names) effectively rename trunk to trunk-incompat and branch-2 to 
> trunk.  I’m unclear how moving "rotting trunk” to “rotting trunk-incompat” is 
> really progress.
> 
> 

Re: Looking to a Hadoop 3 release

[Vinod Kumar Vavilapalli]

Hi,

While welcoming the push for alphas, i think we should set some exit criteria. 
Otherwise, I can imagine us doing 3/4/5 alpha releases, and then getting 
restless about calling it beta or GA of whatever. Essentially, instead of 
today’s questions as to "why we aren’t doing a 3.x release", we’d be fielding a 
"why is 3.x still considered alpha” question. This happened with 2.x alpha 
releases too and it wasn’t fun.

On an unrelated note, offline I was pitching to a bunch of contributors another 
idea to deal with rotting trunk post 3.x: *Make 3.x releases off of trunk 
directly*.

What this gains us is that
 - Trunk is always nearly stable or nearly ready for releases
 - We no longer have some code lying around in some branch (today’s trunk) that 
is not releasable because it gets mixed with other undesirable and incompatible 
changes.
 - This needs to be coupled with more discipline on individual features - 
medium to to large features are always worked upon in branches and get merged 
into trunk (and a nearing release!) when they are ready
 - All incompatible changes go into some sort of a trunk-incompat branch and 
stay there till we accumulate enough of those to warrant another major release.

Thoughts?

+Vinod


> On Apr 21, 2016, at 4:31 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Very optimistically, we're still on track for a 3.0 alpha this month.
> Here's a JIRA query for 3.0 and 2.8:
> 
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20MAPREDUCE%2C%20YARN)%20AND%20%22Target%20Version%2Fs%22%20in%20(3.0.0%2C%202.8.0)%20AND%20statusCategory%20not%20in%20(Complete)%20ORDER%20BY%20priority
> 
> I think two of these are true alpha blockers: HADOOP-12892 and
> HADOOP-12893. I'm trying to help push both of those forward.
> 
> For the rest, I think it's probably okay to delay until the next alpha,
> since we're planning a few alphas leading up to beta. That said, if you are
> the owner of a Blocker targeted at 3.0.0, I'd encourage reviving those
> patches. The earlier the better for incompatible changes.
> 
> In all likelihood, this first release will slip into early May, but I'll be
> disappointed if we don't have an RC out before ApacheCon.
> 
> Best,
> Andrew
> 
> On Mon, Feb 22, 2016 at 3:19 PM, Colin P. McCabe  wrote:
> 
>> I think starting a 3.0 alpha soon would be a great idea.  As some
>> other people commented, this would come with no compatibility
>> guarantees, so that we can iron out any issues.
>> 
>> Colin
>> 
>> On Mon, Feb 22, 2016 at 1:26 PM, Zhe Zhang  wrote:
>>> Thanks Andrew for driving the effort!
>>> 
>>> +1 (non-binding) on starting the 3.0 release process now with 3.0 as an
>>> alpha.
>>> 
>>> I wanted to echo Andrew's point that backporting EC to branch-2 is a lot
>> of
>>> work. Considering that no concrete backporting plan has been proposed, it
>>> seems quite uncertain whether / when it can be released in 2.9. I think
>> we
>>> should rather concentrate our EC dev efforts to harden key features under
>>> the follow-on umbrella HDFS-8031 and make it solid for a 3.0 release.
>>> 
>>> Sincerely,
>>> Zhe
>>> 
>>> On Mon, Feb 22, 2016 at 9:25 AM Colin P. McCabe 
>> wrote:
>>> 
 +1 for a release of 3.0.  There are a lot of significant,
 compatibility-breaking, but necessary changes in this release... we've
 touched on some of them in this thread.
 
 +1 for a parallel release of 2.8 as well.  I think we are pretty close
 to this, barring a dozen or so blockers.
 
 best,
 Colin
 
 On Mon, Feb 22, 2016 at 2:56 AM, Steve Loughran >> 
 wrote:
> 
>> On 20 Feb 2016, at 15:34, Junping Du  wrote:
>> 
>> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds
 reasonable to have two alpha releases to go in parallel. Is EC feature
>> the
 main motivation of releasing hadoop 3 here? If so, I don't understand
>> why
 this feature cannot land on 2.8.x or 2.9.x as an alpha feature.
> 
> 
> 
>> If we release 3.0 in a month like plan proposed below, it means we
>> will
 have 4 active releases going in parallel - two alpha releases (2.8 and
>> 3.0)
 and two stable releases (2.6.x and 2.7.x). It brings a lot of
>> challenges in
 issues tracking and patch committing, not even mention the tremendous
 effort of release verification and voting.
>> I would like to propose to wait 2.8 release become stable (may be 2nd
 release in 2.8 branch cause first release is alpha due to discussion in
 another email thread), then we can move to 3.0 as the only alpha
>> release.
 In the meantime, we can bring more significant features (like ATS v2,
>> etc.)
 to trunk and consolidate stable releases in 2.6.x and 2.7.x. I believe
>> that
 make life easier. :)
>> Thoughts?
>> 
> 
> 2.8.0 is 

Re: Looking to a Hadoop 3 release

[Vinod Kumar Vavilapalli]

I kind of echo Junping’s comment too.

While 2.8 and 3.0 don’t need to be serialized in theory, in practice I’m 
desperately looking for help on 2.8.0. We haven’t been converging on 2.8.0 what 
with 50+ blocker / critical patches still unfinished. If postponing 3.x alpha 
to after a 2.8.0 alpha means undivided attention from the community, I’d 
strongly root for such a proposal.

Thanks
+Vinod

> On Feb 20, 2016, at 9:07 PM, Andrew Wang  wrote:
> 
> Hi Junping, thanks for the mail, inline:
> 
> On Sat, Feb 20, 2016 at 7:34 AM, Junping Du  wrote:
> 
>> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds
>> reasonable to have two alpha releases to go in parallel. Is EC feature the
>> main motivation of releasing hadoop 3 here? If so, I don't understand why
>> this feature cannot land on 2.8.x or 2.9.x as an alpha feature.
>> 
> 
> EC is one motivation, there are others too (JDK8, shell scripts, jar
> bumps). I'm open to EC going into branch-2, but I haven't seen any
> backporting yet and it's a lot of code.
> 
> 
>> If we release 3.0 in a month like plan proposed below, it means we will
>> have 4 active releases going in parallel - two alpha releases (2.8 and 3.0)
>> and two stable releases (2.6.x and 2.7.x). It brings a lot of challenges in
>> issues tracking and patch committing, not even mention the tremendous
>> effort of release verification and voting.
>> I would like to propose to wait 2.8 release become stable (may be 2nd
>> release in 2.8 branch cause first release is alpha due to discussion in
>> another email thread), then we can move to 3.0 as the only alpha release.
>> In the meantime, we can bring more significant features (like ATS v2, etc.)
>> to trunk and consolidate stable releases in 2.6.x and 2.7.x. I believe that
>> make life easier. :)
>> Thoughts?
>> 
>> Based on some earlier mails in this chain, I was planning to release off
> trunk. This way we avoid having to commit to yet-another-branch, and makes
> tracking easier since trunk will always be a superset of the branch-2's.
> This does mean though that trunk needs to be stable, and we need to be more
> judicious with branch merges, and quickly revert broken code.
> 
> Regarding RM/voting/validation efforts, Steve mentioned some scripts that
> he uses to automate Slider releases. This is something I'd like to bring
> over to Hadoop. Ideally, publishing an RC is push-button, and it comes with
> automated validation. I think this will help with the overhead. Also, since
> these will be early alphas, and there will be a lot of them, I'm not
> expecting anyone to do endurance runs on a large cluster before casting a
> +1.
> 
> Best,
> Andrew

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

We are not converging - there’s still 58 more. I need help from the community 
in addressing / review 2.8.0 blockers. If folks can start with reviewing Patch 
available tickets, that’ll be great.


Thanks
+Vinod

> On Apr 4, 2016, at 2:16 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Here we go again. The blocker / critical tickets ballooned up a lot, I see 64 
> now! : https://issues.apache.org/jira/issues/?filter=12334985
> 
> Also, the docs target (mvn package -Pdocs -DskipTests) is completely busted 
> on branch-2, I figured I have to backport a whole bunch of patches that are 
> only on trunk, and may be more fixes on top of that *sigh*
> 
> I’ll start pushing for progress for an RC in a week or two.
> 
> Any help towards this, reviewing/committing outstanding patches and 
> contributing to open items is greatly appreciated.
> 
> Thanks
> +Vinod
> 
>> On Feb 9, 2016, at 11:51 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>> wrote:
>> 
>> Sure. The last time I checked, there were 20 odd blocker/critical tickets 
>> too that’ll need some of my time.
>> 
>> Given that, if you can get them in before a week, we should be good.
>> 
>> +Vinod
>> 
>>> On Feb 5, 2016, at 1:19 PM, Subramaniam V K <subru...@gmail.com> wrote:
>>> 
>>> Vinod,
>>> 
>>> Thanks for initiating the 2.8 release thread. We are in late review stages
>>> for YARN-4420 (Add REST API for listing reservations) and YARN-2575 (Adding
>>> ACLs for reservation system), hoping to get them by next week. Any chance
>>> you can put off cutting 2.8 by a week as we are planning to deploy
>>> ReservationSystem and these are critical for that?
>>> 
>>> Cheers,
>>> Subru
>>> 
>>> On Thu, Feb 4, 2016 at 3:17 PM, Chris Nauroth <cnaur...@hortonworks.com>
>>> wrote:
>>> 
>>>> FYI, I've just needed to raise HDFS-9761 to blocker status for the 2.8.0
>>>> release.
>>>> 
>>>> --Chris Nauroth
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 2/3/16, 6:19 PM, "Karthik Kambatla" <ka...@cloudera.com> wrote:
>>>> 
>>>>> Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me.
>>>>> Let us not call it alpha or beta though, it is quite confusing. :)
>>>>> 
>>>>> On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma <uma.ganguma...@intel.com
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Thanks Vinod. +1 for 2.8 release start.
>>>>>> 
>>>>>> Regards,
>>>>>> Uma
>>>>>> 
>>>>>> On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" <vino...@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> Seems like all the features listed in the Roadmap wiki are in. I¹m
>>>>>> going
>>>>>>> to try cutting an RC this weekend for a first/non-stable release off of
>>>>>>> branch-2.8.
>>>>>>> 
>>>>>>> Let me know if anyone has any objections/concerns.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>> 
>>>>>>>> On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli
>>>>>>>> <vino...@apache.org> wrote:
>>>>>>>> 
>>>>>>>> Branch-2.8 is created.
>>>>>>>> 
>>>>>>>> As mentioned before, the goal on branch-2.8 is to put improvements /
>>>>>>>> fixes to existing features with a goal of converging on an alpha
>>>>>> release
>>>>>>>> soon.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Nov 25, 2015, at 5:30 PM, Vinod Kumar Vavilapalli
>>>>>>>>> <vino...@apache.org> wrote:
>>>>>>>>> 
>>>>>>>>> Forking threads now in order to track all things related to the
>>>>>>>>> release.
>>>>>>>>> 
>>>>>>>>> Creating the branch now.
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> +Vinod
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Nov 25, 2015, at 11:37 AM, Vinod Kumar Vavilapalli
>>>>>>>>>> <vino...@apache.org> wrote:
>>>>>>>>>> 
>>>>>>>>>> I think we¹ve converged at a high level w.r.t 2.8. And as I just
>>>>>> sent
>>>>>>>>>> out an email, I updated the Roadmap wiki reflecting the same:
>>>>>>>>>> https://wiki.apache.org/hadoop/Roadmap
>>>>>>>>>> <https://wiki.apache.org/hadoop/Roadmap>
>>>>>>>>>> 
>>>>>>>>>> I plan to create a 2.8 branch EOD today.
>>>>>>>>>> 
>>>>>>>>>> The goal for all of us should be to restrict improvements & fixes
>>>>>> to
>>>>>>>>>> only (a) the feature-set documented under 2.8 in the RoadMap wiki
>>>>>> and
>>>>>>>>>> (b) other minor features that are already in 2.8.
>>>>>>>>>> 
>>>>>>>>>> Thanks
>>>>>>>>>> +Vinod
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Nov 11, 2015, at 12:13 PM, Vinod Kumar Vavilapalli
>>>>>>>>>>> <vino...@hortonworks.com <mailto:vino...@hortonworks.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> - Cut a branch about two weeks from now
>>>>>>>>>>> - Do an RC mid next month (leaving ~4weeks since branch-cut)
>>>>>>>>>>> - As with 2.7.x series, the first release will still be called as
>>>>>>>>>>> early / alpha release in the interest of
>>>>>>>>>>> ‹ gaining downstream adoption
>>>>>>>>>>> ‹ wider testing,
>>>>>>>>>>> ‹ yet reserving our right to fix any inadvertent
>>>>>> incompatibilities
>>>>>>>>>>> introduced.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli]

Others and I committed a few, I pushed out a few.

Down to just three now!

+Vinod

> On Apr 6, 2016, at 3:00 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Down to only 10 blocker / critical tickets 
> (https://issues.apache.org/jira/issues/?filter=12335343 
> <https://issues.apache.org/jira/issues/?filter=12335343>) now!
> 
> Thanks
> +Vinod
> 
>> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vino...@apache.org 
>> <mailto:vino...@apache.org>> wrote:
>> 
>> Hi all,
>> 
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which 
>> did go out mid February). Got a little busy since.
>> 
>> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. 
>> The focus obviously is to have blocker issues [1], bug-fixes and *no* 
>> features / improvements.
>> 
>> I hope to cut an RC in a week - giving enough time for outstanding blocker / 
>> critical issues. Will start moving out any tickets that are not blockers 
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets 
>> outstanding as of now.
>> 
>> Thanks,
>> +Vinod
>> 
>> [1] 2.7.3 release blockers: 
>> https://issues.apache.org/jira/issues/?filter=12335343 
>> <https://issues.apache.org/jira/issues/?filter=12335343>
> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli]

Down to only 10 blocker / critical tickets 
(https://issues.apache.org/jira/issues/?filter=12335343 
<https://issues.apache.org/jira/issues/?filter=12335343>) now!

Thanks
+Vinod

> On Mar 30, 2016, at 4:18 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Hi all,
> 
> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did 
> go out mid February). Got a little busy since.
> 
> Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. 
> The focus obviously is to have blocker issues [1], bug-fixes and *no* 
> features / improvements.
> 
> I hope to cut an RC in a week - giving enough time for outstanding blocker / 
> critical issues. Will start moving out any tickets that are not blockers 
> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets 
> outstanding as of now.
> 
> Thanks,
> +Vinod
> 
> [1] 2.7.3 release blockers: 
> https://issues.apache.org/jira/issues/?filter=12335343

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli]

Allen,

I see you marked it for 2.7.3, so it’s there on my radar.

That said, I’ll definitely need help from you (and likely others) in getting it 
fixed.

Thanks
+Vinod

> On Apr 6, 2016, at 2:13 PM, Allen Wittenauer 
>  wrote:
> 
> 
>   This is probably a good time to remind/point folks to HADOOP-12893.  
> Apache Hadoop's binary artifacts (with or without native code) and source 
> artifacts are not complying with the licenses of bundled components.  I 
> fairly confident this means releases are off the table until someone audits 
> the entire codebase given how many have been added without any thought to 
> actually following the terms of the license….

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli]

Thanks Chris.

+1 for reverting form 2.7. This is the least we should do. Can you help doing 
the needful?

I personally am not completely sold on a release with *only* the layout 
changes. Like I was saying before, we can let specific users backport this into 
specific 2.x branches they need and leave it only on trunk / branch-2. That 
said, I would love to hear others’ thoughts on this, but let’s fork that 
discussion off from this 2.7.3 thread. Re a fresh 2.8, I have renewed my 
efforts on 2.8 with a view of cutting an RC in weeks. Not sure if that does or 
doesn’t help this discussion.

Thanks
+Vinod

> On Apr 5, 2016, at 2:03 PM, Chris Trezzo <ctre...@gmail.com> wrote:
> 
> In light of the additional conversation on HDFS-8791, I would like to
> re-propose the following:
> 
> 1. Revert the new datanode layout (HDFS-8791) from the 2.7 branch. The
> layout change currently does not support downgrades which breaks our
> upgrade/downgrade policies for dot releases.
> 
> 2. Cut a 2.8 release off of the 2.7.3 release with the addition of
> HDFS-8791. This would give customers a stable release that they could
> deploy with the new layout. As discussed on the jira, this is still in line
> with user expectation for minor releases as we have done layout changes in
> a number of 2.x minor releases already. The current 2.8 would become 2.9
> and continue its current release schedule.
> 
> What does everyone think? If unsupported downgrades between minor releases
> is still not agreeable, then as stated by Vinod, we would need to either
> add support for downgrades with dn layout changes or revert the layout
> change from branch-2. If we are OK with the layout change in a minor
> release, but think that the issue does not affect enough customers to
> warrant a separate release, we could simply leave it in branch-2 and let it
> be released with the current 2.8.
> 
> 
> On Mon, Apr 4, 2016 at 1:48 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
> wrote:
> 
>> I commented on the JIRA way back (see
>> https://issues.apache.org/jira/browse/HDFS-8791?focusedCommentId=1503=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1503),
>> saying what I said below. Unfortunately, I haven’t followed the patch along
>> after my initial comment.
>> 
>> This isn’t about any specific release - starting 2.6 we declared support
>> for rolling upgrades and downgrades. Any patch that breaks this should not
>> be in branch-2.
>> 
>> Two options from where I stand
>> (1) For folks who worked on the patch: Is there a way to make (a) the
>> upgrade-downgrade seamless for people who don’t care about this (b) and
>> have explicit documentation for people who care to switch this behavior on
>> and are willing to risk not having downgrades. If this means a new
>> configuration property, so be it. It’s a necessary evil.
>> (2) Just let specific users backport this into specific 2.x branches they
>> need and leave it only on trunk.
>> 
>> Unless this behavior stops breaking rolling upgrades/downgrades, I think
>> we should just revert it from branch-2 and definitely 2.7.3 as it stands
>> today.
>> 
>> +Vinod
>> 
>> 
>>> On Apr 1, 2016, at 2:54 PM, Chris Trezzo <ctre...@gmail.com> wrote:
>>> 
>>> A few thoughts:
>>> 
>>> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
>>> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
>>> for data nodes depending on your setup.
>>> 
>>> 2. We have already deployed this patch internally so, with my Twitter hat
>>> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
>>> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
>>> releases on a large production cluster that has a diverse set of block
>> ids
>>> without this patch, especially if your data nodes have a large number of
>>> disks or you are using federation. To be clear though: this highly
>> depends
>>> on your setup and at a minimum you should verify that this regression
>> will
>>> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has
>> a
>>> performance regression that gets worse over time. When you see it
>> happening
>>> on a live cluster, it is one of the harder issues to identify a root
>> cause
>>> and debug. I do understand that this is currently only affecting a
>> smaller
>>> number of users, but I also think this number has potential to increase
>> as
>>> time goes on. Maybe we can issue a warning in the release notes for
>>

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

Here we go again. The blocker / critical tickets ballooned up a lot, I see 64 
now! : https://issues.apache.org/jira/issues/?filter=12334985

Also, the docs target (mvn package -Pdocs -DskipTests) is completely busted on 
branch-2, I figured I have to backport a whole bunch of patches that are only 
on trunk, and may be more fixes on top of that *sigh*

I’ll start pushing for progress for an RC in a week or two.

Any help towards this, reviewing/committing outstanding patches and 
contributing to open items is greatly appreciated.

Thanks
+Vinod

> On Feb 9, 2016, at 11:51 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Sure. The last time I checked, there were 20 odd blocker/critical tickets too 
> that’ll need some of my time.
> 
> Given that, if you can get them in before a week, we should be good.
> 
> +Vinod
> 
>> On Feb 5, 2016, at 1:19 PM, Subramaniam V K <subru...@gmail.com> wrote:
>> 
>> Vinod,
>> 
>> Thanks for initiating the 2.8 release thread. We are in late review stages
>> for YARN-4420 (Add REST API for listing reservations) and YARN-2575 (Adding
>> ACLs for reservation system), hoping to get them by next week. Any chance
>> you can put off cutting 2.8 by a week as we are planning to deploy
>> ReservationSystem and these are critical for that?
>> 
>> Cheers,
>> Subru
>> 
>> On Thu, Feb 4, 2016 at 3:17 PM, Chris Nauroth <cnaur...@hortonworks.com>
>> wrote:
>> 
>>> FYI, I've just needed to raise HDFS-9761 to blocker status for the 2.8.0
>>> release.
>>> 
>>> --Chris Nauroth
>>> 
>>> 
>>> 
>>> 
>>> On 2/3/16, 6:19 PM, "Karthik Kambatla" <ka...@cloudera.com> wrote:
>>> 
>>>> Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me.
>>>> Let us not call it alpha or beta though, it is quite confusing. :)
>>>> 
>>>> On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma <uma.ganguma...@intel.com
>>>> 
>>>> wrote:
>>>> 
>>>>> Thanks Vinod. +1 for 2.8 release start.
>>>>> 
>>>>> Regards,
>>>>> Uma
>>>>> 
>>>>> On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" <vino...@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> Seems like all the features listed in the Roadmap wiki are in. I¹m
>>>>> going
>>>>>> to try cutting an RC this weekend for a first/non-stable release off of
>>>>>> branch-2.8.
>>>>>> 
>>>>>> Let me know if anyone has any objections/concerns.
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>>> On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli
>>>>>>> <vino...@apache.org> wrote:
>>>>>>> 
>>>>>>> Branch-2.8 is created.
>>>>>>> 
>>>>>>> As mentioned before, the goal on branch-2.8 is to put improvements /
>>>>>>> fixes to existing features with a goal of converging on an alpha
>>>>> release
>>>>>>> soon.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>> 
>>>>>>> 
>>>>>>>> On Nov 25, 2015, at 5:30 PM, Vinod Kumar Vavilapalli
>>>>>>>> <vino...@apache.org> wrote:
>>>>>>>> 
>>>>>>>> Forking threads now in order to track all things related to the
>>>>>>>> release.
>>>>>>>> 
>>>>>>>> Creating the branch now.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Nov 25, 2015, at 11:37 AM, Vinod Kumar Vavilapalli
>>>>>>>>> <vino...@apache.org> wrote:
>>>>>>>>> 
>>>>>>>>> I think we¹ve converged at a high level w.r.t 2.8. And as I just
>>>>> sent
>>>>>>>>> out an email, I updated the Roadmap wiki reflecting the same:
>>>>>>>>> https://wiki.apache.org/hadoop/Roadmap
>>>>>>>>> <https://wiki.apache.org/hadoop/Roadmap>
>>>>>>>>> 
>>>>>>>>> I plan to create a 2.8 branch EOD today.
>>>>>>>>> 
>>>>>>>>> The goal for all of us should be to restrict improvements & fixes
>>>>> to
>>>>>>>>> only (a) the feature-set documented under 2.8 in the RoadMap wiki
>>>>> and
>>>>>>>>> (b) other minor features that are already in 2.8.
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> +Vinod
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Nov 11, 2015, at 12:13 PM, Vinod Kumar Vavilapalli
>>>>>>>>>> <vino...@hortonworks.com <mailto:vino...@hortonworks.com>> wrote:
>>>>>>>>>> 
>>>>>>>>>> - Cut a branch about two weeks from now
>>>>>>>>>> - Do an RC mid next month (leaving ~4weeks since branch-cut)
>>>>>>>>>> - As with 2.7.x series, the first release will still be called as
>>>>>>>>>> early / alpha release in the interest of
>>>>>>>>>> ‹ gaining downstream adoption
>>>>>>>>>> ‹ wider testing,
>>>>>>>>>> ‹ yet reserving our right to fix any inadvertent
>>>>> incompatibilities
>>>>>>>>>> introduced.
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
> 

Re: 2.7.3 release plan

[Vinod Kumar Vavilapalli]

 that folks will have cluster downtime in order to back
>>>>> things out. If that happens to any substantial number of downstream
>>>>> folks, or any particularly vocal downstream folks, then it is very
>>>>> likely we'll lose the remaining trust of operators for rolling out
>>>>> maintenance releases. That's a pretty steep cost.
>>>>> 
>>>>> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>>>>> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>>>>> unreasonable burden.
>>>>> 
>>>>> I agree that this fix is important, I just think we should either cut
>>>>> a version of 2.8 that includes it or find a way to do it that gives an
>>>>> operational path for rolling downgrade.
>>>>> 
>>>>> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <j...@hortonworks.com>
>>> wrote:
>>>>>> Thanks for bringing up this topic, Sean.
>>>>>> When I released our latest Hadoop release 2.6.4, the patch of
>>> HDFS-8791
>>>>> haven't been committed in so that's why we didn't discuss this
>> earlier.
>>>>>> I remember in JIRA discussion, we treated this layout change as a
>>>>> Blocker bug that fixing a significant performance regression before
>> but
>>> not
>>>>> a normal performance improvement. And I believe HDFS community already
>>> did
>>>>> their best with careful and patient to deliver the fix and other
>> related
>>>>> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>>> you
>>>>> can see 30+ rounds patch review back and forth by senior committers,
>>> not to
>>>>> mention the outstanding performance test data in HDFS-8791.
>>>>>> I would trust our HDFS committers' judgement to land HDFS-8791 on
>>>>> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>>> for
>>>>> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>>>>> 2.6.5 now.
>>>>>> Just my 2 cents.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Junping
>>>>>> 
>>>>>> 
>>>>>> From: Sean Busbey <bus...@cloudera.com>
>>>>>> Sent: Thursday, March 31, 2016 2:57 PM
>>>>>> To: hdfs-...@hadoop.apache.org
>>>>>> Cc: Hadoop Common; yarn-...@hadoop.apache.org;
>>>>> mapreduce-dev@hadoop.apache.org
>>>>>> Subject: Re: 2.7.3 release plan
>>>>>> 
>>>>>> A layout change in a maintenance release sounds very risky. I saw
>> some
>>>>>> discussion on the JIRA about those risks, but the consensus seemed
>> to
>>>>>> be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>>>>>> we did RMs per release rather than per branch? No one claiming to
>> be a
>>>>>> release manager ever spoke up AFAICT.
>>>>>> 
>>>>>> Should this change be included? Should it go into a special 2.8
>>>>>> release as mentioned in the ticket?
>>>>>> 
>>>>>> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>>>>>> <ajisa...@oss.nttdata.co.jp> wrote:
>>>>>>> Thank you Vinod!
>>>>>>> 
>>>>>>> FYI: 2.7.3 will be a bit special release.
>>>>>>> 
>>>>>>> HDFS-8791 bumped up the datanode layout version,
>>>>>>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>>>>>>> is impossible. We can rollback instead.
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/HDFS-8791
>>>>>>> 
>>>>> 
>>> 
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Akira
>>>>>>> 
>>>>>>> 
>>>>>>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>>>>> (which
>>>>>>>> did go out mid February). Got a little busy since.
>>>>>>>> 
>>>>>>>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>>>>>>>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>>>>> and *no*
>>>>>>>> features / improvements.
>>>>>>>> 
>>>>>>>> I hope to cut an RC in a week - giving enough time for outstanding
>>>>> blocker
>>>>>>>> / critical issues. Will start moving out any tickets that are not
>>>>> blockers
>>>>>>>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>>>>> tickets
>>>>>>>> outstanding as of now.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> [1] 2.7.3 release blockers:
>>>>>>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> busbey
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> busbey
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>> 

2.7.3 release plan

[Vinod Kumar Vavilapalli]

Hi all,

Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which did 
go out mid February). Got a little busy since.

Following up the 2.7.2 maintenance release, we should work towards a 2.7.3. The 
focus obviously is to have blocker issues [1], bug-fixes and *no* features / 
improvements.

I hope to cut an RC in a week - giving enough time for outstanding blocker / 
critical issues. Will start moving out any tickets that are not blockers and/or 
won’t fit the timeline - there are 3 blockers and 15 critical tickets 
outstanding as of now.

Thanks,
+Vinod

[1] 2.7.3 release blockers: 
https://issues.apache.org/jira/issues/?filter=12335343

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

Sure. The last time I checked, there were 20 odd blocker/critical tickets too 
that’ll need some of my time.

Given that, if you can get them in before a week, we should be good.

+Vinod

> On Feb 5, 2016, at 1:19 PM, Subramaniam V K <subru...@gmail.com> wrote:
> 
> Vinod,
> 
> Thanks for initiating the 2.8 release thread. We are in late review stages
> for YARN-4420 (Add REST API for listing reservations) and YARN-2575 (Adding
> ACLs for reservation system), hoping to get them by next week. Any chance
> you can put off cutting 2.8 by a week as we are planning to deploy
> ReservationSystem and these are critical for that?
> 
> Cheers,
> Subru
> 
> On Thu, Feb 4, 2016 at 3:17 PM, Chris Nauroth <cnaur...@hortonworks.com>
> wrote:
> 
>> FYI, I've just needed to raise HDFS-9761 to blocker status for the 2.8.0
>> release.
>> 
>> --Chris Nauroth
>> 
>> 
>> 
>> 
>> On 2/3/16, 6:19 PM, "Karthik Kambatla" <ka...@cloudera.com> wrote:
>> 
>>> Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me.
>>> Let us not call it alpha or beta though, it is quite confusing. :)
>>> 
>>> On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma <uma.ganguma...@intel.com
>>> 
>>> wrote:
>>> 
>>>> Thanks Vinod. +1 for 2.8 release start.
>>>> 
>>>> Regards,
>>>> Uma
>>>> 
>>>> On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" <vino...@apache.org>
>>>> wrote:
>>>> 
>>>>> Seems like all the features listed in the Roadmap wiki are in. I¹m
>>>> going
>>>>> to try cutting an RC this weekend for a first/non-stable release off of
>>>>> branch-2.8.
>>>>> 
>>>>> Let me know if anyone has any objections/concerns.
>>>>> 
>>>>> Thanks
>>>>> +Vinod
>>>>> 
>>>>>> On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli
>>>>>> <vino...@apache.org> wrote:
>>>>>> 
>>>>>> Branch-2.8 is created.
>>>>>> 
>>>>>> As mentioned before, the goal on branch-2.8 is to put improvements /
>>>>>> fixes to existing features with a goal of converging on an alpha
>>>> release
>>>>>> soon.
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>> 
>>>>>>> On Nov 25, 2015, at 5:30 PM, Vinod Kumar Vavilapalli
>>>>>>> <vino...@apache.org> wrote:
>>>>>>> 
>>>>>>> Forking threads now in order to track all things related to the
>>>>>>> release.
>>>>>>> 
>>>>>>> Creating the branch now.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> +Vinod
>>>>>>> 
>>>>>>> 
>>>>>>>> On Nov 25, 2015, at 11:37 AM, Vinod Kumar Vavilapalli
>>>>>>>> <vino...@apache.org> wrote:
>>>>>>>> 
>>>>>>>> I think we¹ve converged at a high level w.r.t 2.8. And as I just
>>>> sent
>>>>>>>> out an email, I updated the Roadmap wiki reflecting the same:
>>>>>>>> https://wiki.apache.org/hadoop/Roadmap
>>>>>>>> <https://wiki.apache.org/hadoop/Roadmap>
>>>>>>>> 
>>>>>>>> I plan to create a 2.8 branch EOD today.
>>>>>>>> 
>>>>>>>> The goal for all of us should be to restrict improvements & fixes
>>>> to
>>>>>>>> only (a) the feature-set documented under 2.8 in the RoadMap wiki
>>>> and
>>>>>>>> (b) other minor features that are already in 2.8.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> +Vinod
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Nov 11, 2015, at 12:13 PM, Vinod Kumar Vavilapalli
>>>>>>>>> <vino...@hortonworks.com <mailto:vino...@hortonworks.com>> wrote:
>>>>>>>>> 
>>>>>>>>> - Cut a branch about two weeks from now
>>>>>>>>> - Do an RC mid next month (leaving ~4weeks since branch-cut)
>>>>>>>>> - As with 2.7.x series, the first release will still be called as
>>>>>>>>> early / alpha release in the interest of
>>>>>>>>>  ‹ gaining downstream adoption
>>>>>>>>>  ‹ wider testing,
>>>>>>>>>  ‹ yet reserving our right to fix any inadvertent
>>>> incompatibilities
>>>>>>>>> introduced.
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
>> 

[jira] [Resolved] (MAPREDUCE-6566) Add retry support to mapreduce CLI tool

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved MAPREDUCE-6566.

  Resolution: Fixed
Hadoop Flags: Reviewed

[~suda], this JIRA went on for a while, I am going to close this one as fixed. 
Please file a follow up ticket for the test flakiness. Tx.

> Add retry support to mapreduce CLI tool
> ---
>
> Key: MAPREDUCE-6566
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6566
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6566.001.patch, MAPREDUCE-6566.002.patch
>
>
> MAPREDUCE-6251 added support for retries to JobClient. However the MR CLI 
> class doesn't use the JobClient. It would be useful to add support for 
> retries to the CLI class as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [RESULT][VOTE] Release Apache Hadoop 2.7.2 RC2

[Vinod Kumar Vavilapalli]

Done with all the release-activities, the mirrors came back too. Sending out an 
announcement.

Thanks
+Vinod

> On Jan 25, 2016, at 9:19 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Here’s my +1 to conclude the vote.
> 
> With 12 binding +1s, 11 non-binding +1s and no -1s, this vote passes.
> 
> I’ll push forward the release process.
> 
> Thanks to everyone who voted.
> 
> +Vinod
> 
>> On Jan 14, 2016, at 8:57 PM, Vinod Kumar Vavilapalli <vino...@apache.org 
>> <mailto:vino...@apache.org>> wrote:
>> 
>> Hi all,
>> 
>> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.1.
>> 
>> The RC is available for validation at: 
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/ 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/>
>> 
>> The RC tag in git is: release-2.7.2-RC2
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1027 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1027>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at 
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>> your quick perusal.
>> 
>> As you may have noted,
>>  - I terminated the RC1 related voting thread after finding out that we 
>> didn’t have a bunch of patches that are already in the released 2.6.3 
>> version. After a brief discussion, we decided to keep the parallel 2.6.x and 
>> 2.7.x releases incremental, see [4] for this discussion.
>>  - The RC0 related voting thread got halted due to some critical issues. It 
>> took a while again for getting all those blockers out of the way. See the 
>> previous voting thread [3] for details.
>>  - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite 
>> a bit. This release's related discussion threads are linked below: [1] and 
>> [2].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
>> <http://markmail.org/message/oozq3gvd4nhzsaes>
>> [2]: Planning Apache Hadoop 2.7.2 
>> http://markmail.org/message/iktqss2qdeykgpqk 
>> <http://markmail.org/message/iktqss2qdeykgpqk>
>> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
>> http://markmail.org/message/5txhvr2qdiqglrwc 
>> <http://markmail.org/message/5txhvr2qdiqglrwc>
>> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
>> http://markmail.org/thread/n7ljbsnquihn3wlw 
>> <http://markmail.org/thread/n7ljbsnquihn3wlw>

Re: [DISCUSSION] Release Plan for Apache Hadoop 2.6.4

[Vinod Kumar Vavilapalli]

I just pushed out 2.7.2 to the public.

Let’s get 2.6.4 back into focus now that 2.7.2 is done.

In the mean while, I’ll get go dust off 2.8.0 too ..

Thanks
+Vinod


> On Jan 6, 2016, at 3:59 PM, Junping Du  wrote:
> 
> Hello folks,
>   Hope everyone had a wonderful holiday and good rest in passed weeks. We 
> just had a 2.6.3 release on Dec.17 and now it is a good time to look forward 
> to the next branch-2.6 maintenance release - 2.6.4 in the beginning of new 
> year. As usual, this release will be based on latest release 2.6.3 but 
> include a couple of critical/blocker bug fixes that identified in community 
> recently.
> 
> Things Done/TODO:
> 
>  + Schedule
>-- According to previous email discussion thread in community, we are 
> prefering fast-moving way (roughly as monthly) of releasing fixes on 
> branch-2.6. So I would expect this release could happen around end of Jan. or 
> early of Feb. dependens on if RC & VOTE process going on smoothly. Also, it 
> should be after release 2.7.2 to get rid of any possible disturb to that 
> release.
> 
>  + Branch
>-- 2.6.4 will be based on branch-2.6, which has been open to 2.6.4 commits 
> for a while. I plan to branch out branch-2.6.4 sometime next week according 
> to status of target/fixed fixes.
>-- Before 2.6.4 branch is cut off, new commits coming as fix to 2.6.4 is 
> good to commit/merge to trunk, branch-2 and branch-2.6.
> 
>  + Patches
>-- In last couple of days, I went through all critial/blocker bug fixes in 
> the coming 2.7.2 release and ping authors/committers on JIRA to ask for the 
> same effort on branch-2.6.4. I could do more rounds of the same practice in 
> case I could miss something helpful. Also, a kindly reminder from original 
> author and committer is also appreciated.
>-- So far, 2.6.4 already has 18 fixes as list of [1] shows. However, there 
> are totally 55 fixes target for 2.6.4 for now [2], so there is still a large 
> gap and we need to work harder to resolve these tickets as many as we can and 
> push out some non-critical ones to meet our promise of date.
> 
>  + Release
>-- Still too early to do anything directly related to RC. Will work on 
> resolve fixes in list of [2] first, and appreciate any help on this effort.
> 
> Any thoughts? Especially on schedule and list of patches that should be 
> included?
> 
> 
> Cheers,
> 
> Junping
> 
> [1] Tickets fixed in 2.6.4:
> http://s.apache.org/vJB
> 
> [2] Tickets targets for 2.6.4:
> http://s.apache.org/JpE
> 
> 
> 
> From: Junping Du
> Sent: Thursday, December 17, 2015 1:36 PM
> To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Cc: hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> junping...@apache.org
> Subject: [RESULT] [VOTE] Release Apache Hadoop 2.6.3 RC0
> 
> Hi all in community,
> I give my binding +1 to close the vote for 2.6.3 RC0. With 18 +1s (10 
> binding), one +0 (non-binding) and no -1s, the vote passes.
> 
> Thanks for everyone who tried the release candidate and voted. Thanks Vinod, 
> Sangjin and Steve to provide me advices on release process.
> 
> I'll push the release bits and send out an announcement for 2.6.3 soon.
> 
> Cheers,
> 
> Junping
> 
> From: Xuan Gong 
> Sent: Thursday, December 17, 2015 12:29 AM
> To: common-...@hadoop.apache.org
> Cc: hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; junping...@apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.6.3 RC0
> 
> +1 (binding),
> 
> Build and deploy the cluster from source code.
> Ran a few example jobs and passed successfully.
> 
> Xuan Gong
> 
>> On Dec 16, 2015, at 4:07 PM, Arpit Agarwal  wrote:
>> 
>> +1 (binding)
>> 
>> - Verified signatures for source and binary distributions
>> - Built jars from source with java 1.7.0_79
>> - Deployed single-node pseudo-cluster
>> - Ran example map reduce jobs
>> - Ran hdfs admin commands, verified NN web UI shows expected usages
>> 
>> 
>> 
>> On 12/11/15, 4:16 PM, "Junping Du"  wrote:
>> 
>>> 
>>> Hi all developers in hadoop community,
>>> I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next 
>>> maintenance release to follow up 2.6.2.) according to email thread of 
>>> release plan 2.6.3 [1]. Sorry for this RC coming a bit late as several 
>>> blocker issues were getting committed until yesterday. Below is the details:
>>> 
>>> The RC is available for validation at:
>>> *http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
>>> *
>>> 
>>> The RC tag in git is: release-2.6.3-RC0
>>> 
>>> The maven artifacts are staged via repository.apache.org at:
>>> *https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
>>> *

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Vinod Kumar Vavilapalli]

Eric, is there a JIRA for this?

+Vinod

> On Jan 19, 2016, at 2:27 PM, Eric Payne  
> wrote:
> 
> One minor issue is that when the job was running on the labelled queue, it 
> ran fine but the cluster scheduler web UI showed a blank status bar for that 
> queue in the "Application Queues" section.

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Vinod Kumar Vavilapalli]

Thanks for the retraction, Jason!

+Vinod


> On Jan 19, 2016, at 3:30 PM, Jason Lowe <jl...@yahoo-inc.com.INVALID> wrote:
> 
> That's reasonable, especially if we don't take nearly as long for 2.7.3.  
> Note that there are almost 50 JIRAs already committed to 2.7.3, so hopefully 
> we'll have a plan for that soon.
> +1 (binding) for 2.7.2 RC2.
> Jason
> 
> 
>  From: Vinod Kumar Vavilapalli <vino...@apache.org>
> To: mapreduce-dev@hadoop.apache.org; Jason Lowe <jl...@yahoo-inc.com> 
> Cc: Hadoop Common <common-...@hadoop.apache.org>; 
> "hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; 
> "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>
> Sent: Tuesday, January 19, 2016 5:25 PM
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC2
> 
> The JIRA YARN-4610 links YARN-3434 as the one causing the breakage, and 
> YARN-3434 already exists in 2.7.1 itself. That categorizes the new issue as 
> an existing bug.
> If you agree with that sentiment, and given that there is a clear 
> work-around, in the interest of progress of 2.7.2 (we have spent > 2 months 
> on this now), I’d like to move forward.
> Please LMK what you think.
> Thanks+Vinod
> 
> 
> On Jan 19, 2016, at 3:13 PM, Jason Lowe <jl...@yahoo-inc.com.INVALID> wrote:
> -1 (binding)
> We have been running a release derived from 2.7 on some of our clusters, and 
> we recently hit a bug where an application making large container requests 
> can drastically slow down container allocations for other users in the same 
> queue.  See YARN-4610 for details.  Since 
> yarn.scheduler.capacity.reservations-continue-look-all-nodes is on by 
> default, I think we should fix this.  If we decide to ship 2.7.2 without that 
> fix then the release notes should call out that JIRA and mention the 
> workaround of setting 
> yarn.scheduler.capacity.reservations-continue-look-all-nodes to false.
> Jason
> 
> 
>  From: Vinod Kumar Vavilapalli <vino...@apache.org>
> To: Hadoop Common <common-...@hadoop.apache.org>; hdfs-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org 
> Sent: Thursday, January 14, 2016 10:57 PM
> Subject: [VOTE] Release Apache Hadoop 2.7.2 RC2
> 
> Hi all,
> 
> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.1.
> 
> The RC is available for validation at: 
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
> 
> The RC tag in git is: release-2.7.2-RC2
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1027 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1027>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
> your quick perusal.
> 
> As you may have noted,
> - I terminated the RC1 related voting thread after finding out that we didn’t 
> have a bunch of patches that are already in the released 2.6.3 version. After 
> a brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
> incremental, see [4] for this discussion.
> - The RC0 related voting thread got halted due to some critical issues. It 
> took a while again for getting all those blockers out of the way. See the 
> previous voting thread [3] for details.
> - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
> bit. This release's related discussion threads are linked below: [1] and [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
> <http://markmail.org/message/oozq3gvd4nhzsaes>
> [2]: Planning Apache Hadoop 2.7.2 
> http://markmail.org/message/iktqss2qdeykgpqk 
> <http://markmail.org/message/iktqss2qdeykgpqk>
> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
> http://markmail.org/message/5txhvr2qdiqglrwc 
> <http://markmail.org/message/5txhvr2qdiqglrwc>
> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
> http://markmail.org/thread/n7ljbsnquihn3wlw
> 
> 
> 
> 
> 

[RESULT][VOTE] Release Apache Hadoop 2.7.2 RC2

[Vinod Kumar Vavilapalli]

Here’s my +1 to conclude the vote.

With 12 binding +1s, 11 non-binding +1s and no -1s, this vote passes.

I’ll push forward the release process.

Thanks to everyone who voted.

+Vinod

> On Jan 14, 2016, at 8:57 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Hi all,
> 
> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.1.
> 
> The RC is available for validation at: 
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/ 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/>
> 
> The RC tag in git is: release-2.7.2-RC2
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1027 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1027>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
> your quick perusal.
> 
> As you may have noted,
>  - I terminated the RC1 related voting thread after finding out that we 
> didn’t have a bunch of patches that are already in the released 2.6.3 
> version. After a brief discussion, we decided to keep the parallel 2.6.x and 
> 2.7.x releases incremental, see [4] for this discussion.
>  - The RC0 related voting thread got halted due to some critical issues. It 
> took a while again for getting all those blockers out of the way. See the 
> previous voting thread [3] for details.
>  - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite 
> a bit. This release's related discussion threads are linked below: [1] and 
> [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
> <http://markmail.org/message/oozq3gvd4nhzsaes>
> [2]: Planning Apache Hadoop 2.7.2 
> http://markmail.org/message/iktqss2qdeykgpqk 
> <http://markmail.org/message/iktqss2qdeykgpqk>
> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
> http://markmail.org/message/5txhvr2qdiqglrwc 
> <http://markmail.org/message/5txhvr2qdiqglrwc>
> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
> http://markmail.org/thread/n7ljbsnquihn3wlw 
> <http://markmail.org/thread/n7ljbsnquihn3wlw>

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Vinod Kumar Vavilapalli]

Here’s my +1 to conclude the vote.

With 12 binding +1s, 11 non-binding +1s and no -1s, this vote passes.

I’ll push forward the release process.

Thanks to everyone who voted.

+Vinod

> On Jan 14, 2016, at 8:57 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Hi all,
> 
> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.1.
> 
> The RC is available for validation at: 
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
> 
> The RC tag in git is: release-2.7.2-RC2
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1027 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1027>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
> your quick perusal.
> 
> As you may have noted,
> - I terminated the RC1 related voting thread after finding out that we didn’t 
> have a bunch of patches that are already in the released 2.6.3 version. After 
> a brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
> incremental, see [4] for this discussion.
> - The RC0 related voting thread got halted due to some critical issues. It 
> took a while again for getting all those blockers out of the way. See the 
> previous voting thread [3] for details.
> - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
> bit. This release's related discussion threads are linked below: [1] and [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
> <http://markmail.org/message/oozq3gvd4nhzsaes>
> [2]: Planning Apache Hadoop 2.7.2 
> http://markmail.org/message/iktqss2qdeykgpqk 
> <http://markmail.org/message/iktqss2qdeykgpqk>
> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
> http://markmail.org/message/5txhvr2qdiqglrwc 
> <http://markmail.org/message/5txhvr2qdiqglrwc>
> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
> http://markmail.org/thread/n7ljbsnquihn3wlw

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Vinod Kumar Vavilapalli]

I guess he didn’t fetch the remote correctly.

Thanks for taking care of this everyone. I just deleted brachn-2.8 (misspelled 
branch) to close the loop on this one.

Thanks
+Vinod

> On Jan 15, 2016, at 8:44 AM, Sangjin Lee <sj...@apache.org> wrote:
> 
> I deleted all the obsolete 2928 branches (except for YARN-2928 and 
> feature-YARN-2928) a few days ago. Do you still see them?
> 
> On Fri, Jan 15, 2016 at 5:54 AM, Junping Du <j...@hortonworks.com 
> <mailto:j...@hortonworks.com>> wrote:
> There are also several stale branches for YARN-2928 that we should 
> consolidate:
>   remotes/origin/YARN-2928
>   remotes/origin/YARN-2928-new
>   remotes/origin/YARN-2928-old
>   remotes/origin/YARN-2928-old-2015-11-09
>   remotes/origin/YARN-2928-rebase
>   remotes/origin/feature-YARN-2928  (the latest one)
> I think we should remove all stale ones and rename the latest one 
> feature-YARN-2928 to YARN-2928 to follow our practice for branch development. 
> Thoughts?
> 
> 
> Thanks,
> 
> Junping
> 
> From: sjl...@gmail.com <mailto:sjl...@gmail.com> <sjl...@gmail.com 
> <mailto:sjl...@gmail.com>> on behalf of Sangjin Lee <sj...@apache.org 
> <mailto:sj...@apache.org>>
> Sent: Friday, January 15, 2016 7:17 AM
> To: yarn-...@hadoop.apache.org <mailto:yarn-...@hadoop.apache.org>
> Cc: Hadoop Common; hdfs-...@hadoop.apache.org 
> <mailto:hdfs-...@hadoop.apache.org>; mapreduce-dev@hadoop.apache.org 
> <mailto:mapreduce-dev@hadoop.apache.org>; Vinod Kumar Vavilapalli
> Subject: Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale 
> branches
> 
> Thanks Vinod for the proposal. I deleted my branch (sjlee/hdfs-merge) the
> other day.
> 
> Sangjin
> 
> On Thu, Jan 14, 2016 at 1:26 PM, Vinod Kumar Vavilapalli <vino...@apache.org 
> <mailto:vino...@apache.org>
> > wrote:
> 
> > Hi all,
> >
> > As some of you have noticed, we have an update from ASF infra on git
> > branching policy: We no longer have a ASF wide mandate on disallowing
> > force-pushes on all branches / tags.
> >
> > Summarizing information from the INFRA email for the sake of clarity in
> > the midst of recent confusion
> >  - We now can do force pushes, and branch/tag deletion on any branch or
> > tag except refs/tags/rel
> >  - Any force pushes will be annotated in the commit-email as “[Forced
> > Update!]” for the community to watch out for undesired force-pushes
> >  - Only tags under refs/tags/rel are protected from force-push for the
> > sake of release-provenance: Essentially, the releases that community votes
> > on are archived in their entirety with the development history and we
> > cannot alter that once a tag is created. As one might expect.
> >
> > What this means for us
> >  - Stale branches: There are a few stale branches that got accumulated.
> > — During this branch moratorium, origin/bracnh-2.8 got created (May be
> > as part of HDFS-8785, can’t say for sure)
> > — A couple of stale branches that helped 2.6.1 release:
> > origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
> > — I’ll wait till EOD tomorrow for any yays/nays and delete them
> >  - Feature branch updates: Developers can now go rebase and force-push
> > their feature branches.
> >  - Mainline branches: Mainline branches like trunk, branch-2 have always
> > been configured to avoid force-pushes. In general, force-push continues to
> > be recommended mainly for feature branches and definitely not on any
> > mainline branches from which we make releases.
> >  - Release tags:
> > — To follow ASF provenance policy, we will now push the final release
> > tags under refs/tags/rel. We will first push the RC tags under where they
> > reside now (refs/tags) and if the vote passes, the final tag will be
> > created under refs/tags/rel.
> > — I’ll update our release wiki page
> > http://wiki.apache.org/hadoop/HowToRelease 
> > <http://wiki.apache.org/hadoop/HowToRelease> <
> > http://wiki.apache.org/hadoop/HowToRelease 
> > <http://wiki.apache.org/hadoop/HowToRelease>> with the same details once I
> > can get 2.7.2 release done using this updated process.
> >  - Existing release tags:
> > — There is a general recommendation from INFRA team to take all of our
> > existing release tags under "tags" and copy them to “rel”.
> > — I’ll wait till EOD tomorrow for any yays/nays and copy existing
> > releases under refs/tags/rel following general recommendations.
> >
> > Any comments / thoughts / questions welcome.
> >
> > Thanks
> > +Vinod
> 

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Vinod Kumar Vavilapalli]

I believe this is still in place, though I am not sure how we can verify this 
(without doing a force-push of course)

+Vinod

> One thing that wasn't clear from the INFRA announcement: are trunk,
> branch-* branches protected against force-pushes in the new world? If not,
> should we ask them to be locked up?
> 
> Thanks
> Karthik
> 
> On Thu, Jan 14, 2016 at 10:26 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> 
>> Hi all,
>> 
>> As some of you have noticed, we have an update from ASF infra on git
>> branching policy: We no longer have a ASF wide mandate on disallowing
>> force-pushes on all branches / tags.
>> 
>> Summarizing information from the INFRA email for the sake of clarity in
>> the midst of recent confusion
>> - We now can do force pushes, and branch/tag deletion on any branch or
>> tag except refs/tags/rel
>> - Any force pushes will be annotated in the commit-email as “[Forced
>> Update!]” for the community to watch out for undesired force-pushes
>> - Only tags under refs/tags/rel are protected from force-push for the
>> sake of release-provenance: Essentially, the releases that community votes
>> on are archived in their entirety with the development history and we
>> cannot alter that once a tag is created. As one might expect.
>> 
>> What this means for us
>> - Stale branches: There are a few stale branches that got accumulated.
>>— During this branch moratorium, origin/bracnh-2.8 got created (May be
>> as part of HDFS-8785, can’t say for sure)
>>— A couple of stale branches that helped 2.6.1 release:
>> origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
>>— I’ll wait till EOD tomorrow for any yays/nays and delete them
>> - Feature branch updates: Developers can now go rebase and force-push
>> their feature branches.
>> - Mainline branches: Mainline branches like trunk, branch-2 have always
>> been configured to avoid force-pushes. In general, force-push continues to
>> be recommended mainly for feature branches and definitely not on any
>> mainline branches from which we make releases.
>> - Release tags:
>>— To follow ASF provenance policy, we will now push the final release
>> tags under refs/tags/rel. We will first push the RC tags under where they
>> reside now (refs/tags) and if the vote passes, the final tag will be
>> created under refs/tags/rel.
>>— I’ll update our release wiki page
>> http://wiki.apache.org/hadoop/HowToRelease <
>> http://wiki.apache.org/hadoop/HowToRelease> with the same details once I
>> can get 2.7.2 release done using this updated process.
>> - Existing release tags:
>>— There is a general recommendation from INFRA team to take all of our
>> existing release tags under "tags" and copy them to “rel”.
>>— I’ll wait till EOD tomorrow for any yays/nays and copy existing
>> releases under refs/tags/rel following general recommendations.
>> 
>> Any comments / thoughts / questions welcome.
>> 
>> Thanks
>> +Vinod

[VOTE] Release Apache Hadoop 2.7.2 RC2

[Vinod Kumar Vavilapalli]

Hi all,

I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/

The RC tag in git is: release-2.7.2-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1027 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - I terminated the RC1 related voting thread after finding out that we didn’t 
have a bunch of patches that are already in the released 2.6.3 version. After a 
brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
incremental, see [4] for this discussion.
 - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
 - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 

[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 

[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc 

[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
http://markmail.org/thread/n7ljbsnquihn3wlw

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Vinod Kumar Vavilapalli]

It kind of got lost through my email, but when I talked about force-push, it is 
almost completely in the context of feature-branch rebase.

We almost never do force-push outside of this context, neither do we encourage 
it. And like I pointed, the fact that the mainline branches (trunk, branch-2 
etc) are already protected against force-pushes, and the latest infra’s change 
to notify force-pushes as explicitly marked so, are good enough for handling 
them IMO.

Thanks
+Vinod

> On Jan 14, 2016, at 8:07 PM, Yongjun Zhang  wrote:
> 
> I assume we will have at least a committer review of any force-push. Say,
> if the commit history already has x, y, z, now we need to force push with
> x', we'd better not torget y and z, and if there is conflict putting in y
> and z after x', the conflict is better reviewed.

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Vinod Kumar Vavilapalli]

Mailing infra ate all the formatting up. Here’s the same email with some line 
breaks.


> Hi all,
> 
> As some of you have noticed, we have an update from ASF infra on git 
> branching policy: We no longer have a ASF wide mandate on disallowing 
> force-pushes on all branches / tags.


> Summarizing information from the INFRA email for the sake of clarity in the 
> midst of recent confusion
> - We now can do force pushes, and branch/tag deletion on any branch or tag 
> except refs/tags/rel
> - Any force pushes will be annotated in the commit-email as “[Forced 
> Update!]” for the community to watch out for undesired force-pushes
> - Only tags under refs/tags/rel are protected from force-push for the sake of 
> release-provenance: Essentially, the releases that community votes on are 
> archived in their entirety with the development history and we cannot alter 
> that once a tag is created. As one might expect.
> 


> What this means for us


> - Stale branches: There are a few stale branches that got accumulated.
>— During this branch moratorium, origin/bracnh-2.8 got created (May be as 
> part of HDFS-8785, can’t say for sure)
>— A couple of stale branches that helped 2.6.1 release: 
> origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
>— I’ll wait till EOD tomorrow for any yays/nays and delete them


> - Feature branch updates: Developers can now go rebase and force-push their 
> feature branches.


> - Mainline branches: Mainline branches like trunk, branch-2 have always been 
> configured to avoid force-pushes. In general, force-push continues to be 
> recommended mainly for feature branches and definitely not on any mainline 
> branches from which we make releases.


> - Release tags:
>— To follow ASF provenance policy, we will now push the final release tags 
> under refs/tags/rel. We will first push the RC tags under where they reside 
> now (refs/tags) and if the vote passes, the final tag will be created under 
> refs/tags/rel.
>— I’ll update our release wiki page 
> http://wiki.apache.org/hadoop/HowToRelease 
>  with the same details once I can 
> get 2.7.2 release done using this updated process.


> - Existing release tags:
>— There is a general recommendation from INFRA team to take all of our 
> existing release tags under "tags" and copy them to “rel”.
>— I’ll wait till EOD tomorrow for any yays/nays and copy existing releases 
> under refs/tags/rel following general recommendations.
> 
> Any comments / thoughts / questions welcome.
> 
> Thanks
> +Vinod

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Vinod Kumar Vavilapalli]

Thanks for the comments Tsuyoshi / Andrew / Varun / Junping / Akira.

I agree that where possible we should serialize releases and make them 
incremental w.r.t fixes.

Will roll a new RC for 2.7.2 after the backports.

If there are more thoughts on releases, which I’m sure we will all do have, 
let’s fork the thread off from this voting conversation.

Thanks
+Vinod

> On Dec 29, 2015, at 6:50 AM, Junping Du <j...@hortonworks.com> wrote:
> 
> I am +1 with pulling all of these tickets into 2.7.2. 
> - For “any fix in 2.6.3 to be there in all releases that get out after 2.6.3 
> release date”
> Shall we conclude this as a general rule - "any fix in 2.x.y to be there in 
> all 2.b.c releases (while b>=x) that get out after 2.x.y release date"? I am 
> generally fine with this, but just feel it sounds to set too strong 
> restrictions among branches. Some fixes could be trivial (test case fix, 
> etc.) enough to deserve more flexibility.​ I would prefer this rule only 
> applies on critical/blocker fixes, but not applies on minor/trivial issues.
> Just 2 cents.
> 
> Thanks,
> 
> Junping
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>>
> Sent: Thursday, December 24, 2015 12:47 AM
> To: Junping Du
> Cc: mapreduce-dev@hadoop.apache.org <mailto:mapreduce-dev@hadoop.apache.org>; 
> yarn-...@hadoop.apache.org <mailto:yarn-...@hadoop.apache.org>; 
> common-...@hadoop.apache.org <mailto:common-...@hadoop.apache.org>; 
> hdfs-...@hadoop.apache.org <mailto:hdfs-...@hadoop.apache.org>
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC1
>  
> I retract my -1. I think we will need to discuss this a bit more.
> 
> Beyond those two tickets, there are a bunch more (totaling to 16) that are in 
> 2.6.3 but *not* in 2.7.2. See this: 
> https://issues.apache.org/jira/issues/?jql=key%20in%20%28HADOOP-12526%2CHADOOP-12413%2CHADOOP-11267%2CHADOOP-10668%2CHADOOP-10134%2CYARN-4434%2CYARN-4365%2CYARN-4348%2CYARN-4344%2CYARN-4326%2CYARN-4241%2CYARN-2859%2CMAPREDUCE-6549%2CMAPREDUCE-6540%2CMAPREDUCE-6377%2CMAPREDUCE-5883%2CHDFS-9431%2CHDFS-9289%2CHDFS-8615%29%20and%20fixVersion%20!%3D%202.7.0
>  
> <https://issues.apache.org/jira/issues/?jql=key%20in%20(HADOOP-12526,HADOOP-12413,HADOOP-11267,HADOOP-10668,HADOOP-10134,YARN-4434,YARN-4365,YARN-4348,YARN-4344,YARN-4326,YARN-4241,YARN-2859,MAPREDUCE-6549,MAPREDUCE-6540,MAPREDUCE-6377,MAPREDUCE-5883,HDFS-9431,HDFS-9289,HDFS-8615)%20and%20fixVersion%20!=%202.7.0>
> 
> Two options here, depending on the importance of ‘causality' between 2.6.x 
> and 2.7.x lines.
>  - Ship 2.7.2 as we voted on here
>  - Pull these 16 tickets into 2.7.2 and roll a new RC
> 
> What do people think? Do folks expect “any fix in 2.6.3 to be there in all 
> releases that get out after 2.6.3 release date (December 16th)”?
> 
> Thanks
> +Vinod
> 
>> On Dec 23, 2015, at 12:37 PM, Vinod Kumar Vavilapalli <vino...@apache.org 
>> <mailto:vino...@apache.org>> wrote:
>> 
>> Sigh. Missed this.
>> 
>> To retain causality ("any fix in 2.6.3 will be there in all releases that 
>> got out after 2.6.3”), I’ll get these patches in.
>> 
>> Reverting my +1, and casting -1 for the RC myself.
>> 
>> Will spin a new RC, this voting thread is marked dead.
>> 
>> Thanks
>> +Vinod
>> 
>>> On Dec 22, 2015, at 8:24 AM, Junping Du <j...@hortonworks.com 
>>> <mailto:j...@hortonworks.com>> wrote:
>>> 
>>> However, when I look at our commit log and CHANGES.txt, I found something 
>>> we are missing:
>>> 1. HDFS-9470 and YARN-4424 are missing from the 2.7.2 branch and RC1 tag.
>>> 2. HADOOP-5323, HDFS-8767 are missing in CHANGE.txt
>> 
> 
>