Re: [DISCUSS] Enable github security notifications to all Hadoop committers

[Vinod Kumar Vavilapalli]

Shouldn’t we instead send this to the security list and let that team make 
decisions first there?

What volume are we looking at?

Thanks
+Vinod

> On Oct 24, 2019, at 12:07 PM, Wei-Chiu Chuang  wrote:
> 
> Hi,
> I raised INFRA-19327  to
> enable github security notification. How do people feel if we enable this
> notification to all committers? I already have hundreds of incoming emails
> from various Hadoop alias each day so I don't mind adding a few more.
> 
> If there is no good consensus, then we'll have to let committers to opt-in
> individually.
> 
> Thanks,
> Weichiu


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree [discussion -> lazy vote]

[Vinod Kumar Vavilapalli]

On Mon, Sep 23, 2019 at 4:02 PM Elek, Marton  wrote:

>  As the overall feedback was positive (in fact many of the answers were
> simple +1 votes) I don't think the thread should be repeated under
> [VOTE] subject. Therefore I call it for a lazy consensus.
>

Let's please not do this in the future.

These are large enough changes that we'd like to get formal votes both for
immediate visibility (VOTE threads are more in your face) as well as for
record-keeping in posterity.

Thanks
+Vinod

Re: VOTE PASSED - Hadoop-3.1.3-RC0 Re: [VOTE] Release Hadoop-3.1.3-RC0

[Vinod Kumar Vavilapalli]

Zhankun et.al,

Did we close down the rest of the release process for 3.1.3?

I don't yet see this on the site, so asking.

If it's not done yet and you are running into issues, please let me know
and I can help with the corresponding release management tasks.

Thanks
+Vinod

On Tue, Sep 24, 2019 at 2:16 PM Zhankun Tang  wrote:

> Hi,
>
> Thank you all who spent time to verify and vote 3.1.3 release!
> I‘m pleased to announce that the vote for 3.1.3 RC0 passed! I'll push the
> release bits and send out an announcement for 3.1.3 soon.
>
> Summary of votes for Hadoop-3.1.3-RC0:
>
> 4 binding +1s, from Rohith Sharma K S, Sunil Govindan, Weiwei Yang, Eric
> Payne
>
> 3 non-binding +1s, from Xun Liu, Zac Zhou, Zhankun Tang
>
> 1 non-binding with +0s from Masatake Iwasaki
>
> and *no -1s*.
>
> BR,
> Zhankun
>
> On Tue, 24 Sep 2019 at 15:40, zac yuan  wrote:
>
> > Looking forward to 3.1.3 release. Thanks Zhankun for your great effort~
> >
> > +1 (non-binding)
> >
> > Zac Zhou
> >
> > Xun Liu  于2019年9月23日周一 上午11:49写道：
> >
> > > +1
> > >
> > > Thanks Zhankun for all of your hard work on this release.
> > >
> > >
> > > > On Sep 20, 2019, at 5:34 AM, epa...@apache.org wrote:
> > > >
> > > >
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks Zhankun for all of your hard work on this release.
> > > >
> > > > I downloaded and built the source and ran it on an insecure
> multi-node
> > > pseudo cluster.
> > > >
> > > > I performed various YARN manual tests, including creating custom
> > > resources, creating queue submission ACLs, and queue refreshes.
> > > >
> > > > One concern is that preemption does not seem to be working when only
> > the
> > > custom resources are over the queue capacity, but I don't think this is
> > > something introduced with this release.
> > > >
> > > > -Eric
> > > >
> > > >
> > > >
> > > > On Thursday, September 12, 2019, 3:04:44 AM CDT, Zhankun Tang <
> > > zt...@apache.org> wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi folks,
> > > >
> > > > Thanks to everyone's help on this release. Special thanks to Rohith,
> > > > Wei-Chiu, Akira, Sunil, Wangda!
> > > >
> > > > I have created a release candidate (RC0) for Apache Hadoop 3.1.3.
> > > >
> > > > The RC release artifacts are available at:
> > > > http://home.apache.org/~ztang/hadoop-3.1.3-RC0/
> > > >
> > > > The maven artifacts are staged at:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1228/
> > > >
> > > > The RC tag in git is here:
> > > > https://github.com/apache/hadoop/tree/release-3.1.3-RC0
> > > >
> > > > And my public key is at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.*
> > > >
> > > > For the testing, I have run several Spark and distributed shell jobs
> in
> > > my
> > > > pseudo cluster.
> > > >
> > > > My +1 (non-binding) to start.
> > > >
> > > > BR,
> > > > Zhankun
> > > >
> > > > On Wed, 4 Sep 2019 at 15:56, zhankun tang 
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Thanks for everyone helping in resolving all the blockers targeting
> > > Hadoop
> > > >> 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers
> > > issues
> > > >> to 3.1.4.
> > > >>
> > > >> I'll cut the branch today and call a release vote soon. Thanks!
> > > >>
> > > >>
> > > >> [1]. https://s.apache.org/5hj5i
> > > >>
> > > >> BR,
> > > >> Zhankun
> > > >>
> > > >>
> > > >> On Wed, 21 Aug 2019 at 12:38, Zhankun Tang 
> wrote:
> > > >>
> > > >>> Hi folks,
> > > >>>
> > > >>> We have Apache Hadoop 3.1.2 released on Feb 2019.
> > > >>>
> > > >>> It's been more than 6 months passed and there're
> > > >>>
> > > >>> 246 fixes[1]. 2 blocker and 4 critical Issues [2]
> > > >>>
> > > >>> (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker)
> > > >>>
> > > >>>
> > > >>> I propose my plan to do a maintenance release of 3.1.3 in the next
> > few
> > > >>> (one or two) weeks.
> > > >>>
> > > >>> Hadoop 3.1.3 release plan:
> > > >>>
> > > >>> Code Freezing Date: *25th August 2019 PDT*
> > > >>>
> > > >>> Release Date: *31th August 2019 PDT*
> > > >>>
> > > >>>
> > > >>> Please feel free to share your insights on this. Thanks!
> > > >>>
> > > >>>
> > > >>> [1] https://s.apache.org/zw8l5
> > > >>>
> > > >>> [2] https://s.apache.org/fjol5
> > > >>>
> > > >>>
> > > >>> BR,
> > > >>>
> > > >>> Zhankun
> > > >>>
> > > >>
> > > >
> > > > -
> > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: Does VOTE necessary to create a child repo?

[Vinod Kumar Vavilapalli]

Moving the thread to the dev lists.

Thanks
+Vinod

> On Sep 23, 2019, at 11:43 PM, Vinayakumar B  wrote:
> 
> Thanks Marton,
> 
> Current created 'hadoop-thirdparty' repo is empty right now.
> Whether to use that repo  for shaded artifact or not will be monitored in
> HADOOP-13363 umbrella jira. Please feel free to join the discussion.
> 
> There is no existing codebase is being moved out of hadoop repo. So I think
> right now we are good to go.
> 
> -Vinay
> 
> On Mon, Sep 23, 2019 at 11:38 PM Marton Elek  wrote:
> 
>> 
>> I am not sure if it's defined when is a vote required.
>> 
>> https://www.apache.org/foundation/voting.html
>> 
>> Personally I think it's a big enough change to send a notification to the
>> dev lists with a 'lazy consensus'  closure
>> 
>> Marton
>> 
>> On 2019/09/23 17:46:37, Vinayakumar B  wrote:
>>> Hi,
>>> 
>>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be more in
>> future)
>>> will be kept as a shaded artifact in a separate repo, which will be
>>> referred as dependency in hadoop modules.  This approach avoids shading
>> of
>>> every submodule during build.
>>> 
>>> So question is does any VOTE required before asking to create a git repo?
>>> 
>>> On selfserve platform https://gitbox.apache.org/setup/newrepo.html
>>> I can access see that, requester should be PMC.
>>> 
>>> Wanted to confirm here first.
>>> 
>>> -Vinay
>>> 
>> 
>> -
>> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: private-h...@hadoop.apache.org
>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.2.1 release

[Vinod Kumar Vavilapalli]

Done: https://twitter.com/hadoop/status/1176787511865008128.

If you have tweetdeck, any of the PMC members can do this.

BTW, it looks we haven't published any releases since Nov 2018. Let's get back 
to doing this going forward!

Thanks
+Vinod

> On Sep 25, 2019, at 2:44 PM, Rohith Sharma K S  
> wrote:
> 
> Updated twitter message:
> 
> ``
> Apache Hadoop 3.2.1 is released: https://s.apache.org/96r4h
> 
> Announcement: https://s.apache.org/jhnpe
> Overview: https://s.apache.org/tht6a
> Changes: https://s.apache.org/pd6of
> Release notes: https://s.apache.org/ta50b
> 
> Thanks to our community of developers, operators, and users.
> 
> 
> -Rohith Sharma K S
> 
> 
> On Wed, 25 Sep 2019 at 14:15, Sunil Govindan  wrote:
> 
>> Here the link of Overview URL is old.
>> We should ideally use https://hadoop.apache.org/release/3.2.1.html
>> 
>> Thanks
>> Sunil
>> 
>> On Wed, Sep 25, 2019 at 2:10 PM Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>> 
>>> Can someone help to post this in twitter account?
>>> 
>>> Apache Hadoop 3.2.1 is released: https://s.apache.org/mzdb6
>>> Overview: https://s.apache.org/tht6a
>>> Changes: https://s.apache.org/pd6of
>>> Release notes: https://s.apache.org/ta50b
>>> 
>>> Thanks to our community of developers, operators, and users.
>>> 
>>> -Rohith Sharma K S
>>> 
>>> On Wed, 25 Sep 2019 at 13:44, Rohith Sharma K S <
>>> rohithsharm...@apache.org> wrote:
>>> 
 Hi all,
 
It gives us great pleasure to announce that the Apache Hadoop
 community has
 voted to release Apache Hadoop 3.2.1.
 
 Apache Hadoop 3.2.1 is the stable release of Apache Hadoop 3.2 line,
 which
 includes 493 fixes since Hadoop 3.2.0 release:
 
 - For major changes included in Hadoop 3.2 line, please refer Hadoop
 3.2.1 main page[1].
 - For more details about fixes in 3.2.1 release, please read
 CHANGELOG[2] and RELEASENOTES[3].
 
 The release news is posted on the Hadoop website too, you can go to the
 downloads section directly[4].
 
 Thank you all for contributing to the Apache Hadoop!
 
 Cheers,
 Rohith Sharma K S
 
 
 [1] https://hadoop.apache.org/docs/r3.2.1/index.html
 [2]
 https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/CHANGELOG.3.2.1.html
 [3]
 https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/RELEASENOTES.3.2.1.html
 [4] https://hadoop.apache.org
 
>>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinod Kumar Vavilapalli]

Quick question, being lazy here, lots of JIRA updates on HADOOP-13363 over the 
years not helping either.

Does anyone know what this upgrade will mean w.r.t compatibility for the Hadoop 
releases themselves? Remember that trunk is still 3.x.

Thanks
+Vinod

> On Sep 21, 2019, at 9:55 AM, Vinayakumar B  wrote:
> 
> @Wangda Tan  ,
> Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple
> stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, plugin
> upgrade, shading, etc).
> Right now, first task of jar upgrade is done. So need to update the protoc
> executable in the in build environments.
> 
> @张铎(Duo Zhang)  ,
> Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde
> was possible. Sorry I missed it.
> 
> Plugin update needed to be done for whole project, for which precommit
> jenkins will need more time complete end-to-end runs.
> So plugin update is planned in stages in further subtasks. It could be done
> in 2-3 days.
> 
> -Vinay
> 
> On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang),  wrote:
> 
>> I think this one is alread in place so we have to upgrade...
>> 
>> https://issues.apache.org/jira/browse/HADOOP-16557
>> 
>> Wangda Tan  于2019年9月21日周六 上午7:19写道：
>> 
>>> Hi Vinay,
>>> 
>>> A bit confused, I saw the HADOOP-13363 is still pending. Do we need to
>>> upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed?
>>> 
>>> Thanks,
>>> Wangda
>>> 
>>> On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B 
>>> wrote:
>>> 
 Hi All,
 
 A very long pending task, protobuf upgrade is happening in
>> HADOOP-13363.
>>> As
 part of that protobuf version is upgraded to 3.7.1.
 
 Please update your build environments to have 3.7.1 protobuf version.
 
 BUILIDING.txt has been updated with latest instructions.
 
 This pre-requisite to update protoc dependecy manually is required
>> until
 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
 dynamically resolve required protoc exe.
 
 Dockerfile is being updated to have latest 3.7.1 as default protoc for
>>> test
 environments.
 
 Thanks,
 -Vinay
 
>>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Vinod Kumar Vavilapalli]

Looks to me that the advantages of this additional step are only incremental 
given that you've already decoupled releases and dependencies.

Do you see a Submarine like split-also-into-a-TLP for Ozone? If not now, 
sometime further down the line? If so, why not do both at the same time? I felt 
the same way with Submarine, but couldn't follow up in time.

Thanks
+Vinod

> On Sep 18, 2019, at 4:04 AM, Wangda Tan  wrote:
> 
> +1 (binding).
> 
> From my experiences of Submarine project, I think moving to a separate repo
> helps.
> 
> - Wangda
> 
> On Tue, Sep 17, 2019 at 11:41 AM Subru Krishnan  wrote:
> 
>> +1 (binding).
>> 
>> IIUC, there will not be an Ozone module in trunk anymore as that was my
>> only concern from the original discussion thread? IMHO, this should be the
>> default approach for new modules.
>> 
>> On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
>> slamendo...@bloomberg.net> wrote:
>> 
>>> +1
>>> 
>>> From: e...@apache.org At: 09/17/19 05:48:32To:
>> hdfs-...@hadoop.apache.org,
>>> mapreduce-...@hadoop.apache.org,  common-dev@hadoop.apache.org,
>>> yarn-...@hadoop.apache.org
>>> Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
>>> source tree
>>> 
>>> 
>>> TLDR; I propose to move Ozone related code out from Hadoop trunk and
>>> store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
>>> 
>>> 
>>> When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
>>> be part of the source tree but with separated release cadence, mainly
>>> because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
>>> 
>>> During the last Ozone releases this dependency is removed to provide
>>> more stable releases. Instead of using the latest trunk/SNAPSHOT build
>>> from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
>>> 
>>> As we have no more strict dependency between Hadoop trunk SNAPSHOT and
>>> Ozone trunk I propose to separate the two code base from each other with
>>> creating a new Hadoop git repository (apache/hadoop-ozone.git):
>>> 
>>> With moving Ozone to a separated git repository:
>>> 
>>>  * It would be easier to contribute and understand the build (as of now
>>> we always need `-f pom.ozone.xml` as a Maven parameter)
>>>  * It would be possible to adjust build process without breaking
>>> Hadoop/Ozone builds.
>>>  * It would be possible to use different Readme/.asf.yaml/github
>>> template for the Hadoop Ozone and core Hadoop. (For example the current
>>> github template [2] has a link to the contribution guideline [3]. Ozone
>>> has an extended version [4] from this guideline with additional
>>> information.)
>>>  * Testing would be more safe as it won't be possible to change core
>>> Hadoop and Hadoop Ozone in the same patch.
>>>  * It would be easier to cut branches for Hadoop releases (based on the
>>> original consensus, Ozone should be removed from all the release
>>> branches after creating relase branches from trunk)
>>> 
>>> 
>>> What do you think?
>>> 
>>> Thanks,
>>> Marton
>>> 
>>> [1]:
>>> 
>>> 
>> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
>>> c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
>>> [2]:
>>> 
>>> 
>> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
>>> [3]:
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
>>> [4]:
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
>>> 
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>> 
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this 
enthusiasm turns into building a real, independent but more importantly a 
sustainable community.

Given that there were two official releases off the Apache Hadoop project, I 
doubt if you'd need to go through the incubator process. Instead you can 
directly propose a new TLP at ASF board. The last few times this happened was 
with ORC, and long before that with Hive, HBase etc. Can somebody who have 
cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and 
so need a committer +1? You can be more gently by formally doing a vote once a 
process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, 
you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can 
exercise their will to join this new project as PMC/Committers respectively 
without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu  于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣  于2019年7月22日周一 上午9:38写道：
>>> 
 Hi
 
 I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
 a major in electrical engineering. Our teaching teams and students use
 bigoop submarine for big data analysis and automation control of
>>> electrical
 equipment.
 
 Many thanks to the hadoop community for providing us with machine
>>> learning
 tools like submarine.
 
 I wish hadoop submarine is getting better and better.
 
 
 ==
 赵欣
 东南大学电气工程学院
 
 -
 
 Zhao XIN
 
 School of Electrical Engineering
 
 ==
 2019-07-18
 
 
 *From:* Xun Liu 
 *Date:* 2019-07-18 09:46
 *To:* xinzhao 
 *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
 project?
 
 
 -- Forwarded message -
 发件人： dashuiguailu...@gmail.com 
 Date: 2019年7月17日周三 下午3:17
 Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
 To: Szilard Nemeth , runlin zhang <
 runlin...@gmail.com>
 Cc: Xun Liu , common-dev <
>>> common-dev@hadoop.apache.org>,
 yarn-dev , hdfs-dev <
 hdfs-...@hadoop.apache.org>, mapreduce-dev <
 mapreduce-...@hadoop.apache.org>, submarine-dev <
 submarine-...@hadoop.apache.org>
 
 
 +1 ，Good idea, we are very much looking forward to it.
 
 --
 dashuiguailu...@gmail.com
 
 
 *From:* Szilard Nemeth 
 *Date:* 2019-07-17 14:55
 *To:* runlin zhang 
 *CC:* Xun Liu ; Hadoop Common
 ; yarn-dev ;
 Hdfs-dev ; mapreduce-dev
 ; submarine-dev
 
 *Subject:* Re: Any thoughts making Submarine a separate Apache project?
 +1, this is a very great idea.
 As Hadoop repository has already grown huge and contains many
>> projects, I
 think in general it's a good idea to separate projects in the early
>>> phase.
 
 
 On Wed, Jul 17, 2019, 08:50 runlin zhang  wrote:
 
> +1 ，That will be great ！
> 
>> 在 2019年7月10日，下午3:34，Xun Liu  写道：
>> 
>> Hi all,
>> 
>> This is Xun Liu contributing to the Submarine 

Re: [DISCUSS] Prefer JUnit5 for new tests

[Vinod Kumar Vavilapalli]

+1 if it is indeed possible to have both.

Thanks
+Vinod

> On Jul 26, 2019, at 1:56 PM, Akira Ajisaka  wrote:
> 
> Hi folks,
> 
> Now we are slowly migrating from JUnit4 to JUnit5.
> https://issues.apache.org/jira/browse/HADOOP-14693
> 
> However, as Steve commented [1], if we are going to migrate the
> existing tests, the backporting cost will become too expensive.
> Therefore, I'd like to recommend using JUnit5 for new tests before
> migrating the existing tests. Using junit-vintage-engine, we can mix
> JUnit4 and JUnit5 APIs in the same module, so writing new tests in
> JUnit5 is relatively easy.
> 
> Any thoughts?
> 
> [1] 
> https://issues.apache.org/jira/browse/HADOOP-16318?focusedCommentId=16890955=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16890955
> 
> -Akira
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

[Vinod Kumar Vavilapalli]

Makes sense, +1.

Thanks
+Vinod

> On Jul 17, 2019, at 11:37 AM, Elek, Marton  wrote:
> 
> Hi,
> 
> Github UI (ui!) helps to merge Pull Requests to the proposed branch.
> There are three different ways to do it [1]:
> 
> 1. Keep all the different commits from the PR branch and create one
> additional merge commit ("Create a merge commit")
> 
> 2. Squash all the commits and commit the change as one patch ("Squash
> and merge")
> 
> 3. Keep all the different commits from the PR branch but rebase, merge
> commit will be missing ("Rebase and merge")
> 
> 
> 
> As only the option 2 is compatible with the existing development
> practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
> consensus vote: If no objections withing 3 days, I will ask INFRA to
> disable the options 1 and 3 to make the process less error prone.
> 
> Please let me know, what do you think,
> 
> Thanks a lot
> Marton
> 
> ps: Personally I prefer to merge from local as it enables to sign the
> commits and do a final build before push. But this is a different story,
> this proposal is only about removing the options which are obviously
> risky...
> 
> ps2: You can always do any kind of merge / commits from CLI, for example
> to merge a feature branch together with keeping the history.
> 
> [1]:
> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Fwd: Incorrect NOTICE files in TLP releases

[Vinod Kumar Vavilapalli]

A bit of an old email, but want to make sure this isn't missed.

Has anyone looked into this concern?

Ref https://issues.apache.org/jira/browse/ROL-2138 
.

Thanks
+Vinod

> Begin forwarded message:
> 
> From: sebb 
> Subject: Incorrect NOTICE files in TLP releases
> Date: June 21, 2019 at 2:34:17 AM GMT+5:30
> To: "bo...@apache.org Board" 
> Reply-To: bo...@apache.org
> 
> To whom it may concern:
> 
> I had occasion to download the source for Roller, and happened to look
> at the NOTICE file.
> It does not conform to ASF policies, so I raised ROL-2138.
> 
> One of the replies asked how to manage different N files for binary
> and source releases, and pointed out that Hadoop and Karaf don't
> appear to have multiple copies of the files.
> 
> So I had a look at Hadoop and Karaf; their NOTICE files are also
> non-standard, and it looks like Kafka does not have a NOTICE file in
> the source bundle.
> 
> I suspect these are not the only projects with non-conformant NOTICE files.
> The LICENSE files are also likely to be incorrect (I have not checked).
> 
> Sebb.

Re: June Hadoop Community Meetup

[Vinod Kumar Vavilapalli]

Plus general@ list.

Thanks
+Vinod

> On Jun 4, 2019, at 9:47 PM, Daniel Templeton  
> wrote:
> 
> The meetup page is now live:
> 
>https://www.meetup.com/Hadoop-Contributors/events/262055924
> 
> I'll fill in the agenda details after we get them nailed down.  The meetup 
> will be an all-day event on June 26th with lunch provided and a reception 
> after.  Let me know if there are any questions.
> 
> Hope to see you there!
> Daniel
> 
> On 5/23/19 10:57 AM, Daniel Templeton wrote:
>> Hi, all!  I want to let you know that Cloudera is planning to host a 
>> contributors meetup on June 26 at our Palo Alto headquarters. We're still 
>> working out the details, but we're hoping to follow the format that Oath and 
>> LinkedIn followed during the last two. Please feel free to reach out to me 
>> if you have a topic you'd like to propose for the meetup.  I will also be 
>> reaching out to key folks from the community to solicit ideas.  I will send 
>> out an update with more details when I have more to share.
>> 
>> Thanks!
>> Daniel
> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Impact of Not Enabling KerberosDelegationTokenAuthenticationHandler

[Vinod Kumar Vavilapalli]

Not sure what your question really is.

KerberosDelegationTokenAuthenticationHandler was created mainly to add 
delegation-token support on top of KerberosAuthenticationHandler.

Thanks
+Vinod

> On May 22, 2019, at 10:58 AM, Prabhu Joseph  
> wrote:
> 
> Hi,
> 
> Was testing YARN RM with Http AuthenticationFilter set with
> KerberosAuthenticationHandler which authenticates only through Kerberos and
> not Delegation Token. Have observed everything like Yarn jobs, yarn
> commands, UI and Rest API working fine with this setup with valid kerberos
> ticket.
> 
> Want to understand where the impact is then when not using
> KerberosDelegationTokenAuthenticationHandler.
> 
> Thanks,
> Prabhu Joseph


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Issue in docs page : single cluster yarn setup - https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html#YARN_on_Single_Node

[Vinod Kumar Vavilapalli]

Good catch.

Please file a JIRA ticket here: https://issues.apache.org/jira/browse/YARN 
. Please put a patch up too if you 
can!

The second property is a configuration for MR jobs to figure out where the 
MapReduce framework libraries are.

Thanks
+Vinod

> On May 14, 2019, at 4:43 AM, Vishva Avu  wrote:
> 
> Hi,
> 
> Step 1 for
> https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html#YARN_on_Single_Node
> 
> Configure parameters as follows:
> 
> etc/hadoop/mapred-site.xml:
> 
> 
>
>mapreduce.framework.name
>yarn
>
> 
> 
> 
>
>mapreduce.application.classpath
>
> $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
>
> 
> 
> but setting this will throw an error when running yarn :
> 
> 2019-05-14 16:32:05,815 ERROR org.apache.hadoop.conf.Configuration: error
> parsing conf mapred-site.xml
> com.ctc.wstx.exc.WstxParsingException: Illegal to have multiple roots
> (start tag in epilog?).
> 
> This should be modified to
> 
>
>mapreduce.framework.name
>yarn
>
>
>mapreduce.application.classpath
>
> $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
>
> 
> 
> and also what is the function of second property here ?
> 
> I could not find the issues page where I can log this, so mailing it here
> 
> Regards,
> Vishva

Re: Cannot kill Pre-Commit jenkins builds

[Vinod Kumar Vavilapalli]

Done. Please see if it works for you.

Thanks
+Vinod

> On Feb 14, 2019, at 1:12 PM, Arun Suresh  wrote:
> 
> Hello Vinod
> 
> Kindly add me (asuresh) and Jonathan Hung as well..
> 
> Cheers
> -Arun
> 
> On Thu, Feb 14, 2019, 11:46 AM Vinod Kumar Vavilapalli  <mailto:vino...@apache.org>> wrote:
> Plus Sunil / Wangda.
> 
> Sunil found out some information - "Jenkins uses the Apache LDAP servers for 
> authentication. To give a committer access to Jenkins, the committer must be 
> made a member of the hudson-jobadmin group. This is done using the Whimsy 
> Tool which all PMC Chairs can change."
> 
> I've done this for Sunil & Wangda as they needed help for Submarine.
> 
> Arun Suresh and other release-managers / committers, let me know if you need 
> to be added.
> 
> Thanks
> +Vinod
> 
> > On Jan 22, 2019, at 4:50 PM, Arun Suresh  > <mailto:arun.sur...@gmail.com>> wrote:
> > 
> > Hmmm.. as per this (https://wiki.apache.org/general/Jenkins 
> > <https://wiki.apache.org/general/Jenkins>), looks like my
> > id needs to be added to the hudson-jobadmin group to affect any changes on
> > jenkins.
> > But wondering why it was revoked in the first place.
> > 
> > On Tue, Jan 22, 2019 at 4:21 PM Vinod Kumar Vavilapalli  > <mailto:vino...@apache.org>>
> > wrote:
> > 
> >> Minus private.
> >> 
> >> Which specific job you are looking? I looked at
> >> https://builds.apache.org/job/PreCommit-YARN-Build/ 
> >> <https://builds.apache.org/job/PreCommit-YARN-Build/> but can't seem to
> >> find any user specific auth.
> >> 
> >> +Vinod
> >> 
> >> On Jan 22, 2019, at 10:00 AM, Arun Suresh  >> <mailto:asur...@apache.org>> wrote:
> >> 
> >> Hey Vinod.. Ping!
> >> 
> >> Cheers
> >> -Arun
> >> 
> >> On Fri, Jan 18, 2019 at 9:46 AM Arun Suresh  >> <mailto:asur...@apache.org>> wrote:
> >> 
> >> Hi Vinod
> >> 
> >> Can you please help with this:
> >> https://issues.apache.org/jira/browse/INFRA-17673 
> >> <https://issues.apache.org/jira/browse/INFRA-17673> ?
> >> 
> >> Cheers
> >> -Arun
> >> 
> >> On Wed, Jan 16, 2019, 12:53 PM Arun Suresh  >> <mailto:asur...@apache.org> wrote:
> >> 
> >> Hi
> >> 
> >> We are currently trying to get the branch-2 pre-commit builds working.
> >> I used to be able to kill Pre-Commit jenkins jobs, but looks like I am
> >> not allowed to anymore. Has anything changed recently w.r.t permissions etc
> >> ?
> >> 
> >> Cheers
> >> -Arun
> >> 
> >> 
> >> 
> >> 
> 

Re: [DISCUSS] Moving branch-2 to java 8

[Vinod Kumar Vavilapalli]

The community made a decision long time ago that we'd like to keep the 
compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on 3.x.

I always assumed that most (all?) downstream users build branch-2 on JDK 7 
only, can anyone confirm? If so, there may be an easier way to address these 
test issues.

+Vinod

> On Jan 28, 2019, at 11:24 AM, Jonathan Hung  wrote:
> 
> Hi folks,
> 
> Forking a discussion based on HADOOP-15711. To summarize, there are issues
> with branch-2 tests running on java 7 (openjdk) which don't exist on java
> 8. From our testing, the build can pass with openjdk 8.
> 
> For branch-3, the work to move the build to use java 8 was done in
> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053 was
> filed to backport this OS version change to branch-2 (but without the java
> 7 -> java 8 change). So my proposal is to also make the java 7 -> java 8
> version change in branch-2.
> 
> As mentioned in HADOOP-15711, the main issue is around source and binary
> compatibility. I don't currently have a great answer, but one initial
> thought is to build source/binary against java 7 to ensure compatibility
> and run the rest of the build as java 8.
> 
> Thoughts?
> 
> Jonathan Hung


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Result] [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Vinod Kumar Vavilapalli]

Agree, it isn't productive this way.

I can't seem to find it, but was there a DISCUSS thread for this branch-merge? 
I usually recommend addressing issues on a DISCUSS thread instead of fighting 
things over a VOTE.

+Vinod

> On Dec 13, 2018, at 10:09 AM, Konstantin Shvachko  
> wrote:
> 
> This vote failed due to Daryn Sharp's veto.
> The concern is being addressed by HDFS-13873. I will start a new vote once
> this is committed.
> 
> Note for Daryn. Your non-responsive handling of the veto makes a bad
> precedence and is a bad example of communication on the lists from a
> respected member of this community. Please check your availability for
> followup discussions if you choose to get involved with important decisions.
> 
> On Fri, Dec 7, 2018 at 4:10 PM Konstantin Shvachko 
> wrote:
> 
>> Hi Daryn,
>> 
>> Wanted to backup Chen's earlier response to your concerns about rotating
>> calls in the call queue.
>> Our design
>> 1. targets directly the livelock problem by rejecting calls on the
>> Observer that are not likely to be responded in timely matter: HDFS-13873.
>> 2. The call queue rotation is only done on Observers, and never on the
>> active NN, so it stays free of attacks like you suggest.
>> 
>> If this is a satisfactory mitigation for the problem could you please
>> reconsider your -1, so that people could continue voting on this thread.
>> 
>> Thanks,
>> --Konst
>> 
>> On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:
>> 
>>> -1 pending additional info.  After a cursory scan, I have serious
>>> concerns regarding the design.  This seems like a feature that should have
>>> been purely implemented in hdfs w/o touching the common IPC layer.
>>> 
>>> The biggest issue in the alignment context.  It's purpose appears to be
>>> for allowing handlers to reinsert calls back into the call queue.  That's
>>> completely unacceptable.  A buggy or malicious client can easily cause
>>> livelock in the IPC layer with handlers only looping on calls that never
>>> satisfy the condition.  Why is this not implemented via RetriableExceptions?
>>> 
>>> On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang 
>>> wrote:
>>> 
 Great work guys.
 
 Wonder if we can elaborate what's impact of not having #2 fixed, and why
 #2
 is not needed for the feature to complete?
 2. Need to fix automatic failover with ZKFC. Currently it does not
 doesn't
 know about ObserverNodes trying to convert them to SBNs.
 
 Thanks.
 --Yongjun
 
 
 On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko  
 wrote:
 
> Hi Hadoop developers,
> 
> I would like to propose to merge to trunk the feature branch
 HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale
 read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC
 workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
> 
> The main functionality has been implemented see sub-tasks of
 HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
> 
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new
 feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not
 doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
> 
> I attached a unified patch to the umbrella jira for the review and
 Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec
 12.
> 
> Thanks,
> --Konstantin
> 
 
>>> 
>>> 
>>> --
>>> 
>>> Daryn
>>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Move to gitbox

[Vinod Kumar Vavilapalli]

We need to write up the stages of the process and put a concrete timeline first.

Is it an atomic move? Or will both repos live at the same time? Do we need a 
stop-the-world period and messaging sent out?

When do the git-wip repos get deleted and/or stop getting updates?

Things I can think of that need tracking
 - Wiki pages?
 - Jenkins jobs
 - Email notification setup for the commits?
 - Policies - like no-mainline-branch-deletion support

+Vinod

> On Dec 13, 2018, at 7:57 AM, Elek, Marton  wrote:
> 
> 
> 
> On 12/12/18 12:27 PM, Akira Ajisaka wrote:
>> Thank you for your positive feedback! I'll file a jira to INFRA in this 
>> weekend.
>> 
>>> If I understood well the only bigger task here is to update all the jenkins 
>>> jobs. (I am happy to help/contribute what I can do)
> 
>> Thank you Elek for the information. Do you have the privilege to
>> update the Jenkins jobs?
>> 
> I have, but I am more familiar with the Ozone jenkins jobs. I created a
> jira (HADOOP-16003) to discuss / record the changes / where it can be
> discussed or commented by anybody with more expertise.
> 
> Marton
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Can someone add me to the JIRA admin list

[Vinod Kumar Vavilapalli]

That's right, there's no policy. It's done on an adhoc/need basis. There is no 
automated way to put all committers in that ACL.

+Vinod

> On Dec 13, 2018, at 7:59 AM, Sean Busbey  wrote:
> 
> AFAIK we don't have like a policy or something. If folks want to help
> on one of the JIRA instances I say give them the access. I don't just
> mean that for committers, but especially for committers if we can't
> trust someone to not try to mess things up in one of the trackers we
> have bigger problems.
> On Thu, Dec 13, 2018 at 9:49 AM Wei-Chiu Chuang  wrote:
>> 
>> Quick questions here,
>> Do committers get added to admin role on all Hadoop projects (i.e. HADOOP, 
>> HDFS, YARN, MAPREDUCE and HDDS) or just a subset of them?
>> I am admin on HADOOP/HDFS and for the most part that works fine for me, 
>> because I primarily only work on HDFS and Hadoop Common space.
>> And that's probably a good idea to respect my peer YARN/MAPREDUCE/HDDS 
>> committers, but I am just curious.
>> 
>> On Thu, Dec 13, 2018 at 5:37 AM Sean Busbey  wrote:
>>> 
>>> It looks like I'm only listed as a JIRA admin on the YARN tracker. Could 
>>> someone add me on the rest of the project trackers? I'm trying to get a new 
>>> contributor on-boarded so I can assign them an issue they're working.
>>> 
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.1.2 release plan

[Vinod Kumar Vavilapalli]

231 fixed JIRAs is already quite a bunch!

I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them as 
blocker.

Why not just release now as soon as there are no blockers?

Thanks
+Vinod

> On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> 
> Hi, All
> 
> We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> improve the quality of the release, I plan to release 3.1.2
> by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> and other enhancements. So far there are 231 JIRAs [1] have fix
> version marked to 3.1.2
> 
> I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> 
> Please feel free to share your insights.
> 
> Thanks,
> Wangda Tan
> 
> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
> AND fixVersion = 3.1.2


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop Ozone 0.2.1-alpha release

[Vinod Kumar Vavilapalli]

Congratulations on the release!

Just seeing this, why is this its own website with its own downloads section, 
look & feel etc instead of being part of our larger site? 

+Vinod


> On Oct 1, 2018, at 5:24 PM, Elek, Marton  wrote:
> 
> 
> It gives me great pleasure to announce that the Apache Hadoop community has 
> voted to release Apache Hadoop Ozone 0.2.1-alpha.
> 
> Apache Hadoop Ozone is an object store for Hadoop built using Hadoop 
> Distributed Data Store.
> 
> For more information and to download, please check
> 
> https://hadoop.apache.org/ozone
> 
> Note: This release is alpha quality, it's not recommended to use in 
> production.
> 
> Many thanks to everyone who contributed to the release, and everyone in the 
> Apache Hadoop community! The release is a result of work from many 
> contributors. Thank you for all of them.
> 
> On behalf of the Hadoop community,
> Márton Elek
> 
> 
> ps: Hadoop Ozone and HDDS are released separately from the main Hadoop 
> releases, this release doesn't include new Hadoop Yarn/Mapreduce/Hdfs 
> versions.
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Vinod Kumar Vavilapalli]

Is there no way to host the new site and the old site concurrently? And link 
back & forth?

+Vinod


> On Aug 31, 2018, at 1:07 AM, Elek, Marton  wrote:
> 
> Bumping this thread at last time.
> 
> I have the following proposal:
> 
> 1. I will request a new git repository hadoop-site.git and import the new 
> site to there (which has exactly the same content as the existing site).
> 
> 2. I will ask infra to use the new repository as the source of 
> hadoop.apache.org
> 
> 3. I will sync manually all of the changes in the next two months back to the 
> svn site from the git (release announcements, new committers)
> 
> IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
> 
> If no-one objects within three days, I'll assume lazy consensus and start 
> with this plan. Please comment if you have objections.
> 
> Again: it allows immediate fallback at any time as svn repo will be kept as 
> is (+ I will keep it up-to-date in the next 2 months)
> 
> Thanks,
> Marton
> 
> 
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
>> Thank you very much to bump up this thread.
>> About [2]: (Just for the clarification) the content of the proposed website 
>> is exactly the same as the old one.
>> About [1]. I believe that the "mvn site" is perfect for the documentation 
>> but for website creation there are more simple and powerful tools.
>> Hugo has more simple compared to jekyll. Just one binary, without 
>> dependencies, works everywhere (mac, linux, windows)
>> Hugo has much more powerful compared to "mvn site". Easier to create/use 
>> more modern layout/theme, and easier to handle the content (for example new 
>> release announcements could be generated as part of the release process)
>> I think it's very low risk to try out a new approach for the site (and easy 
>> to rollback in case of problems)
>> Marton
>> ps: I just updated the patch/preview site with the recent releases:
>> ***
>> * http://hadoop.anzix.net *
>> ***
>> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
>>> Got pinged about this offline.
>>> 
>>> Thanks for keeping at it, Marton!
>>> 
>>> I think there are two road-blocks here
>>>   (1) Is the mechanism using which the website is built good enough - 
>>> mvn-site / hugo etc?
>>>   (2) Is the new website good enough?
>>> 
>>> For (1), I just think we need more committer attention and get feedback 
>>> rapidly and get it in.
>>> 
>>> For (2), how about we do it in a different way in the interest of progress?
>>>   - We create a hadoop.apache.org/new-site/ where this new site goes.
>>>   - We then modify the existing web-site to say that there is a new 
>>> site/experience that folks can click on a link and navigate to
>>>   - As this new website matures and gets feedback & fixes, we finally pull 
>>> the plug at a later point of time when we think we are good to go.
>>> 
>>> Thoughts?
>>> 
>>> +Vinod
>>> 
>>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I would like to bump this thread up.
>>>> 
>>>> TLDR; There is a proposed version of a new hadoop site which is available 
>>>> from here: https://elek.github.io/hadoop-site-proposal/ and 
>>>> https://issues.apache.org/jira/browse/HADOOP-14163
>>>> 
>>>> Please let me know what you think about it.
>>>> 
>>>> 
>>>> Longer version:
>>>> 
>>>> This thread started long time ago to use a more modern hadoop site:
>>>> 
>>>> Goals were:
>>>> 
>>>> 1. To make it easier to manage it (the release entries could be created by 
>>>> a script as part of the release process)
>>>> 2. To use a better look-and-feel
>>>> 3. Move it out from svn to git
>>>> 
>>>> I proposed to:
>>>> 
>>>> 1. Move the existing site to git and generate it with hugo (which is a 
>>>> single, standalone binary)
>>>> 2. Move both the rendered and source branches to git.
>>>> 3. (Create a jenkins job to generate the site automatically)
>>>> 
>>>> NOTE: this is just about forrest based hadoop.apache.org, NOT about the 
>>>> documentation which is generated by mvn-site (as before)
>>>> 
>>>> 
>>>> I got multiple valuable feedback and I improved 

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Vinod Kumar Vavilapalli]

Yongjun,

Looks like you didn't add the links to 3.0.3 binary release on the 
http://hadoop.apache.org/releases.html page.

I just did it, FYI: 
https://svn.apache.org/viewvc?view=revision=1837967 


Thanks
+Vinod


> On May 31, 2018, at 10:48 PM, Yongjun Zhang  wrote:
> 
> Greetings all,
> 
> I've created the first release candidate (RC0) for Apache Hadoop
> 3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
> about 249
> important fixes and improvements, among which there are 8 blockers. See
> https://issues.apache.org/jira/issues/?filter=12343997
> 
> The RC artifacts are available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
> 
> The maven artifacts are available via
> https://repository.apache.org/content/repositories/orgapachehadoop-1126
> 
> Please try the release and vote; the vote will run for the usual 5 working
> days, ending on 06/07/2018 PST time. Would really appreciate your
> participation here.
> 
> I bumped into quite some issues along the way, many thanks to quite a few
> people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.
> 
> Thanks,
> 
> --Yongjun

Re: Releases table needs to be cleaned up

[Vinod Kumar Vavilapalli]

Done.

Thanks
+Vinod

> On Jul 2, 2018, at 7:12 AM, Andrew Wang  
> wrote:
> 
> Hi folks,
> 
> https://hadoop.apache.org/releases.html
> 
> The table of releases here is supposed to only contain one row per release
> line. We've got a lot of old dupes at this point, e.g. 3.0.2, 2.9.0, 2.8.3.
> It'd also be good to keep this sorted by version number, so 3.1.0 is at the
> top.
> 
> Could a recent RM take care of cleaning this up?
> 
> Thanks,
> Andrew


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.7

[Vinod Kumar Vavilapalli]

Steve,

Looks like you didn't add the links to 2.7.7 source release at all on the 
http://hadoop.apache.org/releases.html page!

I just did it, FYI: 
https://svn.apache.org/viewvc?view=revision=1837966 


Thanks
+Vinod

> On Jul 16, 2018, at 4:23 PM, Steve Loughran  wrote:
> 
> and of course, my own +1 (binding)
> 
> * downloaded
> * verified all signatures
> * rebuilt on a Docker image, including native bits
> * retested, inc hadoop-aws and hadoop-azure. Failure on KMS 
> (testStartStopHttpKerberos, "Connection Reset"), which I didn't look into.
> * Basic command line
> * did a spark build, so verifying poms, but not a full test run there. Not 
> been building it enough to know if a failure would have been a regression 
> anyway.
> 
> 
> Nobody else has noticed, but the 2.7 native bits don't build on macosI 
> didn't pull that patch in from branch-2 but I will for future branch 2.7 
> maint. I also had to do changes to get the docker image to work (but work it 
> did!); these have been done outside the source tree, again, I'll try and get 
> that it. What it does mean is that the source distro here can't completely 
> bring up the docker image for a full release build, including GPG signing.
> 
> I realise I could be better set up for qualifying releases here. I normally 
> get to rely on our automated QE tests telling me that something has broken; I 
> plan to see what I can do with bigtop locally, so have a better set up for 
> end-to-end testing of packaging.
> 
> Now, let me count those votes.
> 
> -Steve
> 
> 
> 
> 
> 
> 
> 
> 
> On 16 Jul 2018, at 21:50, Eric Badger 
> mailto:ebad...@oath.com>> wrote:
> 
> +1 (non-binding)
> 
> - Verified all hashes and checksums
> - Built from source on macOS 10.13.5, Java 1.8.0u65
> - Deployed a pseudo cluster
> - Ran some example jobs
> 
> 
> On Mon, Jul 16, 2018 at 2:18 PM, Kihwal Lee 
> mailto:kih...@oath.com.invalid>> wrote:
> +1 (binding)
> -Built from the source
> -Brought up a single node pseudo-distributed cluster
> -Ran basic tests
> -Verified basic hdfs and dfsadmin commands working
> 
> 
> On Wed, Jul 11, 2018 at 10:05 AM Steve Loughran 
> mailto:ste...@hortonworks.com>>
> wrote:
> 
>> 
>> 
>> Hi
>> 
>> I've got RC0 of Hadoop 2.7.7 up for people to download and play with
>> 
>> http://people.apache.org/~stevel/hadoop-2.7.7/RC0/
>> 
>> Nexus artifacts 2.7.7 are up in staging
>> https://repository.apache.org/content/repositories/orgapachehadoop-1135
>> 
>> Git (signed) tag release-2.7.7-RC0, checksum
>> c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac
>> 
>> My current GPG key is 38237EE425050285077DB57AD22CF846DBB162A0
>> you can download this direct (and it MUST be direct) from the ASF HTTPS
>> site
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> Thanks
>> 
>> -Steve
>> 
>> (I should add: this is my first ever attempt at a Hadoop release, please
>> be (a) rigorous and (b) forgiving. Credit to Wei-Chiu Chuang for his
>> advice).
>> 
>> 
>> -
>> To unsubscribe, e-mail: 
>> common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: 
>> common-dev-h...@hadoop.apache.org
>> 
>> 
> 
> 

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Vinod Kumar Vavilapalli]

+1

Thanks
+Vinod

> On Jul 6, 2018, at 11:12 AM, Sunil G  wrote:
> 
> I just checked.  YARN-7556 and YARN-7451 can be cherry-picked.
> I cherry-picked in my local and compiled. Things are good.
> 
> I can push this now  which will restore trunk to its original.
> I can do this if there are no objection.
> 
> - Sunil
> 
> On Fri, Jul 6, 2018 at 11:10 AM Arpit Agarwal  <mailto:aagar...@hortonworks.com>>
> wrote:
> 
>> afaict YARN-8435 is still in trunk. YARN-7556 and YARN-7451 are not.
>> 
>> 
>> From: Giovanni Matteo Fumarola 
>> Date: Friday, July 6, 2018 at 10:59 AM
>> To: Vinod Kumar Vavilapalli 
>> Cc: Anu Engineer , Arpit Agarwal <
>> aagar...@hortonworks.com>, "su...@apache.org" , "
>> yarn-...@hadoop.apache.org" , "
>> hdfs-...@hadoop.apache.org" , "
>> common-dev@hadoop.apache.org" , "
>> mapreduce-...@hadoop.apache.org" 
>> Subject: Re: [VOTE] reset/force push to clean up inadvertent merge commit
>> pushed to trunk
>> 
>> Everything seems ok except the 3 commits: YARN-8435, YARN-7556, YARN-7451
>> are not anymore in trunk due to the revert.
>> 
>> Haibo/Robert if you can recommit your patches I will commit mine
>> subsequently to preserve the original order.
>> 
>> (My apology for the mess I did with the merge commit)
>> 
>> On Fri, Jul 6, 2018 at 10:42 AM, Vinod Kumar Vavilapalli <
>> vino...@apache.org <mailto:vino...@apache.org><mailto:vino...@apache.org 
>> <mailto:vino...@apache.org>>> wrote:
>> I will add that the branch also successfully compiles.
>> 
>> Let's just move forward as is, unblock commits and just fix things if
>> anything is broken.
>> 
>> +Vinod
>> 
>>> On Jul 6, 2018, at 10:30 AM, Anu Engineer >> <mailto:aengin...@hortonworks.com>
>> <mailto:aengin...@hortonworks.com <mailto:aengin...@hortonworks.com>>> wrote:
>>> 
>>> Hi All,
>>> 
>>> [ Thanks to Arpit for working offline and verifying that branch is
>> indeed good.]
>>> 
>>> I want to summarize what I know of this issue and also solicit other
>> points of view.
>>> 
>>> We reverted the commit(c163d1797) from the branch, as soon as we noticed
>> it. That is, we have made no other commits after the merge commit.
>>> 
>>> We used the following command to revert
>>> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
>>> 
>>> Giovanni's branch had three commits + merge, The JIRAs he had were
>> YARN-7451, YARN-7556, YARN-8435.
>>> 
>>> The issue seems to be the revert of merge has some diffs. I am not a
>> YARN developer, so the only problem is to look at the revert and see if
>> there were any spurious edits in Giovanni's original commit + merge.
>>> If there are none, we don't need a reset/force push.  But if we find an
>> issue I am more than willing to go the force commit route.
>>> 
>>> The revert takes the trunk back to the point of the first commit from
>> Giovanni which is YARN-8435. His branch was also rewriting the order of
>> commits which we have lost due to the revert.
>>> 
>>> Based on what I know so far, I am -1 on the force push.
>>> 
>>> In other words, I am trying to understand why we need the force push. I
>> have left a similar comment in JIRA (
>> https://issues.apache.org/jira/browse/INFRA-16727 
>> <https://issues.apache.org/jira/browse/INFRA-16727>) too.
>>> 
>>> 
>>> Thanks
>>> Anu
>>> 
>>> 
>>> On 7/6/18, 10:24 AM, "Arpit Agarwal" >> <mailto:aagar...@hortonworks.com>> aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>> wrote:
>>> 
>>>   -1 for the force push. Nothing is broken in trunk. The history looks
>> ugly for two commits and we can live with it.
>>> 
>>>   The revert restored the branch to Giovanni's intent. i.e. only
>> YARN-8435 is applied. Verified there is no delta between hashes 0d9804d and
>> 39ad989 (HEAD).
>>> 
>>>   39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch
>> 't...
>>>   c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of
>> https://git- <https://git-/>...
>>>   99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to
>> veri...
>>>   1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler
>> configurat...
>>>   0d9804d 2018-07-05 gifuma@apa o │

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Vinod Kumar Vavilapalli]

I will add that the branch also successfully compiles.

Let's just move forward as is, unblock commits and just fix things if anything 
is broken.

+Vinod

> On Jul 6, 2018, at 10:30 AM, Anu Engineer  wrote:
> 
> Hi All,
> 
> [ Thanks to Arpit for working offline and verifying that branch is indeed 
> good.]
> 
> I want to summarize what I know of this issue and also solicit other points 
> of view.
> 
> We reverted the commit(c163d1797) from the branch, as soon as we noticed it. 
> That is, we have made no other commits after the merge commit.
> 
> We used the following command to revert 
> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
> 
> Giovanni's branch had three commits + merge, The JIRAs he had were YARN-7451, 
> YARN-7556, YARN-8435.
> 
> The issue seems to be the revert of merge has some diffs. I am not a YARN 
> developer, so the only problem is to look at the revert and see if there were 
> any spurious edits in Giovanni's original commit + merge. 
> If there are none, we don't need a reset/force push.  But if we find an issue 
> I am more than willing to go the force commit route.
> 
> The revert takes the trunk back to the point of the first commit from 
> Giovanni which is YARN-8435. His branch was also rewriting the order of 
> commits which we have lost due to the revert.
> 
> Based on what I know so far, I am -1 on the force push.
> 
> In other words, I am trying to understand why we need the force push. I have 
> left a similar comment in JIRA 
> (https://issues.apache.org/jira/browse/INFRA-16727) too.
> 
> 
> Thanks
> Anu
> 
> 
> On 7/6/18, 10:24 AM, "Arpit Agarwal"  wrote:
> 
>-1 for the force push. Nothing is broken in trunk. The history looks ugly 
> for two commits and we can live with it.
> 
>The revert restored the branch to Giovanni's intent. i.e. only YARN-8435 
> is applied. Verified there is no delta between hashes 0d9804d and 39ad989 
> (HEAD).
> 
>39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch 't...
>c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of https://git-...
>99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to veri...
>1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler configurat...
>0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same cli...
>71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce NodeStateManager...
> 
>Regards,
>Arpit
> 
> 
>On 7/5/18, 2:37 PM, "Subru Krishnan"  wrote:
> 
>Folks,
> 
>There was a merge commit accidentally pushed to trunk, you can find the
>details in the mail thread [1].
> 
>I have raised an INFRA ticket [2] to reset/force push to clean up 
> trunk.
> 
>Can we have a quick vote for INFRA sign-off to proceed as this is 
> blocking
>all commits?
> 
>Thanks,
>Subru
> 
>[1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
>[2] https://issues.apache.org/jira/browse/INFRA-16727
> 
> 
> 
>-
>To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15571) After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration object are forced to have the same umask

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created HADOOP-15571:


 Summary: After HADOOP-13440, multiple filesystems/file-contexts 
created with the same Configuration object are forced to have the same umask
 Key: HADOOP-15571
 URL: https://issues.apache.org/jira/browse/HADOOP-15571
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli


Ran into a super hard-to-debug due to this.
h4. Issue

Configuration conf = new Configuration();
 fc1 = FileContext.getFileContext(uri1, conf);
 fc2 = FileContext.getFileContext(uri2, conf);
 fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!

This was not the case before HADOOP-13440.
h4. Symptoms:
h5. Scenario I ran into

When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager 
tries to replicate the directory structure on the local file-system 
($yarn-local-dirs/filecache/my/dir/1.txt).

Now depending on whether NM has ever done a log-aggregation (completely 
unrelated code that sets umask to be 137 for its own files on HDFS), the 
directories /my and /my/dir on local-fs may have different permissions. In the 
specific case where NM did log-aggregation, /my/dir was created with 137 umask 
and so localization of 1.txt completely failed due to absent directory 
executable permissions!
h5. Previous scenarios:

We ran into this before in test-cases and instead of fixing the root-cause, we 
just fixed the test-cases: YARN-5679 / YARN-5749



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Vinod Kumar Vavilapalli]

Got pinged about this offline.

Thanks for keeping at it, Marton!

I think there are two road-blocks here
 (1) Is the mechanism using which the website is built good enough - mvn-site / 
hugo etc?
 (2) Is the new website good enough?

For (1), I just think we need more committer attention and get feedback rapidly 
and get it in.

For (2), how about we do it in a different way in the interest of progress?
 - We create a hadoop.apache.org/new-site/ where this new site goes.
 - We then modify the existing web-site to say that there is a new 
site/experience that folks can click on a link and navigate to
 - As this new website matures and gets feedback & fixes, we finally pull the 
plug at a later point of time when we think we are good to go.

Thoughts?

+Vinod

> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> 
> Hi,
> 
> I would like to bump this thread up.
> 
> TLDR; There is a proposed version of a new hadoop site which is available 
> from here: https://elek.github.io/hadoop-site-proposal/ and 
> https://issues.apache.org/jira/browse/HADOOP-14163
> 
> Please let me know what you think about it.
> 
> 
> Longer version:
> 
> This thread started long time ago to use a more modern hadoop site:
> 
> Goals were:
> 
> 1. To make it easier to manage it (the release entries could be created by a 
> script as part of the release process)
> 2. To use a better look-and-feel
> 3. Move it out from svn to git
> 
> I proposed to:
> 
> 1. Move the existing site to git and generate it with hugo (which is a 
> single, standalone binary)
> 2. Move both the rendered and source branches to git.
> 3. (Create a jenkins job to generate the site automatically)
> 
> NOTE: this is just about forrest based hadoop.apache.org, NOT about the 
> documentation which is generated by mvn-site (as before)
> 
> 
> I got multiple valuable feedback and I improved the proposed site according 
> to the comments. Allen had some concerns about the used technologies (hugo 
> vs. mvn-site) and I answered all the questions why I think mvn-site is the 
> best for documentation and hugo is best for generating site.
> 
> 
> I would like to finish this effort/jira: I would like to start a discussion 
> about using this proposed version and approach as a new site of Apache 
> Hadoop. Please let me know what you think.
> 
> 
> Thanks a lot,
> Marton
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15527) Sometimes daemons keep running even after "kill -9" from daemon-stop script

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created HADOOP-15527:


 Summary: Sometimes daemons keep running even after "kill -9" from 
daemon-stop script
 Key: HADOOP-15527
 URL: https://issues.apache.org/jira/browse/HADOOP-15527
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli


I'm seeing that sometimes daemons keep running for a little while even after 
"kill -9" from daemon-stop scripts.

Debugging more, I see several instances of "ERROR: Unable to kill ${pid}".

Saw this specifically with ResourceManager & NodeManager -  {{yarn --daemon 
stop nodemanager}}. Though it is possible that other daemons may run into this 
too.

Saw this on both Centos as well as Ubuntu.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

That is a great observation. And I missed your previous email about the shaded 
vs unshaded jars already getting fixed.

I guess we are good to go.

--

Looking at the RC. Went through my usual check-list. Here's my summary.

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form 
"hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B 
B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, 
"/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 
9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

+1 binding.

Thanks
+Vinod

> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> Hi Vinod / Arpit,
> 
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> 
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> 
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
> 
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
> 
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
> 
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
> 
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
> 
> I guess the differences come from our release process.
> 
> Thanks,
> Wangda
> 
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
> 
> 
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
> wrote:
> 
>> We vote on the source code. The binaries are convenience artifacts.
>> 
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>> 
>> Thanks
>> +Vinod
>> 
>> 
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>> 
>> HI Arpit,
>> 
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>> 
>> Thanks,
>> Wangda
>> 
>> 
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
>> wrote:
>> 
>>> Thanks Wangda, I see the shaded jars now.
>>> 
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>> 
>>> +1 (binding), modulo that remaining question.
>>> 
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>> 
>>> Thanks!
>>> 
>>> 
>>> ----------
>>> From: Wangda Tan <wheele...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aagar...@hortonworks.com>
>>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-...@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
>>> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>> 
>>> As point

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as the 
binaries to be consistent and correct. And then (b) Give a couple more days for 
folks who tested on the binaries to reverify - I count one such clear vote as 
of now.

Thanks
+Vinod

> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> HI Arpit,
> 
> I think it won't match if we do rebuild. It should be fine as far as they're 
> signed, correct? I don't see any policy doesn't allow this. 
> 
> Thanks,
> Wangda
> 
> 
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com 
> <mailto:aagar...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
> 
> Are the repo jars required to be the same as the binary release? They don’t 
> match right now, probably they got rebuilt.
> 
> +1 (binding), modulo that remaining question.
> 
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
> 
> Thanks!
> 
> 
> --
> From: Wangda Tan <wheele...@gmail.com <mailto:wheele...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>
> Cc: Gera Shegalov <ger...@gmail.com <mailto:ger...@gmail.com>>, Sunil G 
> <sun...@apache.org <mailto:sun...@apache.org>>, "yarn-...@hadoop.apache.org 
> <mailto:yarn-...@hadoop.apache.org>" <yarn-...@hadoop.apache.org 
> <mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev <hdfs-...@hadoop.apache.org 
> <mailto:hdfs-...@hadoop.apache.org>>, Hadoop Common 
> <common-dev@hadoop.apache.org <mailto:common-dev@hadoop.apache.org>>, 
> "mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>" 
> <mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>>, 
> Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> 
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
> redeployed jars and staged. @Arpit, could you please check the updated Maven 
> repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
> 
> Since the jars inside binary tarballs are correct 
> (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need 
> roll another RC, just update Maven repo should be sufficient. 
> 
> Best,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> Hi Arpit,
> 
> Thanks for pointing out this.
> 
> I just removed all .md5 files from artifacts. I found md5 checksums still 
> exist in .mds files and I didn't remove them from .mds file because it is 
> generated by create-release script and Apache guidance is "should not" 
> instead of "must not". Please let me know if you think they need to be 
> removed as well. 
> 
> - Wangda
> 
> 
> 
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
> <mailto:aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
> 
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> 
> https://www.apache.org/dev/release-distribution#sigs-and-sums 
> <https://www.apache.org/dev/release-distribution#sigs-and-sums>
> 
>  
> 
> 
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> 
> Hi Gera,
> 
> It's my bad, I thought only src/bin tarball is enough.
> 
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> 
> Please let me know if you have any other comments.
> 
> Thanks,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com 
> <mailto:ger...@gmail.com>> wrote:
> 
> 
> Thanks, Wangda!
> 
> There are many more artifacts in previou

[jira] [Resolved] (HADOOP-13500) Concurrency issues when using Configuration iterator

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-13500.
--
Resolution: Duplicate

Already fixed by HADOOP-13556. Closing as a dup. Please reopen if that isn't 
the case.

> Concurrency issues when using Configuration iterator
> 
>
> Key: HADOOP-13500
> URL: https://issues.apache.org/jira/browse/HADOOP-13500
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: conf
>Reporter: Jason Lowe
>Assignee: Ajay Kumar
>Priority: Major
>
> It is possible to encounter a ConcurrentModificationException while trying to 
> iterate a Configuration object.  The iterator method tries to walk the 
> underlying Property object without proper synchronization, so another thread 
> simultaneously calling the set method can trigger it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15208) DistCp to offer -xtrack option to save src/dest filesets as alternative to delete()

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-15208.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> DistCp to offer -xtrack  option to save src/dest filesets as 
> alternative to delete()
> --
>
> Key: HADOOP-15208
> URL: https://issues.apache.org/jira/browse/HADOOP-15208
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, 
> HADOOP-15208-002.patch, HADOOP-15208-003.patch
>
>
> There are opportunities to improve distcp delete performance and scalability 
> with object stores, but you need to test with production datasets to 
> determine if the optimizations work, don't run out of memory, etc.
> By adding the option to save the sequence files of source, dest listings, 
> people (myself included) can experiment with different strategies before 
> trying to commit one which doesn't scale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-15208) DistCp to offer -xtrack option to save src/dest filesets as alternative to delete()

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-15208:
--

Reopening and closing this instead as a dup of HADOOP-15209 as I can't find any 
patch for this in 3.1.0 for this JIRA. Revert back if this is incorrect.

> DistCp to offer -xtrack  option to save src/dest filesets as 
> alternative to delete()
> --
>
> Key: HADOOP-15208
> URL: https://issues.apache.org/jira/browse/HADOOP-15208
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-15208-001.patch, HADOOP-15208-002.patch, 
> HADOOP-15208-002.patch, HADOOP-15208-003.patch
>
>
> There are opportunities to improve distcp delete performance and scalability 
> with object stores, but you need to test with production datasets to 
> determine if the optimizations work, don't run out of memory, etc.
> By adding the option to save the sequence files of source, dest listings, 
> people (myself included) can experiment with different strategies before 
> trying to commit one which doesn't scale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14974) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation fails in trunk

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-14974.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
>  fails in trunk
> ---
>
> Key: HADOOP-14974
> URL: https://issues.apache.org/jira/browse/HADOOP-14974
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: John Zhuge
>Priority: Blocker
>
> {code}
> org.apache.hadoop.metrics2.MetricsException: Metrics source 
> QueueMetrics,q0=root already exists!
>   at 
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
>   at 
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
>   at 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:239)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:141)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:131)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:90)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:267)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:639)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:331)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:391)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>   at 
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:756)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1152)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1313)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:161)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:140)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:136)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testExcessReservationThanNodeManagerCapacity(TestContainerAllocation.java:90)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14974) org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation fails in trunk

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-14974:
--

HADOOP-14954 never made it to a release. And there's no other patch for this 
JIRA in the 3.1.0 release. Reopening and closing this instead as a dup.

> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
>  fails in trunk
> ---
>
> Key: HADOOP-14974
> URL: https://issues.apache.org/jira/browse/HADOOP-14974
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: John Zhuge
>Priority: Blocker
> Fix For: 3.1.0
>
>
> {code}
> org.apache.hadoop.metrics2.MetricsException: Metrics source 
> QueueMetrics,q0=root already exists!
>   at 
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
>   at 
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
>   at 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:239)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:141)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:131)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:90)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:267)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:639)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:331)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:391)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>   at 
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:756)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1152)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.serviceInit(MockRM.java:1313)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:161)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:140)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.(MockRM.java:136)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testExcessReservationThanNodeManagerCapacity(TestContainerAllocation.java:90)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14714) handle InternalError in bulk object delete through retries

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-14714:
--

Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any 
patch for this in the source-code. Please revert back if this is incorrect.

> handle InternalError in bulk object delete through retries
> --
>
> Key: HADOOP-14714
> URL: https://issues.apache.org/jira/browse/HADOOP-14714
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> There's some more detail appearing on HADOOP-11572 about the errors seen 
> here; sounds like its large fileset related (or just probability working 
> against you). Most importantly: retries may make it go away. 
> Proposed: implement a retry policy.
> Issue: delete is not idempotent, not if someone else adds things.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14714) handle InternalError in bulk object delete through retries

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-14714.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> handle InternalError in bulk object delete through retries
> --
>
> Key: HADOOP-14714
> URL: https://issues.apache.org/jira/browse/HADOOP-14714
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> There's some more detail appearing on HADOOP-11572 about the errors seen 
> here; sounds like its large fileset related (or just probability working 
> against you). Most importantly: retries may make it go away. 
> Proposed: implement a retry policy.
> Issue: delete is not idempotent, not if someone else adds things.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14381) S3AUtils.translateException to map 503 reponse to => throttling failure

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-14381.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> S3AUtils.translateException to map 503 reponse to => throttling failure
> ---
>
> Key: HADOOP-14381
> URL: https://issues.apache.org/jira/browse/HADOOP-14381
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> When AWS S3 returns "503", it means that the overall set of requests on a 
> part of an S3 bucket exceeds the permitted limit; the client(s) need to 
> throttle back or away for some rebalancing to complete.
> The aws SDK retries 3 times on a 503, but then throws it up. Our code doesn't 
> do anything with that other than create a generic {{AWSS3IOException}}.
> Proposed
> * add a new exception, {{AWSOverloadedException}}
> * raise it on a 503 from S3 (& for s3guard, on DDB complaints)
> * have it include a link to a wiki page on the topic, as well as the path
> * and any other diags
> Code talking to S3 may then be able to catch this and choose to react. Some 
> retry with exponential backoff is the obvious option. Failing, well, that 
> could trigger task reattempts at that part of the query, then job retry 
> —which will again fail, *unless the number of tasks run in parallel is 
> reduced*
> As this throttling is across all clients talking to the same part of a 
> bucket, fixing it is potentially a high level option. We can at least start 
> by reporting things better



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14381) S3AUtils.translateException to map 503 reponse to => throttling failure

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-14381:
--

Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any 
patch for this in the source-code. Please revert back if this is incorrect.

> S3AUtils.translateException to map 503 reponse to => throttling failure
> ---
>
> Key: HADOOP-14381
> URL: https://issues.apache.org/jira/browse/HADOOP-14381
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> When AWS S3 returns "503", it means that the overall set of requests on a 
> part of an S3 bucket exceeds the permitted limit; the client(s) need to 
> throttle back or away for some rebalancing to complete.
> The aws SDK retries 3 times on a 503, but then throws it up. Our code doesn't 
> do anything with that other than create a generic {{AWSS3IOException}}.
> Proposed
> * add a new exception, {{AWSOverloadedException}}
> * raise it on a 503 from S3 (& for s3guard, on DDB complaints)
> * have it include a link to a wiki page on the topic, as well as the path
> * and any other diags
> Code talking to S3 may then be able to catch this and choose to react. Some 
> retry with exponential backoff is the obvious option. Failing, well, that 
> could trigger task reattempts at that part of the query, then job retry 
> —which will again fail, *unless the number of tasks run in parallel is 
> reduced*
> As this throttling is across all clients talking to the same part of a 
> bucket, fixing it is potentially a high level option. We can at least start 
> by reporting things better



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13205) S3A to support custom retry policies; failfast on unknown host

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-13205.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> S3A to support custom retry policies; failfast on unknown host
> --
>
> Key: HADOOP-13205
> URL: https://issues.apache.org/jira/browse/HADOOP-13205
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Noticed today that when connections are down, S3A retries on 
> UnknownHostExceptions logging noisily in the process.
> # it should be possible to define or customize retry policies for an FS 
> instance (fail fast, exponential backoff, etc)
> # we may want to explicitly have a fail-fast-if-offline retry policy, 
> catching the common connectivity ones.
> Testing will be fun here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-13205) S3A to support custom retry policies; failfast on unknown host

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-13205:
--

Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any 
patch for this JIRA in the source-code. Please revert back if this is incorrect.

> S3A to support custom retry policies; failfast on unknown host
> --
>
> Key: HADOOP-13205
> URL: https://issues.apache.org/jira/browse/HADOOP-13205
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.1.0
>
>
> Noticed today that when connections are down, S3A retries on 
> UnknownHostExceptions logging noisily in the process.
> # it should be possible to define or customize retry policies for an FS 
> instance (fail fast, exponential backoff, etc)
> # we may want to explicitly have a fail-fast-if-offline retry policy, 
> catching the common connectivity ones.
> Testing will be fun here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-13811:
--

Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any 
patch for this JIRA in the source-code. Please revert back if this is incorrect.

> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to 
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting 
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document 
> destined for handler class}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-13811.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to 
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting 
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document 
> destined for handler class}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-14303) Review retry logic on all S3 SDK calls, implement where needed

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-14303:
--

Reopening and closing this instead as a dup of HADOOP-13786 as I don't find any 
patch for this in the source-code. Please revert back if this is incorrect.

> Review retry logic on all S3 SDK calls, implement where needed
> --
>
> Key: HADOOP-14303
> URL: https://issues.apache.org/jira/browse/HADOOP-14303
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> AWS S3, IAM, KMS, DDB etc all throttle callers: the S3A code needs to handle 
> this without failing, as if it slows down its requests it can recover.
> 1. Look at all the places where we are calling S3A via the AWS SDK and make 
> sure we are retrying with some backoff & jitter policy, ideally something 
> unified. This must be more systematic than the case-by-case, 
> problem-by-problem strategy we are implicitly using.
> 2. Many of the AWS S3 SDK calls do implement retry (e.g PUT/multipart PUT), 
> but we need to check the other parts of the process: login, initiate/complete 
> MPU, ...
> Related
> HADOOP-13811 Failed to sanitize XML document destined for handler class
> HADOOP-13664 S3AInputStream to use a retry policy on read failures
> This stuff is all hard to test. A key need is to be able to differentiate 
> recoverable throttle & network failures from unrecoverable problems like: 
> auth, network config (e.g bad endpoint), etc.
> May be the opportunity to add a faulting subclass of Amazon S3 client which 
> can be configured in IT Tests to fail at specific points. Ryan Blue's mcok S3 
> client does this in HADOOP-13786, but it is for 100% mock. I'm thinking of 
> something with similar fault raising, but in front of the real S3A client 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14303) Review retry logic on all S3 SDK calls, implement where needed

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-14303.
--
   Resolution: Duplicate
Fix Version/s: (was: 3.1.0)

> Review retry logic on all S3 SDK calls, implement where needed
> --
>
> Key: HADOOP-14303
> URL: https://issues.apache.org/jira/browse/HADOOP-14303
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> AWS S3, IAM, KMS, DDB etc all throttle callers: the S3A code needs to handle 
> this without failing, as if it slows down its requests it can recover.
> 1. Look at all the places where we are calling S3A via the AWS SDK and make 
> sure we are retrying with some backoff & jitter policy, ideally something 
> unified. This must be more systematic than the case-by-case, 
> problem-by-problem strategy we are implicitly using.
> 2. Many of the AWS S3 SDK calls do implement retry (e.g PUT/multipart PUT), 
> but we need to check the other parts of the process: login, initiate/complete 
> MPU, ...
> Related
> HADOOP-13811 Failed to sanitize XML document destined for handler class
> HADOOP-13664 S3AInputStream to use a retry policy on read failures
> This stuff is all hard to test. A key need is to be able to differentiate 
> recoverable throttle & network failures from unrecoverable problems like: 
> auth, network config (e.g bad endpoint), etc.
> May be the opportunity to add a faulting subclass of Amazon S3 client which 
> can be configured in IT Tests to fail at specific points. Ryan Blue's mcok S3 
> client does this in HADOOP-13786, but it is for 100% mock. I'm thinking of 
> something with similar fault raising, but in front of the real S3A client 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

Thanks for reporting this.

It's straight forward. Done, commit pushed.

Thanks
+Vinod

> On Mar 19, 2018, at 2:54 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote:
> 
> I just pulled the latest from branch-3.1 and noticed that now that it was
> reset to trunk, the version in the pom.xml files is 3.2.0-SNAPSHOT instead
> of 3.1.0-SNAPSHOT. Is somebody going to submit a new commit to change this
> version back to 3.1.0-SNAPSHOT in this branch?
> 
> Thank you,
> Jonathan
> 
> On Mon, Mar 19, 2018 at 11:43 AM Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
> 
>> Thanks Wangda.
>> 
>> 
>> On 3/19/18, 11:38 AM, "Wangda Tan" <wheele...@gmail.com> wrote:
>> 
>>Done JIRA fix version update:
>> 
>>Moved all JIRAs with fixVersion = 3.2.0 to 3.1.0 except following few
>> fixes
>>(which committed after 49c747ab187d0650143205ba57ca19607ec4c6bd)
>> 
>>YARN-8002. Support NOT_SELF and ALL namespace types for allocation
>> tag.
>>(Weiwe
>>i Yang via wangda)
>> 
>>HADOOP-15262. AliyunOSS: move files under a directory in parallel
>> when
>>rename a directory. Contributed by Jinhu Wu.
>> 
>>MAPREDUCE-7066. TestQueue fails on Java9
>> 
>>YARN-8028. Support authorizeUserAccessToQueue in RMWebServices.
>>Contributed by Wangda Tan.
>> 
>>YARN-8040. [UI2] New YARN UI webapp does not respect current
>> pathname
>>for REST api. Contributed by Sunil G.
>> 
>>Thanks,
>>Wangda
>> 
>>On Mon, Mar 19, 2018 at 11:12 AM, Wangda Tan <wheele...@gmail.com>
>> wrote:
>> 
>>> Thanks Akira for the additional vote,
>>> 
>>> With help from Apache Infra Team (Daniel Takamori), we just reset
>>> branch-3.1 to trunk (SHA: 49c747ab187d0650143205ba57ca19607ec4c6bd).
>> Will
>>> update JIRA fix version shortly.
>>> 
>>> - Wangda
>>> 
>>> On Sun, Mar 18, 2018 at 6:10 PM, Akira Ajisaka <
>> ajisa...@oss.nttdata.co.jp
>>>> wrote:
>>> 
>>>> +1 for resetting branch-3.1.
>>>> 
>>>> Thanks,
>>>> Akira
>>>> 
>>>> 
>>>> On 2018/03/18 12:51, Wangda Tan wrote:
>>>> 
>>>>> Thanks for sharing your thoughts.
>>>>> 
>>>>> We have done build and single node cluster deploy / test for the
>> latest
>>>>> trunk code (commit: 49c747ab187d0650143205ba57ca19607ec4c6bd).
>> Since
>>>>> there
>>>>> are no objections, so I will go ahead to do the branch replace.
>>>>> 
>>>>> Since we don't have force push permission to release branches. I
>> just
>>>>> filed
>>>>> https://issues.apache.org/jira/browse/INFRA-16204 to get help from
>>>>> Apache
>>>>> infra team.
>>>>> 
>>>>> Please hold any commits to branch-3.1, will keep this email thread
>>>>> posted.
>>>>> 
>>>>> Best,
>>>>> Wangda
>>>>> 
>>>>> On Wed, Mar 14, 2018 at 3:14 PM, Vinod Kumar Vavilapalli <
>>>>> vino...@apache.org
>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>> 
>>>>> I see one new feature:
>> https://issues.apache.org/jira/browse/YARN-7626:
>>>>>> Allow regular expression matching in container-executor.cfg for
>> devices
>>>>>> and
>>>>>> named docker volumes mount.
>>>>>> 
>>>>>> There are 21 sub-tasks. There are three feature-type JIRAs in
>> those -
>>>>>> https://issues.apache.org/jira/browse/YARN-7972,
>>>>>> https://issues.apache.org/jira/browse/YARN-7891 and
>>>>>> https://issues.apache.org/jira/browse/YARN-5015. These should be
>> okay -
>>>>>> not major disrupting features.
>>>>>> 
>>>>>> Everything else is either a bug-fix or an improvement so we
>> should be
>>>>>> good.
>>>>>> 
>>>>>> From the list, it doesn't look like resetting will destabilize
>> 3.1, +1
>>>>>> for
>>>>>> doing this.
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>> On Mar

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

I see one new feature: https://issues.apache.org/jira/browse/YARN-7626: Allow 
regular expression matching in container-executor.cfg for devices and named 
docker volumes mount.

There are 21 sub-tasks. There are three feature-type JIRAs in those - 
https://issues.apache.org/jira/browse/YARN-7972, 
https://issues.apache.org/jira/browse/YARN-7891 and 
https://issues.apache.org/jira/browse/YARN-5015. These should be okay - not 
major disrupting features.

Everything else is either a bug-fix or an improvement so we should be good.

From the list, it doesn't look like resetting will destabilize 3.1, +1 for 
doing this.

Thanks
+Vinod

> On Mar 14, 2018, at 1:54 PM, Wangda Tan  wrote:
> 
> Hi mapreduce/yarn/common/hdfs-devs, 
> 
> As of now, we have all blockers done for 3.1.0 release [1]. The release is 
> running behind schedule due to a few security-related issues. Because of this 
> and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is already 
> diverging. There're 64 commits in trunk but not in branch-3.1. [2]
> 
> I took a quick scan of them, most of them are good fixes which we should 
> bring to 3.1.0 as well. And this can also reduce differences between 3.2.0 
> and 3.1.0 release for less maintenance burden in the future.
> 
> Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days and cut 
> RC after that.
> 
> Thoughts?
> 
> - Wangda
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, 
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0 ORDER BY 
> priority DESC
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) AND 
> fixVersion not in (3.1.0)


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Vinod Kumar Vavilapalli]

Was out of country and away from email for weeks.

We have been using this in the past: 
https://www.meetup.com/Hadoop-Contributors/ 
. I believe this doesn't have the 
per-meetup-charge issue, but will double check. Let me know if there are more 
meetups planned in the near future and we can use this.

Thanks
+Vinod

> On Mar 6, 2018, at 7:48 PM, Chris Douglas  wrote:
> 
> Found a meetup alternative (thanks Subru):
> https://meetingstar.io/event/fk13172f1d75KN 
> 
> 
> So we can get a rough headcount, please add (local) if you plan to
> attend in-person. -C
> 
> 
> On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas  > wrote:
>> [Cross-posting, as this affects the rest of the project]
>> 
>> Hey folks-
>> 
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>> 
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>> 
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>> 
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>> 
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>> 
>> [1]: https://s.apache.org/nEoQ
>> [2]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org 
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Yes, JIRAs will be filed, the wiki-page idea from YARN meetup is to record all 
combinations of testing that need to be done and correspondingly capture all 
the testing that someone in the community has already done and record it for 
future perusal.

From what you are saying, I guess we haven't advertised to the public yet on 
rolling upgrades, but in our meetups etc so far, you have been saying that 
rolling upgrades is supported - so I assumed we did put it in our messaging.

The important question is if we are or are not allowed to make potentially 
incompatible changes to fix bugs in the process of supporting 2.x to 3.x 
upgrades whether rolling or not.

+Vinod

> On Dec 13, 2017, at 1:05 PM, Andrew Wang  wrote:
> 
> I'm hoping we can address YARN-7588 and any remaining rolling upgrade issues 
> in 3.0.x maintenance releases. Beyond a wiki page, it would be really great 
> to get JIRAs filed and targeted for tracking as soon as possible.
> 
> Vinod, what do you think we need to do regarding caveating rolling upgrade 
> support? We haven't advertised rolling upgrade support between major releases 
> outside of dev lists and JIRA. As a new major release, our compat guidelines 
> allow us to break compatibility, so I don't think it's expected by users.
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Good stuff Andrew, and thanks everyone!

+Vinod

> On Dec 13, 2017, at 1:05 PM, Andrew Wang  wrote:
> 
> To close this out, the vote passes successfully with 13 binding +1s, 5 
> non-binding +1s, and no -1s. Thanks everyone for voting! I'll work on staging.
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

I was waiting for Daniel to post the minutes from YARN meetup to talk about 
this. Anyways, in that discussion, we identified a bunch of key upgrade related 
scenarios that no-one seems to have validated - atleast from the representation 
in the YARN meetup. I'm going to create a wiki-page listing all these scenarios.

But back to the bug that Junping raised. At this point, we don't have a clear 
path towards running 2.x applications on 3.0.0 clusters. So, our claim of 
rolling-upgrades already working is not accurate.

One of the two options that Junping proposed should be pursued before we close 
the release. I'm in favor of calling out rolling-upgrade support be with-drawn 
or caveated and push for progress instead of blocking the release.

Thanks
+Vinod

> On Dec 12, 2017, at 5:44 PM, Junping Du  wrote:
> 
> Thanks Andrew for pushing new RC for 3.0.0. I was out last week, just get 
> chance to validate new RC now.
> 
> Basically, I found two critical issues with the same rolling upgrade scenario 
> as where HADOOP-15059 get found previously:
> HDFS-12920, we changed value format for some hdfs configurations that old 
> version MR client doesn't understand when fetching these configurations. Some 
> quick workarounds are to add old value (without time unit) in hdfs-site.xml 
> to override new default values but will generate many annoying warnings. I 
> provided my fix suggestions on the JIRA already for more discussion.
> The other one is YARN-7646. After we workaround HDFS-12920, will hit the 
> issue that old version MR AppMaster cannot communicate with new version of 
> YARN RM - could be related to resource profile changes from YARN side but 
> root cause are still in investigation.
> 
> The first issue may not belong to a blocker given we can workaround this 
> without code change. I am not sure if we can workaround 2nd issue so far. If 
> not, we may have to fix this or compromise with withdrawing support of 
> rolling upgrade or calling it a stable release.
> 
> 
> Thanks,
> 
> Junping
> 
> 
> From: Robert Kanter 
> Sent: Tuesday, December 12, 2017 3:10 PM
> To: Arun Suresh
> Cc: Andrew Wang; Lei Xu; Wei-Chiu Chuang; Ajay Kumar; Xiao Chen; Aaron T. 
> Myers; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC1
> 
> +1 (binding)
> 
> + Downloaded the binary release
> + Deployed on a 3 node cluster on CentOS 7.3
> + Ran some MR jobs, clicked around the UI, etc
> + Ran some CLI commands (yarn logs, etc)
> 
> Good job everyone on Hadoop 3!
> 
> 
> - Robert
> 
> On Tue, Dec 12, 2017 at 1:56 PM, Arun Suresh  wrote:
> 
>> +1 (binding)
>> 
>> - Verified signatures of the source tarball.
>> - built from source - using the docker build environment.
>> - set up a pseudo-distributed test cluster.
>> - ran basic HDFS commands
>> - ran some basic MR jobs
>> 
>> Cheers
>> -Arun
>> 
>> On Tue, Dec 12, 2017 at 1:52 PM, Andrew Wang 
>> wrote:
>> 
>>> Hi everyone,
>>> 
>>> As a reminder, this vote closes tomorrow at 12:31pm, so please give it a
>>> whack if you have time. There are already enough binding +1s to pass this
>>> vote, but it'd be great to get additional validation.
>>> 
>>> Thanks to everyone who's voted thus far!
>>> 
>>> Best,
>>> Andrew
>>> 
>>> 
>>> 
>>> On Tue, Dec 12, 2017 at 11:08 AM, Lei Xu  wrote:
>>> 
 +1 (binding)
 
 * Verified src tarball and bin tarball, verified md5 of each.
 * Build source with -Pdist,native
 * Started a pseudo cluster
 * Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run hdfs
 dfs put/get/cat on "/" with XOR-2-1 policy.
 
 Thanks Andrew for this great effort!
 
 Best,
 
 
 On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang >> 
 wrote:
> Hi Wei-Chiu,
> 
> The patchprocess directory is left over from the create-release
>>> process,
> and it looks empty to me. We should still file a create-release JIRA
>> to
 fix
> this, but I think this is not a blocker. Would you agree?
> 
> Best,
> Andrew
> 
> On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang <
>> weic...@cloudera.com
 
> wrote:
> 
>> Hi Andrew, thanks the tremendous effort.
>> I found an empty "patchprocess" directory in the source tarball,
>> that
>>> is
>> not there if you clone from github. Any chance you might have some
 leftover
>> trash when you made the tarball?
>> Not wanting to nitpicking, but you might want to double check so we
 don't
>> ship anything private to you in public :)
>> 
>> 
>> 
>> On Tue, Dec 12, 2017 at 7:48 AM, Ajay Kumar <
>>> ajay.ku...@hortonworks.com
> 
>> wrote:
>> 
>>> +1 (non-binding)
>>> Thanks for driving 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Looked at RC1. Went through my usual check-list. Here's my summary.

+1 (binding) overall

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
- [Check] Generating dist tarballs from source tar-ball
- [Check] Validating the layout of the binary tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

Few issues as before found during testing, but shouldn't be blockers
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty. Will file a ticket.
 - resourcemanager-metrics.out is going into current directory instead of log 
directory. Will file a ticket.

One thing I want to make sure folks agree to. Allen filed a ticket to remove 
yarn-historyserver option per our previous discussion - 
https://issues.apache.org/jira/browse/YARN-7588 
. It isn't done - I am hoping 
this 'incompatible change' can be put in 3.0.1 and not 4.0.

Thanks
+Vinod


> On Dec 8, 2017, at 12:31 PM, Andrew Wang  wrote:
> 
> Hi all,
> 
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
> 
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
> 
> You can find the artifacts here:
> 
> http://home.apache.org/~wang/3.0.0-RC1/
> 
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
> 
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
> 
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Missed this response on the old thread, but closing the loop here..

The incompatibility conundrum with Dot-zeroes did indeed happen, in early 2.x 
releases - multiple times at that. And the downstream projects did raise 
concerns at these unfixable situations.

I wasn't advocating a new formalism, this was more of a lesson taken from real 
life experience that I wanted share with fellow RMs - as IMO the effort was 
worth the value for the releases where I used it.

If RMs of these more recent releases choose to not do this if it is perceived 
that a release won't run into those past issues at all, it's clearly their 
call. It's just that we are bound to potentially make the same mistakes and 
learn the same lesson all over again..

+Vinod

> On Nov 9, 2017, at 9:51 AM, Chris Douglas <cdoug...@apache.org> wrote:
> 
> The labor required for these release formalisms is exceeding their
> value. Our minor releases have more bugs than our patch releases (we
> hope), but every consumer should understand how software versioning
> works. Every device I own has bugs on major OS updates. That doesn't
> imply that every minor release is strictly less stable than a patch
> release, and users need to be warned off it.
> 
> In contrast, we should warn users about features that compromise
> invariants like security or durability, either by design or due to
> their early stage of development. We can't reasonably expect them to
> understand those tradeoffs, since they depend on internal details of
> Hadoop.
> 
> On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli
> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
>> When we tried option (b), we used to make .0 as a GA release, but downstream 
>> projects like Tez, Hive, Spark would come back and find an incompatible 
>> change - and now we were forced into a conundrum - is fixing this 
>> incompatible change itself an incompatibility?
> 
> Every project takes these case-by-case. Most of the time we'll
> accommodate the old semantics- and we try to be explicit where we
> promise compatibility- but this isn't a logic problem, it's a
> practical one. If it's an easy fix to an obscure API, we probably
> won't even hear about it.
> 
>> Long story short, I'd just add to your voting thread and release notes that 
>> 2.9.0 still needs to be tested downstream and so users may want to wait for 
>> subsequent point releases.
> 
> It's uncomfortable to have four active release branches, with 3.1
> coming in early 2018. We all benefit from the shared deployment
> experiences that harden these releases, and fragmentation creates
> incentives to compete for that attention. Rather than tacitly
> scuffling over waning interest in the 2.x series, I'd endorse your
> other thread encouraging consolidation around 3.x.
> 
> To that end, there is no policy or precedent that requires that new
> minor releases be labeled as "alpha". If there is cause to believe
> that 2.9.0 is not ready to release in the stable line, then we
> shouldn't release it. -C
> 
>>> On Nov 8, 2017, at 12:43 AM, Subru Krishnan <su...@apache.org> wrote:
>>> 
>>> We are canceling the RC due to the issue that Rohith/Sunil identified. The
>>> issue was difficult to track down as it only happens when you use IP for ZK
>>> (works fine with host names) and moreover if ZK and RM are co-located on
>>> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
>>> 
>>> Thanks to everyone for the extensive testing/validation. Hopefully cost to
>>> replicate with RC1 is much lower.
>>> 
>>> -Subru/Arun.
>>> 
>>> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos <kkarana...@gmail.com
>>>> wrote:
>>> 
>>>> +1 from me too.
>>>> 
>>>> Did the following:
>>>> 1) set up a 9-node cluster;
>>>> 2) ran some Gridmix jobs;
>>>> 3) ran (2) after enabling opportunistic containers (used a mix of
>>>> guaranteed and opportunistic containers for each job);
>>>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>>>> containers.
>>>> 
>>>> All the above worked with no issues.
>>>> 
>>>> Thanks for all the effort guys!
>>>> 
>>>> Konstantinos
>>>> 
>>>> 
>>>> 
>>>> Konstantinos
>>>> 
>>>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger <ebad...@oath.com.invalid>
>>>> wrote:
>>>> 
>>>>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
>>>>> 
>>>>&

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
>> work. Not just deprecated in favor of timelineserver as was advertised.
> 
>   This works for me in trunk and the bash code doesn’t appear to have 
> changed in a very long time.  Probably something local to your install.  (I 
> do notice that the deprecation message says “starting” which is awkward when 
> the stop command is given though.)  Also: is the deprecation message even 
> true at this point?


Sorry, I mischaracterized the problem.

The real issue is that I cannot use this command line when the MapReduce 
JobHistoryServer is already started on the same machine.

~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.
DEPRECATED: Use of this command to start the timeline server is deprecated.
Instead use the timelineserver command for it.
Starting the History Server anyway...
historyserver is running as process 86156.  Stop it first.

So, it looks like in shell-scripts, there can ever be only one daemon of a 
given name, irrespective of which daemon scripts are invoked.

We need to figure out two things here
 (a) The behavior of this command. Clearly, it will conflict with the MapReduce 
JHS - only one of them can be started on the same node.
 (b) We need to figure out if this V1 TimelineService should even be support 
given ATSv2.

@Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please comment 
on (b).

Thanks
+Vinod

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>> 
>> 
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>> 
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
> 
>   -1 on making yarn-ui always build.
> 
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:
>> Info :-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
>> not in the CLI node -list
>> 
> 
> Is this a blocker? Could we get a JIRA?


I just cross verified this with others - looks like an issue with my 
environment - not a blocker, ignore.

+Vinod
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

 - [Check] Successful recompilation from source tar-ball
 - [Check] Signature verification
 - [Check] Generating dist tarballs from source tar-ball
 - [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
 - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
 - One decommissioned node in YARN ResourceManager UI always appears to start 
with, even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI 
node -list

Minor
 - resourcemanager-metrics.out is going into current directory instead of log 
directory
 - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work. 
Not just deprecated in favor of timelineserver as was advertised.
 - Spurious warnings on CLI
17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

 - When did we stop putting CHANGES files into the source artifacts?
 - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see 
how this can be avoided.
 - Compatibility notes
-- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME is 
not automatically inherited. Correct behavior)
-- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Compilation passed for me. Using jdk1.8.0_40.jdk.

+Vinod

> On Nov 20, 2017, at 4:16 PM, Wei-Chiu Chuang <weic...@cloudera.com> wrote:
> 
> @vinod
> I followed your command but I could not reproduce your problem.
> 
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
> ommon/target/hadoop-common-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
> hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
> s/target/hadoop-hdfs-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
> hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
> 
> During compilation I found the following error with a Java 1.8.0_5 JDK:
> 
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven
> -compiler-plugin:3.1:testCompile (default-testCompile) on project
> hadoop-aws: Compilation failure: Compilation failure:
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> 
> And then I realized Ray filed HADOOP-14900
> <https://issues.apache.org/jira/browse/HADOOP-14900> for the same
> issue. This problem is not reproducible with a more recent JDK 8, such as
> 1.8.0_151
> Maybe it would be a good idea to name a list of JDKs that are known to be
> buggy. Can we get this documented somewhere? I don't consider it a blocker
> so a release note in a later release or a wiki entry should be good enough.
> 
> On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> 
>> Quick question.
>> 
>> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
>> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
>> voted on (hadoop-3.0.0-src.tar.gz).
>> 
>> The idea is to install HDFS, YARN, MR separately in separate
>> root-directories from the generated individual dist tarballs.
>> 
>> But now I see that HDFS and common dist tarballs are empty
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
>> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
>> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>> 
>> But YARN and MR are fine
>> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
>> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
>> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
>> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>> 
>> Is it just me? Or is this broken?
>> 
>> Thanks
>> +Vinod
>> 
>>> On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.w...@cloude

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Quick question.

I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being voted 
on (hadoop-3.0.0-src.tar.gz).

The idea is to install HDFS, YARN, MR separately in separate root-directories 
from the generated individual dist tarballs.

But now I see that HDFS and common dist tarballs are empty
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz - 
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

But YARN and MR are fine
-rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
-rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz

Is it just me? Or is this broken?

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15041) XInclude support in .xml configuration file is broken after "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved HADOOP-15041.
--
Resolution: Duplicate

Closing instead as a duplicate.

{code}
commit 9a89a3e0b09a2eae6aabadd510fa48c5e18c0dcc
Author: Arun Suresh <asur...@apache.org>
Date:   Mon Nov 13 13:49:08 2017 -0800

Addendum patch for Configuration fix. (Jason Lowe via asuresh)

(cherry picked from commit 4e847d63a39268d8a34e0f22ddb5e40c5ef71e3a)
{code}

> XInclude support in .xml configuration file is broken after 
> "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit
> -
>
> Key: HADOOP-15041
> URL: https://issues.apache.org/jira/browse/HADOOP-15041
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: SammiChen
>Assignee: Arun Suresh
>Priority: Critical
>
> XInclude support in .xml configuration file is broken after following 
> check-in.  
> Since there is no JIRA number in the following commit message, create a new 
> JIRA to track the issue. 
> commit 5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1
> Author: Arun Suresh <asur...@apache.org>
> Date:   Thu Nov 9 15:15:51 2017 -0800
> Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-15041) XInclude support in .xml configuration file is broken after "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened HADOOP-15041:
--

> XInclude support in .xml configuration file is broken after 
> "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit
> -
>
> Key: HADOOP-15041
> URL: https://issues.apache.org/jira/browse/HADOOP-15041
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: SammiChen
>Assignee: Arun Suresh
>Priority: Critical
>
> XInclude support in .xml configuration file is broken after following 
> check-in.  
> Since there is no JIRA number in the following commit message, create a new 
> JIRA to track the issue. 
> commit 5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1
> Author: Arun Suresh <asur...@apache.org>
> Date:   Thu Nov 9 15:15:51 2017 -0800
> Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

I'd definitely extend it for a few more days. I only see 3 binding +1s so far - 
not a great number to brag about on our first major release in years.

Also going to nudge folks into voting.

+Vinod

> On Nov 17, 2017, at 3:26 PM, Andrew Wang  wrote:
> 
> Hi Arpit,
> 
> I agree the timing is not great here, but extending it to meaningfully
> avoid the holidays would mean extending it an extra week (e.g. to the
> 29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
> really like to get the RC out before then.
> 
> In terms of downstream testing, we've done extensive integration testing
> with downstreams via the alphas and betas, and we have continuous
> integration running at Cloudera against branch-3.0. Because of this, I have
> more confidence in our integration for 3.0.0 than most Hadoop releases.
> 
> Is it meaningful to extend to say, the 21st, which provides for a full week
> of voting?
> 
> Best,
> Andrew
> 
> On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
> wrote:
> 
>> Hi Andrew,
>> 
>> Thank you for your hard work in getting us to this step. This is our first
>> major GA release in many years.
>> 
>> I feel a 5-day vote window ending over the weekend before thanksgiving may
>> not provide sufficient time to evaluate this RC especially for downstream
>> components.
>> 
>> Would you please consider extending the voting deadline until a few days
>> after the thanksgiving holiday? It would be a courtesy to our broader
>> community and I see no harm in giving everyone a few days to evaluate it
>> more thoroughly.
>> 
>> On a lighter note, your deadline is also 4 minutes short of the required 5
>> days. :)
>> 
>> Regards,
>> Arpit
>> 
>> 
>> 
>> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>> 
>>Hi folks,
>> 
>>Thanks as always to the many, many contributors who helped with this
>>release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>available here:
>> 
>>http://people.apache.org/~wang/3.0.0-RC0/
>> 
>>This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> 
>>3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>additions include the merge of YARN resource types, API-based
>> configuration
>>of the CapacityScheduler, and HDFS router-based federation.
>> 
>>I've done my traditional testing with a pseudo cluster and a Pi job.
>> My +1
>>to start.
>> 
>>Best,
>>Andrew
>> 
>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

A related point - I thought I mentioned this in one of the release preparation 
threads, but in any case.

Starting 2.7.0, for every .0 release, we've been adding a disclaimer (to the 
voting thread as well as the final release) that the first release can 
potentially go through additional fixes to incompatible changes (besides 
stabilization fixes). We should do this with 2.9.0 too.

This has some history - long before this, we tried two different things: (a) 
downstream projects consume an RC (b) downstream projects consume a release. 
Option (a) was tried many times but it was increasingly getting hard to manage 
this across all the projects that depend on Hadoop. When we tried option (b), 
we used to make .0 as a GA release, but downstream projects like Tez, Hive, 
Spark would come back and find an incompatible change - and now we were forced 
into a conundrum - is fixing this incompatible change itself an 
incompatibility? So to avoid this problem, we've started marking the first few 
releases as alpha eventually making a stable point release. Clearly, specific 
users can still use this in production as long as we the Hadoop community 
reserve the right to fix incompatibilities.

Long story short, I'd just add to your voting thread and release notes that 
2.9.0 still needs to be tested downstream and so users may want to wait for 
subsequent point releases.

Thanks
+Vinod

> On Nov 8, 2017, at 12:43 AM, Subru Krishnan  wrote:
> 
> We are canceling the RC due to the issue that Rohith/Sunil identified. The
> issue was difficult to track down as it only happens when you use IP for ZK
> (works fine with host names) and moreover if ZK and RM are co-located on
> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
> 
> Thanks to everyone for the extensive testing/validation. Hopefully cost to
> replicate with RC1 is much lower.
> 
> -Subru/Arun.
> 
> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos > wrote:
> 
>> +1 from me too.
>> 
>> Did the following:
>> 1) set up a 9-node cluster;
>> 2) ran some Gridmix jobs;
>> 3) ran (2) after enabling opportunistic containers (used a mix of
>> guaranteed and opportunistic containers for each job);
>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>> containers.
>> 
>> All the above worked with no issues.
>> 
>> Thanks for all the effort guys!
>> 
>> Konstantinos
>> 
>> 
>> 
>> Konstantinos
>> 
>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
>> wrote:
>> 
>>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
>>> 
>>> - Verified all hashes and checksums
>>> - Built from source on macOS 10.12.6, Java 1.8.0u65
>>> - Deployed a pseudo cluster
>>> - Ran some example jobs
>>> 
>>> Thanks,
>>> 
>>> Eric
>>> 
>>> On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan  wrote:
>>> 
 Sunil / Rohith,
 
 Could you check if your configs are same as Jonathan posted configs?
 https://issues.apache.org/jira/browse/YARN-7453?
>>> focusedCommentId=16242693&
 page=com.atlassian.jira.plugin.system.issuetabpanels:
 comment-tabpanel#comment-16242693
 
 And could you try if using Jonathan's configs can still reproduce the
 issue?
 
 Thanks,
 Wangda
 
 
 On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
>> wrote:
 
> Thanks for testing Rohith and Sunil
> 
> Can you please confirm if it is not a config issue at your end ?
> We (both Jonathan and myself) just tried testing this on a fresh
>>> cluster
> (both automatic and manual) and we are not able to reproduce this.
>> I've
> updated the YARN-7453 > jira/browse/YARN-7453
 
> JIRA
> with details of testing.
> 
> Cheers
> -Arun/Subru
> 
> On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
> rohithsharm...@apache.org
>> wrote:
> 
>> Thanks Sunil for confirmation. Btw, I have raised YARN-7453
>>  JIRA to track
>> this
>> issue.
>> 
>> - Rohith Sharma K S
>> 
>> On 7 November 2017 at 16:44, Sunil G  wrote:
>> 
>>> Hi Subru and Arun.
>>> 
>>> Thanks for driving 2.9 release. Great work!
>>> 
>>> I installed cluster built from source.
>>> - Ran few MR jobs with application priority enabled. Runs fine.
>>> - Accessed new UI and it also seems fine.
>>> 
>>> However I am also getting same issue as Rohith reported.
>>> - Started an HA cluster
>>> - Pushed RM to standby
>>> - Pushed back RM to active then seeing an exception.
>>> 
>>> org.apache.hadoop.ha.ServiceFailedException: RM could not
>>> transition
 to
>>> Active
>>>at
>>> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyE
>>> lectorBasedElectorServic
>>>

Re: [DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Thanks for your comments, Zheng. Replies inline.


> On the other hand, I've discussed with quite a few 3.0 potential users, it 
> looks like most of them are interested in the erasure coding feature and a 
> major scenario for that is to back up their large volume of data to save 
> storage cost. They might run analytics workload using Hive, Spark, Impala and 
> Kylin on the new cluster based on the version, but it's not a must at the 
> first time. They understand there might be some gaps so they'd migrate their 
> workloads incrementally. For the major analytics workload, we've performed 
> lots of benchmark and integration tests as well as other sides I believe, we 
> did find some issues but they should be fixed in downstream projects. I 
> thought the release of GA will accelerate the progress and expose the issues 
> if any. We couldn't wait for it being matured. There isn't perfectness.


3.0 is a GA release from the Apache Hadoop community. So, we cannot assume that 
all usages in the short term are *only* going to be for storage optimization 
features and only on dedicated clusters. We have to make sure that the 
workloads can be migrated right now and/or that existing clusters can be 
upgraded in-place. If not, we shouldn't be calling it GA.


> This sounds a good consideration. I'm thinking if I'm a Hadoop user, for 
> example, I'm using 2.7.4 or 2.8.2 or whatever 2.x version, would I first 
> upgrade to this bridging release then use the bridge support to upgrade to 
> 3.x version? I'm not sure. On the other hand, I might tend to look for some 
> guides or supports in 3.x docs about how to upgrade from 2.7 to 3.x. 



Arun Suresh also asked this same question earlier. I think this will really 
depend on what we discover as part of the migration and user-acceptance 
testing. If we don't find major issues, you are right, folks can jump directly 
from one of 2.7, 2.8 or 2.9 to 3.0.



> Frankly speaking, working on some bridging release not targeting any feature 
> isn't so attractive to me as a contributor. Overall, the final minor release 
> off branch-2 is good, we should also give 3.x more time to evolve and mature, 
> therefore it looks to me we would have to work on two release lines meanwhile 
> for some time. I'd like option C), and suggest we focus on the recent 
> releases.



Answering this question is also one of the goals of my starting this thread. 
Collectively we need to conclude if we are okay or not okay with no longer 
putting any new feature work in general on the 2.x line after 2.9.0 release and 
move over our focus into 3.0.


Thanks
+Vinod

> -Original Message-
> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
> Sent: Tuesday, November 07, 2017 9:43 AM
> To: Andrew Wang <andrew.w...@cloudera.com>
> Cc: Arun Suresh <asur...@apache.org>; common-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; Hdfs-dev <hdfs-...@hadoop.apache.org>; 
> mapreduce-...@hadoop.apache.org
> Subject: Re: [DISCUSS] A final minor release off branch-2?
> 
> The main goal of the bridging release is to ease transition on stuff that is 
> guaranteed to be broken.
> 
> Of the top of my head, one of the biggest areas is application compatibility. 
> When folks move from 2.x to 3.x, are their apps binary compatible? Source 
> compatible? Or need changes?
> 
> In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
> source compatible. This means relooking at the API compatibility in 3.x and 
> their impact of migrating applications. We will have to revist and 
> un-deprecate old APIs, un-delete old APIs and write documentation on how apps 
> can be migrated.
> 
> Most of this work will be in 3.x line. The bridging release on the other hand 
> will have deprecation for APIs that cannot be undeleted. This may be already 
> have been done in many places. But we need to make sure and fill gaps if any.
> 
> Other areas that I can recall from the old days
> - Config migration: Many configs are deprecated or deleted. We need 
> documentation to help folks to move. We also need deprecations in the 
> bridging release for configs that cannot be undeleted.
> - You mentioned rolling-upgrades: It will be good to exactly outline the type 
> of testing. For e.g., the rolling-upgrades orchestration order has direct 
> implication on the testing done.
> - Story for downgrades?
> - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
> Is it broken anywhere that we cannot fix? Do we need bridging features for 
> this work?
> 
> +Vinod
> 
>> On Nov 6, 2017, at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
>> 
>> What are the known gaps that need bridging between 2.x and 3.x?
>> 
>> From an HDFS perspective,

Re: [DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

The main goal of the bridging release is to ease transition on stuff that is 
guaranteed to be broken.

Of the top of my head, one of the biggest areas is application compatibility. 
When folks move from 2.x to 3.x, are their apps binary compatible? Source 
compatible? Or need changes?

In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
source compatible. This means relooking at the API compatibility in 3.x and 
their impact of migrating applications. We will have to revist and un-deprecate 
old APIs, un-delete old APIs and write documentation on how apps can be 
migrated.

Most of this work will be in 3.x line. The bridging release on the other hand 
will have deprecation for APIs that cannot be undeleted. This may be already 
have been done in many places. But we need to make sure and fill gaps if any.

Other areas that I can recall from the old days
 - Config migration: Many configs are deprecated or deleted. We need 
documentation to help folks to move. We also need deprecations in the bridging 
release for configs that cannot be undeleted.
 - You mentioned rolling-upgrades: It will be good to exactly outline the type 
of testing. For e.g., the rolling-upgrades orchestration order has direct 
implication on the testing done.
 - Story for downgrades?
 - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
Is it broken anywhere that we cannot fix? Do we need bridging features for this 
work?

+Vinod

> On Nov 6, 2017, at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
> 
> What are the known gaps that need bridging between 2.x and 3.x?
> 
> From an HDFS perspective, we've tested wire compat, rolling upgrade, and
> rollback.
> 
> From a YARN perspective, we've tested wire compat and rolling upgrade. Arun
> just mentioned an NM rollback issue that I'm not familiar with.
> 
> Anything else? External to this discussion, these should be documented as
> known issues for 3.0.
> 
> Best.
> Andrew
> 
> On Sun, Nov 5, 2017 at 1:46 PM, Arun Suresh <asur...@apache.org> wrote:
> 
>> Thanks for starting this discussion VInod.
>> 
>> I agree (C) is a bad idea.
>> I would prefer (A) given that ATM, branch-2 is still very close to
>> branch-2.9 - and it is a good time to make a collective decision to lock
>> down commits to branch-2.
>> 
>> I think we should also clearly define what the 'bridging' release should
>> be.
>> I assume it means the following:
>> * Any 2.x user wanting to move to 3.x must first upgrade to the bridging
>> release first and then upgrade to the 3.x release.
>> * With regard to state store upgrades (at least NM state stores) the
>> bridging state stores should be aware of all new 3.x keys so the implicit
>> assumption would be that a user can only rollback from the 3.x release to
>> the bridging release and not to the old 2.x release.
>> * Use the opportunity to clean up deprecated API ?
>> * Do we even want to consider a separate bridging release for 2.7, 2.8 an
>> 2.9 lines ?
>> 
>> Cheers
>> -Arun
>> 
>> On Fri, Nov 3, 2017 at 5:07 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org>
>> wrote:
>> 
>>> Hi all,
>>> 
>>> With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out
>>> (tx Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we
>> have
>>> a discussion on how we manage our developmental bandwidth between 2.x
>> line
>>> and 3.x lines.
>>> 
>>> Once 3.0 GA goes out, we will have two parallel and major release lines.
>>> The last time we were in this situation was back when we did 1.x -> 2.x
>>> jump.
>>> 
>>> The parallel releases implies overhead of decisions, branch-merges and
>>> back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1,
>>> 3.0.1 and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of
>>> these lines - for e.g 2.8, 2.9 - are going to be used for a while at a
>>> bunch of large sites! At the same time, our users won't migrate to 3.0 GA
>>> overnight - so we do have to support two parallel lines.
>>> 
>>> I propose we start thinking of the fate of branch-2. The idea is to have
>>> one final release that helps our users migrate from 2.x to 3.x. This
>>> includes any changes on the older line to bridge compatibility issues,
>>> upgrade issues, layout changes, tooling etc.
>>> 
>>> We have a few options I think
>>> (A)
>>>-- Make 2.9.x the last minor release off branch-2
>>>-- Have a maintenance release that bridges 2.9 to 3.x
>>>-- Continue to make more ma

Re: [VOTE] Merge yarn-native-services branch into trunk

[Vinod Kumar Vavilapalli]

Congratulations to all the contributors involved, this is a great step forward!

+Vinod

> On Nov 6, 2017, at 2:40 PM, Jian He <j...@hortonworks.com> wrote:
> 
> Okay, I just merged the branch to trunk (108 commits in total !)
> Again, thanks for all who contributed to this feature!
> 
> Jian
> 
> On Nov 6, 2017, at 1:26 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> 
> Here’s +1 from myself.
> The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.
> 
> Thanks for all who voted. I’ll merge to trunk by the end of today.
> 
> Jian
> 
> On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
> <billie.rina...@gmail.com<mailto:billie.rina...@gmail.com>> wrote:
> 
> +1 (binding)
> 
> On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi All,
> 
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation, 
> DNS, CLI modifications etc. We believe now the feature is in a much better 
> shape.
> 
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to 
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS service 
> to enable users to discover services deployed on YARN via standard DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the existing 
> system, and have no impact on existing system if disabled.
> 
> Special thanks to a team of folks who worked hard towards this: Billie 
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K 
> S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without 
> their ideas and hard work.
> Also thanks Allen for some review and verifications.
> 
> Thanks,
> Jian
> 
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
> 
> 
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Hi all,

With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out (tx 
Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we have a 
discussion on how we manage our developmental bandwidth between 2.x line and 
3.x lines.

Once 3.0 GA goes out, we will have two parallel and major release lines. The 
last time we were in this situation was back when we did 1.x -> 2.x jump.

The parallel releases implies overhead of decisions, branch-merges and 
back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1, 3.0.1 
and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of these lines 
- for e.g 2.8, 2.9 - are going to be used for a while at a bunch of large 
sites! At the same time, our users won't migrate to 3.0 GA overnight - so we do 
have to support two parallel lines.

I propose we start thinking of the fate of branch-2. The idea is to have one 
final release that helps our users migrate from 2.x to 3.x. This includes any 
changes on the older line to bridge compatibility issues, upgrade issues, 
layout changes, tooling etc.

We have a few options I think
 (A)
-- Make 2.9.x the last minor release off branch-2
-- Have a maintenance release that bridges 2.9 to 3.x
-- Continue to make more maintenance releases on 2.8 and 2.9 as necessary
-- All new features obviously only go into the 3.x line as no features can 
go into the maint line.

 (B)
-- Create a new 2.10 release which doesn't have any new features, but as a 
bridging release
-- Continue to make more maintenance releases on 2.8, 2.9 and 2.10 as 
necessary
-- All new features, other than the bridging changes, go into the 3.x line

 (C)
-- Continue making branch-2 releases and postpone this discussion for later

I'm leaning towards (A) or to a lesser extent (B). Willing to hear otherwise.

Now, this obviously doesn't mean blocking of any more minor releases on 
branch-2. Obviously, any interested committer / PMC can roll up his/her 
sleeves, create a release plan and release, but we all need to acknowledge that 
versions are not cheap and figure out how the community bandwidth is split 
overall.

Thanks
+Vinod
PS: The proposal is obviously not to force everyone to go in one direction but 
more of a nudging the community to figure out if we can focus a major part of 
of our bandwidth on one line. I had a similar concern when we were doing 2.8 
and 3.0 in parallel, but the impending possibility of spreading too thin is 
much worse IMO.
PPS: (C) is a bad choice. With 2.8 and 2.9 we are already seeing user adoption 
splintering between two lines. With 2.10, 2.11 etc coexisting with 3.0, 3.1 
etc, we will revisit the mad phase years ago when we had 0.20.x, 0.20-security 
coexisting with 0.21, 0.22 etc.

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Arun / Subru,

Thanks for the great work!

Few quick comments
 - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and their 
signatures and delete everything else? So that it's easy to pick up the 
important bits for the voters. For e.g, like this 
http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/ 

 - Can you put the generated CHANGES.html and releasenotes.html instead of the 
md files, for quicker perusal?

Thanks
+Vinod

> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
> 
> Hi folks,
> 
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
> 
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
> 
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
> 
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
> *
> 
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
> 
> Thanks,
> 
> Arun/Subru

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Vinod Kumar Vavilapalli]

>   At a minimum, it should at least be using it’s own maven module for a 
> lot of the bits that generates it’s own maven jars so that we can split this 
> functionality up at build/test time.


I expected this to be the case, but looks like it isn't.

There's lot of value in splitting the HDFS code into smaller modules. 
Definitely newer code like Ozone.

When we did this for YARN, initially there were concerns about module 
proliferation, but looking back, my observations have been that it has done us 
far more good than expected. Starting with the fact that we had clients and 
servers modularized independently, as well as servers from other servers, with 
far cleaner contracts than what we had in Hadoop 1 world.

Thanks
+Vinod

Re: [DISCUSS] Looking to Apache Hadoop 3.1 release

[Vinod Kumar Vavilapalli]

Thanks for starting this thread, Wangda!

+1 for establishing a faster cadence now itself.

One word of caution though. The same I expressed while we were trying to do 
both 2.8 and 3.0 releases at the same time. Please try avoiding concurrent 
releases and splitting community bandwidth - it's not that we cannot do 
multiple releases in parallel, it's mainly that we will not be able to give our 
best on both.

Thanks
+Vinod

> On Sep 6, 2017, at 11:13 AM, Wangda Tan  wrote:
> 
> Hi all,
> 
> As we discussed on [1], there were proposals from Steve / Vinod etc to have
> a faster cadence of releases and to start thinking of a Hadoop 3.1 release
> earlier than March 2018 as is currently proposed.
> 
> I think this is a good idea. I'd like to start the process sooner, and
> establish timeline etc so that we can be ready when 3.0.0 GA is out. With
> this we can also establish faster cadence for future Hadoop 3.x releases.
> 
> To this end, I propose to target Hadoop 3.1.0 for a release by mid Jan
> 2018. (About 4.5 months from now and 2.5 months after 3.0-GA, instead of
> 6.5 months from now).
> 
> I'd also want to take this opportunity to come up with a more elaborate
> release plan to avoid some of the confusion we had with 3.0 beta. General
> proposal for the timeline (per this other proposal [2])
> - Feature freeze date - all features should be merged by Dec 15, 2017.
> - Code freeze date - blockers/critical only, no more improvements and non
> blocker/critical bug-fixes: Jan 1, 2018.
> - Release date: Jan 15, 2018
> 
> Following is a list of features on my radar which could be candidates for a
> 3.1 release:
> - YARN-5734, Dynamic scheduler queue configuration. (Owner: Jonathan Hung)
> - YARN-5881, Add absolute resource configuration to CapacityScheduler.
> (Owner: Sunil)
> - YARN-5673, Container-executor rewrite for better security, extensibility
> and portability. (Owner Varun Vasudev)
> - YARN-6223, GPU isolation. (Owner: Wangda)
> 
> And from email [3] mentioned by Andrew, there’re several other HDFS
> features want to be released with 3.1 as well, assuming they fit the
> timelines:
> - Storage Policy Satisfier
> - HDFS tiered storage
> 
> Please let me know if I missed any features targeted to 3.1 per this
> timeline.
> 
> And I want to volunteer myself as release manager of 3.1.0 release. Please
> let me know if you have any suggestions/concerns.
> 
> Thanks,
> Wangda Tan
> 
> [1] http://markmail.org/message/hwar5f5ap654ck5o?q=
> Branch+merges+and+3%2E0%2E0-beta1+scope
> [2] http://markmail.org/message/hwar5f5ap654ck5o?q=Branch+
> merges+and+3%2E0%2E0-beta1+scope#query:Branch%20merges%
> 20and%203.0.0-beta1%20scope+page:1+mid:2hqqkhl2dymcikf5+state:results
> [3] http://markmail.org/message/h35obzqrh3ag6dgn?q=Branch+merge
> s+and+3%2E0%2E0-beta1+scope


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Branches and versions for Hadoop 3

[Vinod Kumar Vavilapalli]

+1 to Andrew’s proposal for 3.x releases.

We had fairly elaborate threads on this branching & compatibility topic before. 
One of them’s here: [1]

+1 to what Jason said.
 (a) Incompatible changes are not to be treated lightly.  We need to stop 
breaking stuff and ‘just dump it on trunk'.
 (b) Major versions are expensive. We should hesitate before asking our users 
to move from 2.0 to 3.0 or 3.0 to 4.0 (with incompatible changes) *without* any 
other major value proposition.

Some of the incompatible changes can clear wait while others cannot and so may 
mandate a major release. What are the some of the common types of incompatible 
changes?
 - Renaming APIs, removing deprecated APIs, renaming configuration properties, 
changing the default value of a configuration, changing shell output / logging 
etc:
— Today, we do this on trunk even though the actual effort involved is very 
minimal compared to the overhead it forces in maintaining incompatible trunk.
 - Dependency library updates - updating guava, protobuf etc in Hadoop breaks 
upstreaming applications. I am assuming Classpath Isolation [2] is still a 
blocker for 3.0 GA.
 - JDK upgrades: We tried two different ways with JDK 7 and JDK 8, we need a 
formal policy on this.

If we can managing the above common breaking changes, we can cause less pain to 
our end users.

Here’s what we can do for 3.x / 4.x specifically.
 - Stay on trunk based 3.x releases
 - Avoid all incompatible changes as much as possible
 - If we run into a bunch of minor incompatible changes that have be done, we 
either (a) make the incompatible behavior optional or (b) just park them say 
with an parked-incompatible-change label if making it optional is not possible
 - We create a 4.0 only when (a) we hit the first major incompatible change 
because a major next-step for Hadoop needs it (for e.g. Erasure Coding), and/or 
(b) the number of parked incompatible changes passes a certain threshold. 
Unlike Jason, I don’t see the threshold to be 1 for cases that don’t fit (1).

References
 [1] Looking to a Hadoop 3 release: 
http://markmail.org/thread/2daldggjaeewdmdf#query:+page:1+mid:m6x73t6srlchywsn+state:results
 

 [2] Classpath isolation for downstream client: 
https://issues.apache.org/jira/browse/HADOOP-11656 


Thanks
+Vinod

> On Aug 25, 2017, at 1:23 PM, Jason Lowe  wrote:
> 
> Allen Wittenauer wrote:
> 
> 
>> Doesn't this place an undue burden on the contributor with the first
>> incompatible patch to prove worthiness?  What happens if it is decided that
>> it's not good enough?
> 
> 
> It is a burden for that first, "this can't go anywhere else but 4.x"
> change, but arguably that should not be a change done lightly anyway.  (Or
> any other backwards-incompatible change for that matter.)  If it's worth
> committing then I think it's perfectly reasonable to send out the dev
> announce that there's reason for trunk to diverge from 3.x, cut branch-3,
> and move on.  This is no different than Andrew's recent announcement that
> there's now a need for separating trunk and the 3.0 line based on what's
> about to go in.
> 
> I do not think it makes sense to pay for the maintenance overhead of two
> nearly-identical lines with no backwards-incompatible changes between them
> until we have the need.  Otherwise if past trunk behavior is any
> indication, it ends up mostly enabling people to commit to just trunk,
> forgetting that the thing they are committing is perfectly valid for
> branch-3.  If we can agree that trunk and branch-3 should be equivalent
> until an incompatible change goes into trunk, why pay for the commit
> overhead and potential for accidentally missed commits until it is really
> necessary?
> 
> How many will it take before the dam will break?  Or is there a timeline
>> going to be given before trunk gets set to 4.x?
> 
> 
> I think the threshold count for the dam should be 1.  As soon as we have a
> JIRA that needs to be committed to move the project forward and we cannot
> ship it in a 3.x release then we create branch-3 and move trunk to 4.x.
> As for a timeline going to 4.x, again I don't see it so much as a "baking
> period" as a "when we need it" criteria.  If we need it in a week then we
> should cut it in a week.  Or a year then a year.  It all depends upon when
> that 4.x-only change is ready to go in.
> 
> Given the number of committers that openly ignore discussions like this,
>> who is going to verify that incompatible changes don't get in?
>> 
> 
> The same entities who are verifying other bugs don't get in, i.e.: the
> committers and the Hadoop QA bot running the tests.  Yes, I know that means
> it's inevitable that compatibility breakages will happen, and we can and
> should improve the automation around compatibility testing when possible.
> But I don't think there's a magic 

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

> From a release management perspective, it's *extremely* reasonable to block 
> the inclusion of new features a month from the planned release date. A 
> typical software development lifecycle includes weeks of feature freeze and 
> weeks of code freeze. It is no knock on any developer or any feature to say 
> that we should not include something in 3.0.0.


We have never followed the ‘typical' lifecycle that I am guessing you are 
referring to. If we are, you'll need to publish some of the following: a 
feature freeze date, blockers-criticals-only-from-now date, testing-finish 
date, documentation-finish date, final release date and so on.

What we do with Apache releases typically is instead we say ‘this' is roughly 
when we want to release, and roughly what features must land and let the rest 
figure out itself.

Neither is right or wrong. If we want to change the process, we should 
communicate as such.

Proposing a feature freeze date on the fly is only going to confuse people.


> I've been very open and clear about the goals, schedule, and scope of 3.0.0 
> over the last year plus. The point of the extended alpha process was to get 
> all our features in during alpha, and the alpha merge window has been open 
> for a year. I'm unmoved by arguments about how long a feature has been worked 
> on. None of these were not part of the original 3.0.0 scope, and our users 
> have been waiting even longer for big-ticket 3.0 items like JDK8 and HDFS EC 
> that were part of the discussed scope.


Except our schedule is so fluid (not due to the release management process to 
be fair) that it is hard for folks to plan their features. IIRC, our schedule 
was a GA release beginning of this year. Again, this is not a critique of 3.0 
release process - I have myself done enough releases to know that sticking to a 
date and herding the crowd has been an extremely hard job.

The only way out of this is get 3.0 out and stick to 1-2 month minor releases. 
May be we start that by planning a 3.1 right now and push one in January 
assuming 3.0 GA happens in November. Cadence is a habit.


> I see that two VOTEs have gone out since I was out. I still plan to follow 
> the proposal in my original email. This means I'll cut branch-3 and 
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open up 
> development for Hadoop 3.1.0 and 4.0.0.
> 
> I'm reaching out to the lead contributor of each of these features 
> individually to discuss. We need to close on this quickly, and email is too 
> low bandwidth at this stage.


My only concern in all of this is if some of these branch contributors decide 
that they don’t want and then proceed to having a parallel release and cause 
pains for everyone. This is what I tried avoiding parallel 2.9 and 2.10 offline 
and that’s what I am trying to do here. For now, I don’t have a horse in this 
race - that’s between you and the branch-contributors in question for now. If 
you can reach an agreement, we are all good for it.

+Vinod



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Agreed. I was very clearly not advocating for rushing in features. If you have 
followed my past emails, I have only strongly advocated features be worked in 
branches and get merged when they are in a reasonable state.

Each branch contributor group should look at their readiness and merge stuff in 
assuming that the branch reached a satisfactory state. That’s it.

From release management perspective, blocking features just because we are a 
month close to the deadline is not reasonable. Let the branch contributors 
rationalize, make this decision and the rest of us can support them in making 
the decision.

+Vinod

> At this point, there have been three planned alphas from September 2016 until 
> July 2017 to "get in features".  While a couple of upcoming features are "a 
> few weeks" away, I think all of us are aware how predictable software 
> development schedules can be.  I think we can also all agree that rushing 
> just to meet a release deadline isn't the best practice when it comes to 
> software development either.
> 
> Andrew has been very clear about his goals at each step and I think Wangda's 
> willingness to not rush in resource types was an appropriate response.  I'm 
> sympathetic to the goals of getting in a feature for 3.0, but it might be a 
> good idea for each project that is a "few weeks away" to seriously look at 
> the readiness compared to the features which have been testing for 6+ months 
> already.
> 
> -Ray

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Vinod Kumar Vavilapalli]

Such a great community effort - hats off, team!

Thanks
+Vinod

> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vrushalic2...@gmail.com> 
> wrote:
> 
> Hi folks,
> 
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
> 
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
> 
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
> 
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
> 
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
> 
> There are a total of 82 subtasks that were completed as part of this effort.
> 
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
> 
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> 
> Regards,
> Vrushali
> 
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Steve,

You can be strict & ruthless about the timelines. Anything that doesn’t get in 
by mid-September, as was originally planned, can move to the next release - 
whether it is feature work on branches or feature work on trunk.

The problem I see here is that code & branches being worked on for a year are 
now (apparently) close to being done and we are telling them to hold for 7 more 
months - this is not a reasonable ask..

If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ can 
volunteer. But this is how you get competing releases and split bandwidth.

As for compatibility / testing etc, it seems like there is a belief that the 
current ‘scoped’ features are all tested well in these areas and so adding more 
is going to hurt the release. There is no way this is the reality, trunk has so 
many features that have been landing for years, the only way we can 
collectively attempt towards making this stable is by getting as many parties 
together as possible, each verifying stuff that they need. Not by excluding 
specific features.

+Vinod

> This is one of those curse-of-cadence things: The higher your release 
> cadence, the less pressure to get "everything in". With a slower cadence, 
> more pressure to get stuff in, more pressure to hold up the release, slows 
> the cadence, gets even more stuff in, etc. etc.
> 
> - Andrew has been working on the release for months, we all need to 
> appreciate how much hard work that is and has been, especially for what is 
> going to be a major release.
> 
> - We know that things will be unstable in 3.0; Andrew's concern is about 
> making sure that the newest, unstablest (?) features can at least be bypassed 
> if there are problems. I we should also call out in the release notes what we 
> think are the unstable bits where people need to use caution (example: 
> S3Guard in "authoritative" mode)
> 
> - Anything related to wire compatibility has been problematic in the past; I 
> think it's essential that whatever packets get sent around are going to be 
> stable, so changes there need to be in, or at least the payloads set up ready 
> for the features. Same for new public APIs.
> 
> - As fpr the rest, I don't know. I think being strict about it and ruthless 
> in assessing the feature's stability & consequences of postponing the feature 
> until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
> with a 3.2 in the summer.
> 
> Then: start planning that 3.1 release. Maybe I should put my hand up as 
> release manager for that one. Then everyone would realise how amenable Andrew 
> is being today.
> 
> 
> One other thing: alongside the big branches, there's the eternal backlog of 
> small patches. We should organise spending a few days updating, reviewing & 
> merging them in
> 
> -Steve
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
> 

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Andrew,

Each of the branches below have been created more than a year ago (!) and have 
been consistently worked upon and are now finally seeing the light of the day. 
When they are "few weeks” away, pushing them out by 7 *more* months just 
doesn’t make sense.

While I deeply appreciate the push for the dates, we shouldn’t be putting 
moratorium on features like this till the proposed date is due. As a release 
manager, it’s good to say that we will release a specific version as soon as 
so-and-so features are in, but let’s not put exclusions like this.

I propose that, as we have done with every release so far, (and more 
specifically the way we did 2.x alphas, betas, GA back in the day), we float 
the date, let the individual branch contributors decide their fate. As long as 
of course, they meet the timelines and the usual branch merge bar, testing / 
compatibility / impact on rest of the code-base etc.

Anything short of that, we will just be incentivizing contributors away from 
doing work in branches and towards putting stuff directly into trunk.

+Vinod

> On Aug 18, 2017, at 6:22 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> As you might have seen, we've had a number of branch merges floated this
> past week targeted for 3.0.0-beta1, which is planned for about a month from
> now.
> 
> In total, I'm currently tracking these branches:
> 
> YARN-2915: YARN federation (recently merged)
> HADOOP-13345: S3Guard (currently being voted on)
> YARN-5355: TSv2 alpha2 ("few weeks")
> YARN-5079: Native services ("few weeks")
> YARN-3926: Resource profiles ("few weeks")
> 
> We should effectively be in code freeze (only blockers/criticals), so the
> volume of merge proposals at this point came as a surprise. Despite our
> best efforts as software engineers, big code movement always comes with
> risk.
> 
> Since we started the 3.0 release series close to a year ago, I'm also loath
> to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
> HDFS EC, and our users deserve a production-quality release with these
> features. We've also been good about the release cadence thus far in 3.x,
> so a 3.1 isn't that far out.
> 
> Here's my proposal:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 
> My rationale for inclusion and exclusion is as follows:
> 
> Inclusion:
> 
> * YARN federation has been run in production, does not touch existing code,
> adds no new APIs, and is off by default.
> * S3Guard has been run in production and is off by default.
> * The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
> committed to this for 3.0.0 GA. It's off by default and adds no impact.
> 
> Exclusion:
> 
> * The primary reason for exclusion is to maintain the planned release
> schedule. Native services and resource profiles are still a few weeks from
> being ready for merge.
> * A reminder that 3.1 is only another 3 months after 3.0 given our release
> cadence thus far. If there's demand, we could even do a 3.1 immediately
> following 3.0.
> 
> I'm happy to talk with the contributors of each of these features to
> understand their timelines and requirements, with the caveat that I'll be
> out through Wednesday next week. Please reach out.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Vinod Kumar Vavilapalli]

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
> 
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
> 
> Thanks,
> 
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-dev@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
> 
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
> 
> Jason
> 
> 
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
> 
> 
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
> 
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
> 
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
> 
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
> 
>  Please share thoughts if you have any. Thanks!
> 
> Thanks,
> 
> Junping
> 
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
> 
> Hi all,
> 
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
> 
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
> 
> Thanks
> +Vinod
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

Thanks for taking 2.7.4 over Konstantin!

Regarding rolling RC next week, I still see that there are 4 blocker / critical 
tickets targeted for 2.7.4: 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4
 
.

We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 
 definitely was something 
that was deemed a blocker for 2.8.2, not sure about 2.7.4.

I’m ‘back’ - let me know if you need any help.

Thanks
+Vinod

> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko  wrote:
> 
> Hi everybody.
> 
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
> 
> I plan to build an RC next week. If there are no objection.
> 
> Thanks,
> --Konst
> 
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko 
> wrote:
> 
>> Hey guys.
>> 
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>> 
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>> 
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>> 
>> There is a plan to enable BigTop for 2.7.4 testing.
>> 
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>> 
>> Regards,
>> --Konstantin
>> 
>> 
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka 
>> wrote:
>> 
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>> 
>>> -Akira
>>> 
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>> 
 Couple of more JIRAs need to be back ported for 2.7.4 release. These will
 solve RM HA unstability issues.
 https://issues.apache.org/jira/browse/YARN-5333
 https://issues.apache.org/jira/browse/YARN-5988
 https://issues.apache.org/jira/browse/YARN-6304
 
 I will raise a JIRAs to back port it.
 
 @Akira , could  you help to add these JIRAs into wiki?
 
 Thanks & Regards
 Rohith Sharma K S
 
 On 29 May 2017 at 12:19, Akira Ajisaka  wrote:
 
 Created a page for 2.7.4 release.
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
> 
> If you want to edit this wiki, please ping me.
> 
> Regards,
> Akira
> 
> 
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
> 
> Hi Konstantin Shvachko
>> 
>> 
>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>> trunk in following link.??
>> 
>> 
>> https://cwiki.apache.org/confluence/display/HADOOP
>> 
>> 
>> 
>> From: Konstantin Shvachko 
>> Sent: Saturday, May 13, 2017 3:58 AM
>> To: Akira Ajisaka
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>> yarn-...@hadoop.apache.org
>> Subject: Re: About 2.7.4 Release
>> 
>> Latest update on the links and filters. Here is the correct link for
>> the
>> filter:
>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>> requestId=12340814
>> 
>> Also updated: https://s.apache.org/Dzg4
>> 
>> Had to do some Jira debugging. Sorry for confusion.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>> shv.had...@gmail.com>
>> wrote:
>> 
>> Hey Akira,
>> 
>>> 
>>> I didn't have private filters. Most probably Jira caches something.
>>> Your filter is in the right direction, but for some reason it lists
>>> only
>>> 22 issues, while mine has 29.
>>> It misses e.g. YARN-5543 >> a/browse/YARN-5543>
>>> .
>>> 
>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release 

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Vinod Kumar Vavilapalli]

Thanks for taking the mantle from me on 2.8.0 and some persistent work getting 
2.8.0 out the door, Junping!

Apologies for bringing this up late, but I’d like to add one comment.

We should repeat what we did for 2.7.0 and In line with our experience there, 
we should annotate this release as not ready for production use. See the 
releases page - 
http://hadoop.apache.org/releases.html#25+August%2C+2016%3A+Release+2.7.3+available
 for our messaging on 2.7.0.

The expectation is that more downstream projects pick up the bits, iron out any 
incompatibilities we might have missed, and production users then pick up a 
solid 2.8.1.

Thanks
+Vinod

> On Mar 14, 2017, at 1:41 AM, Junping Du  wrote:
> 
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
> 
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
> 
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
> 
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
> 
>  The RC tag in git is: release-2.8.0-RC2
> 
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
> 
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
> 
> Thanks,
> 
> Junping


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Here’s my +1 to conclude the vote.

With 8 binding +1s, 9 non-binding +1s and no -1s, this vote passes.

I’ll push the RC forward to an official release.

Thanks to everyone who voted!

+Vinod

> On Aug 24, 2016, at 10:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> +1 Binding.
> 
> - Built and deploy a single node cluster from source code.
> - Ran some example jobs.
> 
> Thanks,
> Wangda
> 
> On Tue, Aug 23, 2016 at 12:35 PM, Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
> 
>> +1 (non-binding)
>> 
>> - Downloaded source and tar verified the checksum
>> - Installed pseudo cluster with Node Labels and ATSV1 enabled.
>> - Verified with running MR apps with and without labels.
>> - Verified ATSV1 UI.
>> 
>> Regards,
>> + Naga
>> 
>> On Mon, Aug 22, 2016 at 11:55 PM, Jason Lowe <jl...@yahoo-inc.com.invalid>
>> wrote:
>> 
>>> +1 (binding)
>>> - Verified signatures and digests- Successfully built from source with
>>> native support- Deployed a single-node cluster- Ran some sample jobs
>>> successfully
>>> 
>>> Jason
>>> 
>>>  From: Vinod Kumar Vavilapalli <vino...@apache.org>
>>> To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>;
>>> hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; "
>>> mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>
>>> Cc: Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Sent: Wednesday, August 17, 2016 9:05 PM
>>> Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2
>>> 
>>> Hi all,
>>> 
>>> I've created a new release candidate RC2 for Apache Hadoop 2.7.3.
>>> 
>>> As discussed before, this is the next maintenance release to follow up
>>> 2.7.2.
>>> 
>>> The RC is available for validation at: http://home.apache.org/~
>>> vinodkv/hadoop-2.7.3-RC2/ <http://home.apache.org/~
>>> vinodkv/hadoop-2.7.3-RC2/>
>>> 
>>> The RC tag in git is: release-2.7.3-RC2
>>> 
>>> The maven artifacts are available via repository.apache.org <
>>> http://repository.apache.org/> at https://repository.apache.org/
>>> content/repositories/orgapachehadoop-1046 <https://repository.apache.
>>> org/content/repositories/orgapachehadoop-1046>
>>> 
>>> The release-notes are inside the tar-balls at location
>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>>> hosted this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
>>> releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
>>> releasenotes.html> for your quick perusal.
>>> 
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - few more issues with RC1 forced a RC2 [2]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
>>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
>>> bit. This release's related discussion thread is linked below: [3].
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> Thanks,
>>> Vinod
>>> 
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
>> https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/index.html#26106 <
>>> https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/index.html#26106
>>> 
>>> [2] [VOTE] Release Apache Hadoop 2.7.3 RC1:
>> https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/msg26336.html <
>> https://www.mail-archive.com/
>>> hdfs-...@hadoop.apache.org/msg26336.html>
>>> [3] 2.7.3 release plan: https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
>>> 6yv2fyrs4jlepmmr>
>>> 
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13544) JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created HADOOP-13544:


 Summary: JDiff reports unncessarily show unannotated APIs and 
cause confusion while our javadocs only show annotated and public APIs
 Key: HADOOP-13544
 URL: https://issues.apache.org/jira/browse/HADOOP-13544
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker


Our javadocs only show annotated and @Public APIs (original JIRAs HADOOP-7782, 
HADOOP-6658).

But the jdiff shows all APIs that are not annotated @Private. This causes 
confusion on how we read the reports and what APIs we really broke.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13543) [Umbrella] Analyse 2.8.0 and 3.0.0-alpha1 jdiff reports and fix any issues

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created HADOOP-13543:


 Summary: [Umbrella] Analyse 2.8.0 and 3.0.0-alpha1 jdiff reports 
and fix any issues
 Key: HADOOP-13543
 URL: https://issues.apache.org/jira/browse/HADOOP-13543
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker


Now that we have fixed JDiff report generation for 2.8.0 and above, we should 
analyse them.

For the previous releases, I was applying the jdiff patches myself, and 
analysed them offline. It's better to track them here now that the reports are 
automatically getting generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Too late for 2.7.3 - I want us to quickly get back to a regular cadence of 
releases.

Doesn’t look like a regression IAC. Let’s put it in the next release if needed.

Thanks
+Vinod

> On Aug 22, 2016, at 6:19 AM, Brahma Reddy Battula 
>  wrote:
> 
> Felt like HDFS-8388, should be in for 2.7.3

Re: Updated 2.8.0-SNAPSHOT artifact

[Vinod Kumar Vavilapalli]

Jon,

That is around the time when I branched 2.8, so I guess you were getting 
SNAPSHOT artifacts till then from the branch-2 nightly builds.

If you need it, we can set up SNAPSHOT builds. Or just wait for the first RC, 
which is around the corner.

+Vinod

> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles  wrote:
> 
> Latest snapshot is uploaded in Nov 2015, but checkins are still coming in
> quite frequently.
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-yarn-api/
> 
> Are there any plans to start producing updated SNAPSHOT artifacts for
> current hadoop development lines?


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: adding contributor roles timing out again

[Vinod Kumar Vavilapalli]

It happens to me too on both Firefox / Chrome.

+Vinod

> On Aug 18, 2016, at 8:39 AM, Chris Nauroth  wrote:
> 
> It’s odd that Firefox didn’t work for you.  My standard workaround is to use 
> Firefox, and that’s what I just did successfully for shenyinjie.
> 
> It’s quite mysterious to me that this problem would be browser-specific at 
> all though.
> 
> --Chris Nauroth
> 
> On 8/18/16, 6:53 AM, "Steve Loughran"  wrote:
> 
> 
>I'm trying to add a new contributor, "shenyinjie", but I'm getting the 
> couldn't connect to server message; tried on chrome and firefox, and tried to 
> paste the username in rather than rely on any popup completion.
> 
>no joy: has anyone else succeeded at this recently? 
> 
> 
> 
> 
>-
>To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
> 
> 
> 
> B�CB��[��X��ܚX�KK[XZ[���[[ۋY]�][��X��ܚX�PY���\X�K�ܙ�B��܈Y][ۘ[��[X[��K[XZ[���[[ۋY]�Z[Y���\X�K�ܙ�B


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Canceling the release vote for this and other issues reported.

+Vinod

> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <ajisa...@oss.nttdata.co.jp> 
> wrote:
> 
> -1 (binding)
> 
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not 
> reflected in the release note.
> 
> -Akira
> 
> On 8/17/16 13:29, Allen Wittenauer wrote:
>> 
>> 
>> -1
>> 
>> HDFS-9395 is an incompatible change:
>> 
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>> 
>> 
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>>> wrote:
>>> 
>>> Hi all,
>>> 
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>> 
>>> As discussed before, this is the next maintenance release to follow up 
>>> 2.7.2.
>>> 
>>> The RC is available for validation at: 
>>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>> 
>>> The RC tag in git is: release-2.7.3-RC1
>>> 
>>> The maven artifacts are available via repository.apache.org 
>>> <http://repository.apache.org/> at 
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>>> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>> 
>>> The release-notes are inside the tar-balls at location 
>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>>> your quick perusal.
>>> 
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a 
>>> bit. This release's related discussion thread is linked below: [2].
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> Thanks,
>>> Vinod
>>> 
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: 
>>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>> 
>> 
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

I always look at CHANGES.txt entries for incompatible-changes and this JIRA 
obviously wasn’t there.

Anyways, this shouldn’t be in any of branch-2.* as committers there clearly 
mentioned that this is an incompatible change.

I am reverting the patch from branch-2* .

Thanks
+Vinod

> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <a...@effectivemachines.com> 
> wrote:
> 
> 
> 
> -1
> 
> HDFS-9395 is an incompatible change:
> 
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
> 
> 
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>> wrote:
>> 
>> Hi all,
>> 
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>> 
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>> 
>> The RC tag in git is: release-2.7.3-RC1
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>> your quick perusal.
>> 
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
>> This release's related discussion thread is linked below: [2].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Thanks Steve, this is one area that isn’t very well release-tested usually!

+Vinod

> On Aug 16, 2016, at 2:25 AM, Steve Loughran  wrote:
> 
> I've just looked at the staged JARs and how they worked with downstream apps 
> —that being a key way that Hadoop artifacts are adopted.

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Thanks Marco. It was a Thursday late-night slip-up.

Fixed the dates and replaced the bits, so the voting can continue.

FYI, they aren’t binding though - as it all depends on how the release voting 
goes. One should usually only trust the release-date published on the website.

Thanks
+Vinod

> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzueh...@gmail.com 
> <mailto:mzueh...@gmail.com>> wrote:
> 
> Hi Vinod,
> 
> I'm not sure if this is relevant, but you changed the release date in the 
> CHANGES.txt 
> <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
>  files to 2016-09-19.
> I guess you have meant 2016-08-19.
> 
> See: 
> https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
>  
> <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
> 
> 
> Thanks,
> Marco
> 
> 
> 
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vino...@apache.org 
> <mailto:vino...@apache.org>>:
> Hi all,
> 
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
> 
> The RC tag in git is: release-2.7.3-RC1
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> <http://repository.apache.org/ 
> <http://repository.apache.org/>> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
> your quick perusal.
> 
> As you may have noted,
>  - few issues with RC0 forced a RC1 [1]
>  - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
> This release's related discussion thread is linked below: [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
> 

[VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 


The RC tag in git is: release-2.7.3-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

But, everyone please do continue your sanity checking on RC0 in case there are 
more issues to be fixed.

Thanks
+Vinod

> On Jul 26, 2016, at 12:11 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Thanks Daniel and Wei.
> 
> I think these are worth fixing, I’m withdrawing this RC. Will look at fixing 
> these issues and roll a new candidate with the fixes as soon as possible.
> 
> Thanks
> +Vinod
> 
>> On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang <weic...@apache.org 
>> <mailto:weic...@apache.org>> wrote:
>> 
>> I noticed two issues:
>> 
>> (1) I ran hadoop checknative, but it seems the binary tarball was not 
>> compiled with native library for Linux. On the contrary, the Hadoop built 
>> from source tarball with maven -Pnative can find the native libraries on the 
>> same host.
>> 
>> (2) I noticed that the release dates in CHANGES.txt in tag release-2.7.3-RC0 
>> are set to Release 2.7.3 - 2016-07-27.
>> However, the release dates in CHANGES.txt in the source and binary tar balls 
>> are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue though.
>> 
>> * Downloaded source and binary.
>> * Verified signature.
>> * Verified checksum.
>> * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
>> went fine.
>> * Ran hadoop checknative
>> 
>> On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah 
>> <rusha...@yahoo-inc.com.invalid <mailto:rusha...@yahoo-inc.com.invalid>> 
>> wrote:
>> Thanks Vinod for all the release work !
>> +1 (non-binding).
>> * Downloaded from source and built it.* Deployed a pseudo distributed 
>> cluster.
>> * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything works 
>> fine.
>> 
>> 
>> On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
>> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
>> 
>> 
>>  Hi all,
>> 
>> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>> 
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>> 
>> The RC tag in git is: release-2.7.3-RC0
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> <http://repository.apache.org/ 
>> <http://repository.apache.org/>> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/> 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/>>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html> 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
>> your quick perusal.
>> 
>> As you may have noted, a very long fix-cycle for the License & Notice issues 
>> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
>> by quite a bit. This release's related discussion thread is linked below: 
>> [1].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>> 
>>
>> 
> 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

Thanks Daniel and Wei.

I think these are worth fixing, I’m withdrawing this RC. Will look at fixing 
these issues and roll a new candidate with the fixes as soon as possible.

Thanks
+Vinod

> On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang <weic...@apache.org> wrote:
> 
> I noticed two issues:
> 
> (1) I ran hadoop checknative, but it seems the binary tarball was not 
> compiled with native library for Linux. On the contrary, the Hadoop built 
> from source tarball with maven -Pnative can find the native libraries on the 
> same host.
> 
> (2) I noticed that the release dates in CHANGES.txt in tag release-2.7.3-RC0 
> are set to Release 2.7.3 - 2016-07-27.
> However, the release dates in CHANGES.txt in the source and binary tar balls 
> are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue though.
> 
> * Downloaded source and binary.
> * Verified signature.
> * Verified checksum.
> * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
> went fine.
> * Ran hadoop checknative
> 
> On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah <rusha...@yahoo-inc.com.invalid 
> <mailto:rusha...@yahoo-inc.com.invalid>> wrote:
> Thanks Vinod for all the release work !
> +1 (non-binding).
> * Downloaded from source and built it.* Deployed a pseudo distributed cluster.
> * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything works 
> fine.
> 
> 
> On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
> 
> 
>  Hi all,
> 
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
> 
> The RC tag in git is: release-2.7.3-RC0
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> <http://repository.apache.org/ 
> <http://repository.apache.org/>> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/> 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/>>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html> 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
> your quick perusal.
> 
> As you may have noted, a very long fix-cycle for the License & Notice issues 
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
> by quite a bit. This release's related discussion thread is linked below: [1].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
> 
>
> 

[DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Vinod Kumar Vavilapalli]

Forking the thread to make sure it attracts enough eye-balls. The earlier one 
was about 3.0.0 specifically and I don’t think enough people were watching that.

I’ll try to summarize a bit.

# Today’s state of release numbering and ordering:
So far, all the releases we have done, we have followed a simple timeline 
ordering. By this I mean that we always lined up releases one after another. 
Due to this, it was relatively simple to follow the causal ordering of 
releases, and we could answer ordering questions like “when was this patch 
first committed”.

The first time this started getting hairy was with parallel 2.6.x and 2.7.x 
releases. We managed the confusion by ordering them one after another: 2.7.1 -> 
2.6.2 -> 2.6.3 -> 2.7.2 -> 2.6.4 -> 2.7.3. This worked okay with two concurrent 
releases. 

Yes, by doing this, we could no longer answer questions by simply looking 
at the release numbers. But Sangjin / Junping / myself who did these 2.x 
releases ordered them one after another to avoid further confusion to 
developers and still retain the ability to just go back, look at the history 
and quickly answer the question of "what happened before and what happened 
after”. It wasn’t perfect, but we did what we did to keep it under control.

# What’s coming
Obviously, with 2.8.0 and 3.0.0-alpha1 release trains starting, this 
confusion goes to a whole new level. We’ll have 2.6.x, 2.7.x, 2.8.x, 3.0.0.x 
(and 2.9.x if we chose to make one) and following the ordering becomes a 
nightmare. The related problems that were discussed in the original thread was 
about how we mark fix-versions for patches, and how we generate change-logs - 
the answers for both of these follow the solution to the root problem of 
concurrent releases.

# Proposals?
There were some discussions about semantic versioning in the thread form 
which I forked this one from, but it’s not clear to me how semantic versioning 
supports concurrent release trains.

IAC, it’s time we look at this afresh and if need be, make some new rules 
before we make more of these releases and make it that much harder to follow 
for everyone.

Thanks
+Vinod
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Yes/No newbie question on contributing

[Vinod Kumar Vavilapalli]

The short answer is that it is expected to pass without any errors.

On branch-2.x, that command passes cleanly without any errors though it takes 
north of 10 minutes. Note that I run it with -DskipTests - you don’t want to 
wait for all the unit tests to run, that’ll take too much time. I expect trunk 
to be the same too.

Which branch are you running this against? What errors are you seeing? If it is 
unit-tests you are talking about, you can instead run with skipTests, run only 
specific tests or all tests in the module you are touching, make sure they pass 
and then let Jenkins infrastructure run the remaining tests when you submit the 
patch.

+Vinod

> On Jul 26, 2016, at 11:41 AM, Martin Rosse  wrote:
> 
> Hi,
> 
> In the How To Contribute doc, it says:
> 
> "Try getting the project to build and test locally before writing code"
> 
> So, just to be 100% certain before I keep troubleshooting things, this
> means I should be able to run
> 
> mvn clean install -Pdist -Dtar
> 
> without getting any failures or errors at all...none...zero, right?
> 
> I am surprised at how long this is taking as errors keep cropping up.
> Should I just expect it to really take many hours (already at 10+) to work
> through these issues? I am setting up a dev environment on an Ubuntu 14.04
> 64-bit desktop from the AWS marketplace running on EC2.
> 
> It would seem it's an obvious YES answer, but given the time investment
> I've been making I just wanted to be absolutely sure before continuing.
> 
> I thought it possible that maybe some errors, depending on their nature,
> can be overlooked, and that other developers may be doing that in practice.
> And hence perhaps I should as well to save time. Yes or No??
> 
> Thank you,
> 
> Martin


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

+1

Thanks
+Vinod

> On Jul 26, 2016, at 7:39 AM, Wangda Tan  wrote:
> 
> lets try to use both jdiff and the new tool and compare results because this 
> is the first time with the new tool.
> 
> Appreciate your time to help us about this effort!

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He <j...@hortonworks.com> wrote:
> 
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
> 
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
> 
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
> 
> Jian
> 
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:
> 
> Sounds good to me :).
> 
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
> 
> Will keep this thread updated.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>> wrote:
> 
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
> 
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
> 
> Thanks,
> Sangjin
> 
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:
> 
> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> for
> every releases.
> 
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>> wrote:
> 
> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
> 
> But I see a lot of JIRAs with no patches or very stale patches. It
> would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
> 
> Thanks,
> Sangjin
> 
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> <ste...@hortonworks.com<mailto:ste...@hortonworks.com>
> 
> wrote:
> 
> 
> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org<mailto:vino...@apache.org>>
> wrote:
> 
> We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start
> with
> reviewing Patch available tickets, that’ll be great.
> 
> 
> 
> 
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
> 
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure
> instrumentation
> 
> https://issues.apache.org/jira/browse/HADOOP-13028
> 
> 
> 
> 
> 
> 
> 
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] 2.6.x line releases

[Vinod Kumar Vavilapalli]

It wasn’t a vote. PMC doesn’t vote on code-changes.

It was an extensive community discussion. We arrived at that decision to 
reflect the reality that not many users were running JDK6 anyways at that point 
of time.

Sorry to bust the bubble, this is not one of those things that supports your 
propaganda about how branch-2 is bust.

+Vinod

> On Jul 22, 2016, at 3:59 PM, Allen Wittenauer  
> wrote:
> 
> (It's interesting that ~2 years later, we're still dealing with the fallout 
> of the JRE compatibility break in 2.7.  I wonder how many of those PMCs who 
> voted for it are still actively involved.)