Re: [VOTE] Merge YARN-3926 (resource profile) to trunk

Really looking forward to getting this in. Couple of questions: * Can you maybe comment a bit on the type of scale testing done ? Specifically, the number of resources tested with and any point where it is discovered that performance might take a hit. Also, given that we do not have AM's that

2017-08-25 Hadoop 3 release status update

Hi all, I've written up a status report for the current state of Hadoop 3 on the wiki. I've also pasted it below for your convenience. Cheers, Andrew https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates 2017-08-25 Another month flew by without an update. This is

Re: Branch merges and 3.0.0-beta1 scope

Jonathan, thanks for the heads up. I don't have much familiarity with YARN, but gave the PBs and pom changes a look, and left a few small comments on the umbrella JIRA. This seems like a smaller change than some of the other branch merges we're discussing, but I'm again reticent about adding

Re: Branch merges and 3.0.0-beta1 scope

Here's a summary of some 1-on-1 conversations I had with contributors of the different features I'm tracking. Storage Policy Satisfier (Uma Gangumalla) * Target version: 3.1.0, maybe 3.0.0 GA * We're resolving some design discussions on JIRA. Plan is to do some MVP work on the API to get this

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Hi Subru, Thanks for your vote and your response! Regarding your question about merging to branch2, I think, after the merge to trunk is done, it will be good to merge to branch-2 as soon as we can. A lot of testing with trunk has been done presently and it will be good to repeat the same with

Re: Branch merges and 3.0.0-beta1 scope

> From a release management perspective, it's *extremely* reasonable to block > the inclusion of new features a month from the planned release date. A > typical software development lifecycle includes weeks of feature freeze and > weeks of code freeze. It is no knock on any developer or any

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Thanks Subru for voting. > What are the timelines you are looking for getting this into branch-2? We haven't yet decided on it and were thinking of discussing this in detail within the team after merge to trunk. The timelines would depend on whether we release whatever we merge to trunk in 2.9

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

+1 (binding). I have been following the effort and had few design discussions around the team especially about how it integrates with Federation. Overall I feel it's a welcome improvement to YARN. What are the timelines you are looking for getting this into branch-2? Thanks, Subru On Fri, Aug

Re: [DISCUSS] Branches and versions for Hadoop 3

Allen Wittenauer wrote: > Doesn't this place an undue burden on the contributor with the first > incompatible patch to prove worthiness? What happens if it is decided that > it's not good enough? It is a burden for that first, "this can't go anywhere else but 4.x" change, but arguably that

Re: [DISCUSS] Branches and versions for Hadoop 3

Hi Andrew, Thanks for updating the proposal, +1. - Wangda On Fri, Aug 25, 2017 at 12:16 PM, Allen Wittenauer wrote: > > > On Aug 25, 2017, at 10:36 AM, Andrew Wang > wrote: > > > Until we need to make incompatible changes, there's no

Re: Branch merges and 3.0.0-beta1 scope

Hi Andrew, Thanks for starting the discussion - we have a feature YARN-5734 for API based scheduler configuration that I feel is pretty close to merge (also "a few weeks"). It's almost completely code and API additions and we were careful to design it so that it's compatible (feature is also

Re: [DISCUSS] Branches and versions for Hadoop 3

> On Aug 25, 2017, at 10:36 AM, Andrew Wang wrote: > Until we need to make incompatible changes, there's no need for > a Hadoop 4.0 version. Some questions: Doesn't this place an undue burden on the contributor with the first incompatible patch to prove

Re: Branch merges and 3.0.0-beta1 scope

Resource profile is similar to TSv2, the feature is: - Alpha feature, we will not freeze new added APIs. And all added APIs are explicitly marked to @Unstable. - Allow rolling upgrade from branch-2. - Touched existing code, but we have, and will continue tests to make sure changes are safe.

Re: [DISCUSS] Branches and versions for Hadoop 3

+1 for this branching proposal.-Eric From: Andrew Wang To: "common-...@hadoop.apache.org" ; "mapreduce-dev@hadoop.apache.org" ; "hdfs-...@hadoop.apache.org" ;

Re: Branch merges and 3.0.0-beta1 scope

On 25 August 2017 at 22:39, Andrew Wang wrote: > Hi Rohith, > > Given that we're advertising TSv2 as an alpha feature, I think we're > allowed to break compatibility. Let's make sure this is clear in the > release notes and documentation. > > That said, with TSv2 phase

[DISCUSS] Branches and versions for Hadoop 3

Hi folks, With 3.0.0-beta1 fast approaching, I wanted to go over the proposed branching strategy. In the early 2.x days, moving trunk immediately to 3.0.0 was a mistake. branch-2 and trunk were virtually identical, and increased backport complexity. Until we need to make incompatible changes,

Re: Branch merges and 3.0.0-beta1 scope

Hi Jason, I agree with this proposal. I'll start another email thread spelling this out, and gather additional feedback. Best, Andrew On Fri, Aug 25, 2017 at 6:27 AM, Jason Lowe wrote: > Andrew Wang wrote: > > >> This means I'll cut branch-3 and >> branch-3.0, and move trunk

Re: Branch merges and 3.0.0-beta1 scope

Hi Rohith, Given that we're advertising TSv2 as an alpha feature, I think we're allowed to break compatibility. Let's make sure this is clear in the release notes and documentation. That said, with TSv2 phase 2, is the API going to be frozen? The umbrella JIRA refers to "TSv2 alpha2" which

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

+1 (binding) I've built the current branch, and checked out a few basic areas including documentation. Also perused the most recent changes that went in. Thanks much for the great team work! I look forward to seeing it in action. Regards, Sangjin On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

+1 from my side. More from the perspective of ensuring there is no impact of ATSv2 when it is off (by default), I deployed the latest YARN-5355 bits into a few clusters and ran internal Smoke tests. The tests shows no impact when ATSv2 is off. Best, Haibo On Thu, Aug 24, 2017 at 7:51 AM, Sunil

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/ [Aug 24, 2017 7:26:37 AM] (jzhuge) HDFS-12318. Fix IOException condition for openInfo in DFSInputStream. [Aug 24, 2017 11:04:38 AM] (bibinchundatt) YARN-7074. Fix NM state store update comment.

Re: Branch merges and 3.0.0-beta1 scope

Andrew Wang wrote: > This means I'll cut branch-3 and > branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open > up development for Hadoop 3.1.0 and 4.0.0. I can see a need for branch-3.0, but please do not create branch-3. Doing so will relegate trunk back to the "patch

[jira] [Created] (MAPREDUCE-6947) Moving logging APIs over to slf4j in hadoop-mapreduce-examples

Gergely Novák created MAPREDUCE-6947: Summary: Moving logging APIs over to slf4j in hadoop-mapreduce-examples Key: MAPREDUCE-6947 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6947

Re: Branch merges and 3.0.0-beta1 scope

Hi Andrew Thanks for update on release plan! I would like to discuss specifically regarding compatibility of releases. What is the compatibility to be maintained for GA if we don't merge to beta1 release? IIUC, till now all the releases were alpha where compatibility was not that important. All