Re: [VOTE] Merge YARN-3926 (resource profile) to trunk

[Arun Suresh]

Really looking forward to getting this in.

Couple of questions:
* Can you maybe comment a bit on the type of scale testing done ?
Specifically, the number of resources tested with and any point where it is
discovered that performance might take a hit. Also, given that we do not
have AM's that currently use this feature, can you folks point me to any
test application / framework or has this been integrated with MapReduce ?
* Is there a plan to merge this with branch-2 ? - Since we would like to
see this in 2.9.0 as well.

Just to clarify, I am a +1 for merging, irrespective of the above - given
that this is an opt-in feature after all. I am just eager to start using it
:)

Cheers
-Arun


On Thu, Aug 24, 2017 at 10:54 AM, Sunil G  wrote:

> Thank you very much Varun Vasudev, Wangda Tan, Daniel and all the folks who
> helped in getting this feature in this level.
>
> Starting with my +1 (binding).
>
>
> # Tested a 5 node cluster with resource profiles enabled/disabled (feature
> is disabled by default)
>
> # All apis added are marked as Unstable/Evolving (very few)
>
> # There is no compatibility break with older versions (we have added UT
> cases also to ensure same)
>
> # Performance tests were done using SLS and also with some tight loops unit
> tests. There is no much regression with current trunk.
>
> # Latest jenkins +1 on YARN-7013 for whole branch code.
>
> # Verified old RM UI and new YARN UI (newly added resources could be seen
> easily)
>
>
> Once again thanks all the folks who helped in getting this feature. Kudos!
>
>
> Thanks
>
> - Sunil
>
>
> On Thu, Aug 24, 2017 at 12:20 AM Wangda Tan  wrote:
>
> >  Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-3926 (Resource profile) to trunk. The vote will run
> for
> > 7 days and will end August 30 10:00 AM PDT.
> >
> > Briefly, YARN-3926 can extend resource model of YARN to support resource
> > types other than CPU and memory, so it will be a cornerstone of features
> > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139), FPGA
> > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In
> > addition to that, YARN-3926 allows admin to preconfigure resource
> profiles
> > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64 GB
> > disk>, so applications can request "m3.large" profile instead of
> specifying
> > all resource types’s values.
> >
> > There are 32 subtasks that were completed as part of this effort.
> >
> > This feature needs to be explicitly turned on before use. We paid close
> > attention to compatibility, performance, and scalability of this feature,
> > mentioned in [1], we didn't see observable performance regression in
> large
> > scale SLS (scheduler load simulator) executions and saw less than 5%
> > performance regression by using micro benchmark added by YARN-6775.
> >
> > This feature works from end-to-end (including UI/CLI/application/server),
> > we have setup a cluster with this feature turned on runs for several
> weeks,
> > we didn't see any issues by far.
> >
> > Merge JIRA: YARN-7013 (Jenkins gave +1 already).
> > Documentation: YARN-7056
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort including design discussion/development/reviews, etc.: Varun
> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> > Karthik Kambatla, Jason Lowe, Arun Suresh.
> >
> > Regards,
> > Wangda Tan
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/
> 201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%
> 3DBbp4G8inQZmaMg%40mail.gmail.com%3E
> >
>

2017-08-25 Hadoop 3 release status update

[Andrew Wang]

Hi all,

I've written up a status report for the current state of Hadoop 3 on the
wiki. I've also pasted it below for your convenience.

Cheers,
Andrew

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-08-25

Another month flew by without an update. This is a big one.

Red flags:

   - 11 blockers still on the dashboard, with some filed recently. Need to
   burn these down.
   - There are many branch merges proposals flying around for features that
   were not originally being tracked for beta1 and GA. Introducing new code
   always comes with risk, so I'm working with the different contributors
   involved to discuss target versions, confirm readiness, and define quality
   bars for merge.

Miscellaneous blockers:

   - HADOOP-14284  (Shade
   Guava everywhere): We have agreement to shade the yarn client JAR. Shading
   hadoop-hdfs is still being discussed.
   - HADOOP-13363  (Upgrade
   to protobuf 3): Waiting on the Guava shading first.
   - YARN-7076 : New
   blocker, we need an assignee.
   - YARN-7094  (Document
   that server-side graceful decom is currently not recommended): Robert has a
   patch up, needs review. This is a stopgap for the old blocker YARN-5464.
   - YARN-5536  (Multiple
   format support (JSON, etc.) for exclude node file in NM graceful
   decommission with timeout): Robert has a proposal that needs to be pushed
   on.

beta1 features:

   - Erasure coding
  - There are three must-dos. Two have patches, one might not be a
  must-do.
  - I pinged the pluggable policy JIRA to see if metadata and API
  compatibility is complete.
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
  - Sean has HDFS rolling upgrade scripts up, waiting on Ray to add
  some YARN/MR coverage too.
  - Need to do a final runthrough of the JACC reports for YARN and HDFS.
   - Classpath isolation (HADOOP-11656)
  - We're down to the wire on this, I pinged Sean for an update.
   - Compat guide (HADOOP-13714
   )
  - I pinged the JIRA on this too, no updated patch since May

Features under discussion:

I discussed with a number of lead contributors on these features that were
previously not on my radar.

3.0.0-beta1:

   - YARN native services (Jian He)
  - I was convinced that this is very separate from the core. I'll get
  someone from Cloudera to run it through our integration tests to
verify it
  doesn't break anything downstream, then happy to merge.
   - TSv2 alpha 2 (Vrushali C)
   - Despite being called "alpha 2", this is more like "beta" in terms of
  readiness. Twitter is planning to roll it out to production. Seems quite
  done.
  - I double checked with Haibo, and he successfully ran it through our
  internal integration testing.

3.0.0 GA:

   - Resource profiles (Wangda Tan)
  - Alpha feature, APIs are not stable yet. Has some compatible PB
  changes, will verify rolling upgrade from branch-2. Touches some
core parts
  of YARN.
  - Decided that it's too close to beta1 for this, we're going to test
  it a lot and make sure it's ready for 3.0.0 GA.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
  run in production at MSFT.
  - If it passes Cloudera internal integration testing, I'm fine
  putting this in for GA.

3.1.0:

   - Storage Policy Satisfier (Uma Gangumalla)
  - We're resolving some design discussions on JIRA. Plan is to do some
  MVP work on the API to get this into 3.1, and if we're happy with the
  second phase, consider for 3.0 GA.
   - HDFS tiered storage (Chris Douglas):
   - This touches some core stuff, and the write path is still being worked
  on. Still somewhat useful with just the read path. Targeting at
3.1.0 gives
  enough time to wrap this up.

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Jonathan, thanks for the heads up. I don't have much familiarity with YARN,
but gave the PBs and pom changes a look, and left a few small comments on
the umbrella JIRA.

This seems like a smaller change than some of the other branch merges we're
discussing, but I'm again reticent about adding scope if we can avoid it.

In your mind, is this truly a "must-have" for 3.0? It looks compatible, and
thus something we could add in a minor release like 2.9 or 3.1.

Best,
Andrew

On Fri, Aug 25, 2017 at 12:31 PM, Jonathan Hung 
wrote:

> Hi Andrew,
>
> Thanks for starting the discussion - we have a feature YARN-5734 for API
> based scheduler configuration that I feel is pretty close to merge (also "a
> few weeks"). It's almost completely code and API additions and we were
> careful to design it so that it's compatible (feature is also turned off by
> default). Hoping to get this in before 3.0.0-GA. Just wanted to send this
> note so that we are not caught off guard by this feature.
>
> Thanks!
>
>
> Jonathan Hung
>
> On Fri, Aug 25, 2017 at 11:06 AM, Wangda Tan  wrote:
>
>> Resource profile is similar to TSv2, the feature is:
>> - Alpha feature, we will not freeze new added APIs. And all added APIs are
>> explicitly marked to @Unstable.
>> - Allow rolling upgrade from branch-2.
>> - Touched existing code, but we have, and will continue tests to make sure
>> changes are safe.
>>
>> Discussed with Andrew offline, we decided to not put this to beta1 since
>> beta1 is not far away. But we want to put it before GA if sufficient tests
>> are done.
>>
>> Thanks,
>> Wangda
>>
>>
>>
>> On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>>
>> > On 25 August 2017 at 22:39, Andrew Wang 
>> wrote:
>> >
>> > > Hi Rohith,
>> > >
>> > > Given that we're advertising TSv2 as an alpha feature, I think we're
>> > > allowed to break compatibility. Let's make sure this is clear in the
>> > > release notes and documentation.
>> > >
>> >
>> > > That said, with TSv2 phase 2, is the API going to be frozen? The
>> umbrella
>> > > JIRA refers to "TSv2 alpha2" which indicated to me it was still
>> > alpha-level
>> > > quality and stability.
>> > >
>> > YES, We have decided to freeze API's. I do not think we make any
>> > compatibility break in future.
>> >
>> >
>> >
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> >
>>
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Here's a summary of some 1-on-1 conversations I had with contributors of
the different features I'm tracking.

Storage Policy Satisfier (Uma Gangumalla)
* Target version: 3.1.0, maybe 3.0.0 GA
* We're resolving some design discussions on JIRA. Plan is to do some MVP
work on the API to get this into 3.1, and if we're happy with the second
phase, consider for 3.0 GA.

YARN native services (Jian He)
* Target version: 3.0.0-beta1 as an alpha feature
* I was convinced that this is very separate from the core. I'll get
someone from Cloudera to run it through our integration tests to verify it
doesn't break anything downstream, then happy to merge.

Resource profiles (Wangda Tan)
* Target version: 3.0.0 GA
* Already provided update above, we're going to test it a lot and target
for GA.

HDFS router-based federation (Chris Douglas)
* Target version: 3.0.0 GA
* This is like YARN federation, very separate and doesn't add new APIs, run
in production.
* If it passes our internal integration testing, I'm fine putting this in
late.

HDFS tiered storage (Chris Douglas):
* Target version: 3.1.0
* This touches some core stuff, and the write path is still being worked
on. Still somewhat useful with just the read path. Targeting at 3.1.0 gives
enough time to wrap this up.

TSv2 phase 2 (Vrushali C)
* Target version: 3.0.0-beta1
* This is more like "beta" in terms of readiness, Twitter is planning to
roll it out to production.
* I double checked with Haibo, and he successfully ran it through our
internal integration testing.

Thanks to everyone for meeting with me on short notice, and being very
reasonable about target versions and quality bars. If I mischaracterized
any of our discussions, please reach out or comment.

The branching and versioning discussion is still proceeding. I'd ask that
the pending merge VOTEs watch this carefully; I'm hoping to resolve the
discussion and branch before the VOTEs close, but let's make sure the
branches and versions are ready before doing the actual merges.

Thanks,
Andrew

On Fri, Aug 25, 2017 at 11:06 AM, Wangda Tan  wrote:

> Resource profile is similar to TSv2, the feature is:
> - Alpha feature, we will not freeze new added APIs. And all added APIs are
> explicitly marked to @Unstable.
> - Allow rolling upgrade from branch-2.
> - Touched existing code, but we have, and will continue tests to make sure
> changes are safe.
>
> Discussed with Andrew offline, we decided to not put this to beta1 since
> beta1 is not far away. But we want to put it before GA if sufficient tests
> are done.
>
> Thanks,
> Wangda
>
>
>
> On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
>> On 25 August 2017 at 22:39, Andrew Wang  wrote:
>>
>> > Hi Rohith,
>> >
>> > Given that we're advertising TSv2 as an alpha feature, I think we're
>> > allowed to break compatibility. Let's make sure this is clear in the
>> > release notes and documentation.
>> >
>>
>> > That said, with TSv2 phase 2, is the API going to be frozen? The
>> umbrella
>> > JIRA refers to "TSv2 alpha2" which indicated to me it was still
>> alpha-level
>> > quality and stability.
>> >
>> YES, We have decided to freeze API's. I do not think we make any
>> compatibility break in future.
>>
>>
>>
>> >
>> > Best,
>> > Andrew
>> >
>>
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Vrushali C]

Hi Subru,

Thanks for your vote and your response!

Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.

thanks
Vrushali







On Fri, Aug 25, 2017 at 2:28 PM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K 
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee  wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen 
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2...@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > > 

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

> From a release management perspective, it's *extremely* reasonable to block 
> the inclusion of new features a month from the planned release date. A 
> typical software development lifecycle includes weeks of feature freeze and 
> weeks of code freeze. It is no knock on any developer or any feature to say 
> that we should not include something in 3.0.0.


We have never followed the ‘typical' lifecycle that I am guessing you are 
referring to. If we are, you'll need to publish some of the following: a 
feature freeze date, blockers-criticals-only-from-now date, testing-finish 
date, documentation-finish date, final release date and so on.

What we do with Apache releases typically is instead we say ‘this' is roughly 
when we want to release, and roughly what features must land and let the rest 
figure out itself.

Neither is right or wrong. If we want to change the process, we should 
communicate as such.

Proposing a feature freeze date on the fly is only going to confuse people.


> I've been very open and clear about the goals, schedule, and scope of 3.0.0 
> over the last year plus. The point of the extended alpha process was to get 
> all our features in during alpha, and the alpha merge window has been open 
> for a year. I'm unmoved by arguments about how long a feature has been worked 
> on. None of these were not part of the original 3.0.0 scope, and our users 
> have been waiting even longer for big-ticket 3.0 items like JDK8 and HDFS EC 
> that were part of the discussed scope.


Except our schedule is so fluid (not due to the release management process to 
be fair) that it is hard for folks to plan their features. IIRC, our schedule 
was a GA release beginning of this year. Again, this is not a critique of 3.0 
release process - I have myself done enough releases to know that sticking to a 
date and herding the crowd has been an extremely hard job.

The only way out of this is get 3.0 out and stick to 1-2 month minor releases. 
May be we start that by planning a 3.1 right now and push one in January 
assuming 3.0 GA happens in November. Cadence is a habit.


> I see that two VOTEs have gone out since I was out. I still plan to follow 
> the proposal in my original email. This means I'll cut branch-3 and 
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open up 
> development for Hadoop 3.1.0 and 4.0.0.
> 
> I'm reaching out to the lead contributor of each of these features 
> individually to discuss. We need to close on this quickly, and email is too 
> low bandwidth at this stage.


My only concern in all of this is if some of these branch contributors decide 
that they don’t want and then proceed to having a parallel release and cause 
pains for everyone. This is what I tried avoiding parallel 2.9 and 2.10 offline 
and that’s what I am trying to do here. For now, I don’t have a horse in this 
race - that’s between you and the branch-contributors in question for now. If 
you can reach an agreement, we are all good for it.

+Vinod



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[varunsax...@apache.org]

Thanks Subru for voting.


> What are the timelines you are looking for getting this into branch-2?

We haven't yet decided on it and were thinking of discussing this in detail
within the team after merge to trunk.
The timelines would depend on whether we release whatever we merge to trunk
in 2.9 or would we want to get in few other features which people would
like to see in 2.9
This would require some discussion with the stakeholders.
We were thinking of having a short discussion with you guys as well to find
out whether there are any further gaps in ATSv2 with respect to federation
support and if they can be filled before 2.9 release.

Assuming 2.9 is targetted for October end, we would have to start a merge
vote in September end or October 1st week which leaves us with very little
time to take up large changes anyways.

We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
we rebase with branch-2 regularly. So, if we decide to merge in branch-2
without any additional changes, we would be able to go for branch-2 merge
discussion and vote almost immediately.

Regards,
Varun Saxena.




On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K  wrote:

> +1 (binding).
>
> I have been following the effort and had few design discussions around the
> team especially about how it integrates with Federation. Overall I feel
> it's a welcome improvement to YARN.
>
> What are the timelines you are looking for getting this into branch-2?
>
> Thanks,
> Subru
>
> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee  wrote:
>
> > +1 (binding)
> >
> > I've built the current branch, and checked out a few basic areas
> including
> > documentation. Also perused the most recent changes that went in.
> >
> > Thanks much for the great team work! I look forward to seeing it in
> action.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen 
> > wrote:
> >
> > > +1 from my side.
> > >
> > > More from the perspective of ensuring there is no impact of ATSv2 when
> it
> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> > > clusters and ran internal Smoke tests. The tests shows no impact when
> > ATSv2
> > > is off.
> > >
> > > Best,
> > > Haibo
> > >
> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:
> > >
> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> > this
> > > > happen. Great work, really appreciate the same!!
> > > >
> > > > +1 (binding) from my side:
> > > >
> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
> > Seems
> > > > fine.
> > > > # Based on code, looks like all apis are compatible.
> > > > # REST api docs looks fine as well, I guess we could improve that a
> bit
> > > > more post merge as well.
> > > > # Adding to additional thoughts which are discussed here, native
> > service
> > > > also could publish events to atsv2. I think that work is also
> happened
> > in
> > > > branch.
> > > >
> > > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > > >
> > > > Thanks
> > > > Sunil
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > > vrushalic2...@gmail.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> merge
> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > vote
> > > > will
> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > > >
> > > > > We have previously completed one merge onto trunk [3] and Timeline
> > > > Service
> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > > >
> > > > > Since then, we have been working on extending the capabilities of
> > > > Timeline
> > > > > Service v2 in a feature branch [2] for a while, and we are
> reasonably
> > > > > confident that the state of the feature meets the criteria to be
> > merged
> > > > > onto trunk and we'd love folks to get their hands on it in a test
> > > > capacity
> > > > > and provide valuable feedback so that we can make it
> > production-ready.
> > > > >
> > > > > In a nutshell, Timeline Service v.2 delivers significant
> scalability
> > > and
> > > > > usability improvements based on a new architecture. What we would
> > like
> > > to
> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > > complete end-to-end read/write flow with security and read level
> > > > > authorization via whitelists. You should be able to start setting
> it
> > up
> > > > and
> > > > > testing it.
> > > > >
> > > > > At a high level, the following are the key features that have been
> > > > > implemented since alpha1:
> > > > > - Security via Kerberos Authentication and delegation tokens
> > > > > - Read side simple authorization via whitelist
> > > > > - Client configurable entity sort 

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Subramaniam V K]

+1 (binding).

I have been following the effort and had few design discussions around the
team especially about how it integrates with Federation. Overall I feel
it's a welcome improvement to YARN.

What are the timelines you are looking for getting this into branch-2?

Thanks,
Subru

On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee  wrote:

> +1 (binding)
>
> I've built the current branch, and checked out a few basic areas including
> documentation. Also perused the most recent changes that went in.
>
> Thanks much for the great team work! I look forward to seeing it in action.
>
> Regards,
> Sangjin
>
> On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen 
> wrote:
>
> > +1 from my side.
> >
> > More from the perspective of ensuring there is no impact of ATSv2 when it
> > is off (by default), I deployed the latest YARN-5355 bits into a few
> > clusters and ran internal Smoke tests. The tests shows no impact when
> ATSv2
> > is off.
> >
> > Best,
> > Haibo
> >
> > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:
> >
> > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> this
> > > happen. Great work, really appreciate the same!!
> > >
> > > +1 (binding) from my side:
> > >
> > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > # Accessed new YARN UI which shows various flows/flow activity etc.
> Seems
> > > fine.
> > > # Based on code, looks like all apis are compatible.
> > > # REST api docs looks fine as well, I guess we could improve that a bit
> > > more post merge as well.
> > > # Adding to additional thoughts which are discussed here, native
> service
> > > also could publish events to atsv2. I think that work is also happened
> in
> > > branch.
> > >
> > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > >
> > > Thanks
> > > Sunil
> > >
> > >
> > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > vrushalic2...@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> vote
> > > will
> > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > >
> > > > We have previously completed one merge onto trunk [3] and Timeline
> > > Service
> > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >
> > > > Since then, we have been working on extending the capabilities of
> > > Timeline
> > > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > > confident that the state of the feature meets the criteria to be
> merged
> > > > onto trunk and we'd love folks to get their hands on it in a test
> > > capacity
> > > > and provide valuable feedback so that we can make it
> production-ready.
> > > >
> > > > In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > > > usability improvements based on a new architecture. What we would
> like
> > to
> > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > complete end-to-end read/write flow with security and read level
> > > > authorization via whitelists. You should be able to start setting it
> up
> > > and
> > > > testing it.
> > > >
> > > > At a high level, the following are the key features that have been
> > > > implemented since alpha1:
> > > > - Security via Kerberos Authentication and delegation tokens
> > > > - Read side simple authorization via whitelist
> > > > - Client configurable entity sort ordering
> > > > - Richer REST APIs for apps, app attempts, containers, fetching
> metrics
> > > by
> > > > timerange, pagination, sub-app entities
> > > > - Support for storing sub-application entities (entities that exist
> > > outside
> > > > the scope of an application)
> > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > prefixes,
> > > > configurable hbase cluster
> > > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > > - Uses latest stable HBase release 1.2.6
> > > >
> > > > There are a total of 82 subtasks that were completed as part of this
> > > > effort.
> > > >
> > > > We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > > > does not impact existing functionality when disabled (by default).
> > > >
> > > > Special thanks to a team of folks who worked hard and contributed
> > towards
> > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >
> > > > Regards,
> > > > Vrushali
> > > >
> > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > 383.html
> > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > [4] 

Re: [DISCUSS] Branches and versions for Hadoop 3

[Jason Lowe]

Allen Wittenauer wrote:


> Doesn't this place an undue burden on the contributor with the first
> incompatible patch to prove worthiness?  What happens if it is decided that
> it's not good enough?


It is a burden for that first, "this can't go anywhere else but 4.x"
change, but arguably that should not be a change done lightly anyway.  (Or
any other backwards-incompatible change for that matter.)  If it's worth
committing then I think it's perfectly reasonable to send out the dev
announce that there's reason for trunk to diverge from 3.x, cut branch-3,
and move on.  This is no different than Andrew's recent announcement that
there's now a need for separating trunk and the 3.0 line based on what's
about to go in.

I do not think it makes sense to pay for the maintenance overhead of two
nearly-identical lines with no backwards-incompatible changes between them
until we have the need.  Otherwise if past trunk behavior is any
indication, it ends up mostly enabling people to commit to just trunk,
forgetting that the thing they are committing is perfectly valid for
branch-3.  If we can agree that trunk and branch-3 should be equivalent
until an incompatible change goes into trunk, why pay for the commit
overhead and potential for accidentally missed commits until it is really
necessary?

How many will it take before the dam will break?  Or is there a timeline
> going to be given before trunk gets set to 4.x?


I think the threshold count for the dam should be 1.  As soon as we have a
JIRA that needs to be committed to move the project forward and we cannot
ship it in a 3.x release then we create branch-3 and move trunk to 4.x.
As for a timeline going to 4.x, again I don't see it so much as a "baking
period" as a "when we need it" criteria.  If we need it in a week then we
should cut it in a week.  Or a year then a year.  It all depends upon when
that 4.x-only change is ready to go in.

Given the number of committers that openly ignore discussions like this,
> who is going to verify that incompatible changes don't get in?
>

The same entities who are verifying other bugs don't get in, i.e.: the
committers and the Hadoop QA bot running the tests.  Yes, I know that means
it's inevitable that compatibility breakages will happen, and we can and
should improve the automation around compatibility testing when possible.
But I don't think there's a magic bullet for preventing all compatibility
bugs from being introduced, just like there isn't one for preventing
general bugs.  Does having a trunk branch separate but essentially similar
to branch-3 make this any better?

Longer term:  what is the PMC doing to make sure we start doing major
> releases in a timely fashion again?  In other words, is this really an
> issue if we shoot for another major in (throws dart) 2 years?
>

If we're trying to do semantic versioning then we shouldn't have a regular
cadence for major releases unless we have a regular cadence of changes that
break compatibility.  I'd hope that's not something we would strive
towards.  I do agree that we should try to be better about shipping
releases, major or minor, in a more timely manner, but I don't agree that
we should cut 4.0 simply based on a duration since the last major release.
The release contents and community's desire for those contents should
dictate the release numbering and schedule, respectively.

Jason


On Fri, Aug 25, 2017 at 2:16 PM, Allen Wittenauer 
wrote:

>
> > On Aug 25, 2017, at 10:36 AM, Andrew Wang 
> wrote:
>
> > Until we need to make incompatible changes, there's no need for
> > a Hadoop 4.0 version.
>
> Some questions:
>
> Doesn't this place an undue burden on the contributor with the
> first incompatible patch to prove worthiness?  What happens if it is
> decided that it's not good enough?
>
> How many will it take before the dam will break?  Or is there a
> timeline going to be given before trunk gets set to 4.x?
>
> Given the number of committers that openly ignore discussions like
> this, who is going to verify that incompatible changes don't get in?
>
> Longer term:  what is the PMC doing to make sure we start doing
> major releases in a timely fashion again?  In other words, is this really
> an issue if we shoot for another major in (throws dart) 2 years?
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Wangda Tan]

Hi Andrew,

Thanks for updating the proposal, +1.

- Wangda

On Fri, Aug 25, 2017 at 12:16 PM, Allen Wittenauer  wrote:

>
> > On Aug 25, 2017, at 10:36 AM, Andrew Wang 
> wrote:
>
> > Until we need to make incompatible changes, there's no need for
> > a Hadoop 4.0 version.
>
> Some questions:
>
> Doesn't this place an undue burden on the contributor with the
> first incompatible patch to prove worthiness?  What happens if it is
> decided that it's not good enough?
>
> How many will it take before the dam will break?  Or is there a
> timeline going to be given before trunk gets set to 4.x?
>
> Given the number of committers that openly ignore discussions like
> this, who is going to verify that incompatible changes don't get in?
>
> Longer term:  what is the PMC doing to make sure we start doing
> major releases in a timely fashion again?  In other words, is this really
> an issue if we shoot for another major in (throws dart) 2 years?
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Jonathan Hung]

Hi Andrew,

Thanks for starting the discussion - we have a feature YARN-5734 for API
based scheduler configuration that I feel is pretty close to merge (also "a
few weeks"). It's almost completely code and API additions and we were
careful to design it so that it's compatible (feature is also turned off by
default). Hoping to get this in before 3.0.0-GA. Just wanted to send this
note so that we are not caught off guard by this feature.

Thanks!


Jonathan Hung

On Fri, Aug 25, 2017 at 11:06 AM, Wangda Tan  wrote:

> Resource profile is similar to TSv2, the feature is:
> - Alpha feature, we will not freeze new added APIs. And all added APIs are
> explicitly marked to @Unstable.
> - Allow rolling upgrade from branch-2.
> - Touched existing code, but we have, and will continue tests to make sure
> changes are safe.
>
> Discussed with Andrew offline, we decided to not put this to beta1 since
> beta1 is not far away. But we want to put it before GA if sufficient tests
> are done.
>
> Thanks,
> Wangda
>
>
>
> On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
>
> > On 25 August 2017 at 22:39, Andrew Wang 
> wrote:
> >
> > > Hi Rohith,
> > >
> > > Given that we're advertising TSv2 as an alpha feature, I think we're
> > > allowed to break compatibility. Let's make sure this is clear in the
> > > release notes and documentation.
> > >
> >
> > > That said, with TSv2 phase 2, is the API going to be frozen? The
> umbrella
> > > JIRA refers to "TSv2 alpha2" which indicated to me it was still
> > alpha-level
> > > quality and stability.
> > >
> > YES, We have decided to freeze API's. I do not think we make any
> > compatibility break in future.
> >
> >
> >
> > >
> > > Best,
> > > Andrew
> > >
> >
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Allen Wittenauer]

> On Aug 25, 2017, at 10:36 AM, Andrew Wang  wrote:

> Until we need to make incompatible changes, there's no need for
> a Hadoop 4.0 version.

Some questions:

Doesn't this place an undue burden on the contributor with the first 
incompatible patch to prove worthiness?  What happens if it is decided that 
it's not good enough?

How many will it take before the dam will break?  Or is there a 
timeline going to be given before trunk gets set to 4.x?  

Given the number of committers that openly ignore discussions like 
this, who is going to verify that incompatible changes don't get in?

Longer term:  what is the PMC doing to make sure we start doing major 
releases in a timely fashion again?  In other words, is this really an issue if 
we shoot for another major in (throws dart) 2 years?
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Wangda Tan]

Resource profile is similar to TSv2, the feature is:
- Alpha feature, we will not freeze new added APIs. And all added APIs are
explicitly marked to @Unstable.
- Allow rolling upgrade from branch-2.
- Touched existing code, but we have, and will continue tests to make sure
changes are safe.

Discussed with Andrew offline, we decided to not put this to beta1 since
beta1 is not far away. But we want to put it before GA if sufficient tests
are done.

Thanks,
Wangda



On Fri, Aug 25, 2017 at 10:54 AM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> On 25 August 2017 at 22:39, Andrew Wang  wrote:
>
> > Hi Rohith,
> >
> > Given that we're advertising TSv2 as an alpha feature, I think we're
> > allowed to break compatibility. Let's make sure this is clear in the
> > release notes and documentation.
> >
>
> > That said, with TSv2 phase 2, is the API going to be frozen? The umbrella
> > JIRA refers to "TSv2 alpha2" which indicated to me it was still
> alpha-level
> > quality and stability.
> >
> YES, We have decided to freeze API's. I do not think we make any
> compatibility break in future.
>
>
>
> >
> > Best,
> > Andrew
> >
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Eric Payne]

+1 for this branching proposal.-Eric


  From: Andrew Wang 
 To: "common-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org"  
 Sent: Friday, August 25, 2017 12:36 PM
 Subject: [DISCUSS] Branches and versions for Hadoop 3
   
Hi folks,

With 3.0.0-beta1 fast approaching, I wanted to go over the proposed
branching strategy.

In the early 2.x days, moving trunk immediately to 3.0.0 was a mistake.
branch-2 and trunk were virtually identical, and increased backport
complexity. Until we need to make incompatible changes, there's no need for
a Hadoop 4.0 version.

Thus, here's a proposal of branches and versions:

trunk: 3.1.0-SNAPSHOT
branch-3.0: 3.0.0-beta1-SNAPSHOT
branch-2 and etc: remain as is

LMK questions/comments/etc. Appreciate your attentiveness; I'm hoping to
build consensus quickly since we have a number of open VOTEs for branch
merges.

Thanks,
Andrew


   

Re: Branch merges and 3.0.0-beta1 scope

[Rohith Sharma K S]

On 25 August 2017 at 22:39, Andrew Wang  wrote:

> Hi Rohith,
>
> Given that we're advertising TSv2 as an alpha feature, I think we're
> allowed to break compatibility. Let's make sure this is clear in the
> release notes and documentation.
>

> That said, with TSv2 phase 2, is the API going to be frozen? The umbrella
> JIRA refers to "TSv2 alpha2" which indicated to me it was still alpha-level
> quality and stability.
>
YES, We have decided to freeze API's. I do not think we make any
compatibility break in future.



>
> Best,
> Andrew
>

[DISCUSS] Branches and versions for Hadoop 3

[Andrew Wang]

Hi folks,

With 3.0.0-beta1 fast approaching, I wanted to go over the proposed
branching strategy.

In the early 2.x days, moving trunk immediately to 3.0.0 was a mistake.
branch-2 and trunk were virtually identical, and increased backport
complexity. Until we need to make incompatible changes, there's no need for
a Hadoop 4.0 version.

Thus, here's a proposal of branches and versions:

trunk: 3.1.0-SNAPSHOT
branch-3.0: 3.0.0-beta1-SNAPSHOT
branch-2 and etc: remain as is

LMK questions/comments/etc. Appreciate your attentiveness; I'm hoping to
build consensus quickly since we have a number of open VOTEs for branch
merges.

Thanks,
Andrew

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi Jason,

I agree with this proposal. I'll start another email thread spelling this
out, and gather additional feedback.

Best,
Andrew

On Fri, Aug 25, 2017 at 6:27 AM, Jason Lowe  wrote:

> Andrew Wang wrote:
>
>
>> This means I'll cut branch-3 and
>> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
>> up development for Hadoop 3.1.0 and 4.0.0.
>
>
> I can see a need for branch-3.0, but please do not create branch-3.  Doing
> so will relegate trunk back to the "patch purgatory" branch, a place where
> patches won't see a release for years.  Unless something is imminently
> going in that will break backwards compatibility and warrant a new 4.x
> release, I don't see the need to distinguish trunk from the 3.x line.
> Leaving trunk as the 3.x line means less branches to commit patches through
> and more testing of every patch since trunk would remain an active area for
> testing and releasing.  If we separate trunk and branch-3 then it's almost
> certain only-trunk patches will start to accumulate and never get any
> "real" testing until someone eventually decides it's time to go to Hadoop
> 4.x.  Looking back at trunk-as-3.x for an example, patches committed there
> in the early days after branch-2 was cut didn't see a release for almost 6
> years.
>
> My apologies if I've missed a feature that is just going to miss the 3.0
> release and will break compatibility when it goes in.  If so then we need
> to cut branch-3, but if not then here's my plea to hold off until we do
> need it.
>
> Jason
>
>
> On Thu, Aug 24, 2017 at 3:33 PM, Andrew Wang 
> wrote:
>
>> Glad to see the discussion continued in my absence :)
>>
>> From a release management perspective, it's *extremely* reasonable to
>> block
>> the inclusion of new features a month from the planned release date. A
>> typical software development lifecycle includes weeks of feature freeze
>> and
>> weeks of code freeze. It is no knock on any developer or any feature to
>> say
>> that we should not include something in 3.0.0.
>>
>> I've been very open and clear about the goals, schedule, and scope of
>> 3.0.0
>> over the last year plus. The point of the extended alpha process was to
>> get
>> all our features in during alpha, and the alpha merge window has been open
>> for a year. I'm unmoved by arguments about how long a feature has been
>> worked on. None of these were not part of the original 3.0.0 scope, and
>> our
>> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
>> HDFS EC that were part of the discussed scope.
>>
>> I see that two VOTEs have gone out since I was out. I still plan to follow
>> the proposal in my original email. This means I'll cut branch-3 and
>> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
>> up development for Hadoop 3.1.0 and 4.0.0.
>>
>> I'm reaching out to the lead contributor of each of these features
>> individually to discuss. We need to close on this quickly, and email is
>> too
>> low bandwidth at this stage.
>>
>> Best,
>> Andrew
>>
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi Rohith,

Given that we're advertising TSv2 as an alpha feature, I think we're
allowed to break compatibility. Let's make sure this is clear in the
release notes and documentation.

That said, with TSv2 phase 2, is the API going to be frozen? The umbrella
JIRA refers to "TSv2 alpha2" which indicated to me it was still alpha-level
quality and stability.

Best,
Andrew

On Thu, Aug 24, 2017 at 11:47 PM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> Hi Andrew
>
> Thanks for update on release plan!
>
> I would like to discuss specifically regarding compatibility of releases.
> What is the compatibility to be maintained for GA if we don't merge to
> beta1 release? IIUC, till now all the releases were alpha where
> compatibility was not that important. All the public interfaces are
> subjected to modifications. Once we release beta, compatibility would be a
> matter.
> During this gap i.e between beta-GA release, should we maintain
> compatibility ?
> If my understanding is right then TSv2 have to be merged with beta1
> release. In TSv2 phase-2, we have compatibility changes from phase-1.
>
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 25 August 2017 at 02:03, Andrew Wang  wrote:
>
> > Glad to see the discussion continued in my absence :)
> >
> > From a release management perspective, it's *extremely* reasonable to
> block
> > the inclusion of new features a month from the planned release date. A
> > typical software development lifecycle includes weeks of feature freeze
> and
> > weeks of code freeze. It is no knock on any developer or any feature to
> say
> > that we should not include something in 3.0.0.
> >
> > I've been very open and clear about the goals, schedule, and scope of
> 3.0.0
> > over the last year plus. The point of the extended alpha process was to
> get
> > all our features in during alpha, and the alpha merge window has been
> open
> > for a year. I'm unmoved by arguments about how long a feature has been
> > worked on. None of these were not part of the original 3.0.0 scope, and
> our
> > users have been waiting even longer for big-ticket 3.0 items like JDK8
> and
> > HDFS EC that were part of the discussed scope.
> >
> > I see that two VOTEs have gone out since I was out. I still plan to
> follow
> > the proposal in my original email. This means I'll cut branch-3 and
> > branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will
> open
> > up development for Hadoop 3.1.0 and 4.0.0.
> >
> > I'm reaching out to the lead contributor of each of these features
> > individually to discuss. We need to close on this quickly, and email is
> too
> > low bandwidth at this stage.
> >
> > Best,
> > Andrew
> >
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Sangjin Lee]

+1 (binding)

I've built the current branch, and checked out a few basic areas including
documentation. Also perused the most recent changes that went in.

Thanks much for the great team work! I look forward to seeing it in action.

Regards,
Sangjin

On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen  wrote:

> +1 from my side.
>
> More from the perspective of ensuring there is no impact of ATSv2 when it
> is off (by default), I deployed the latest YARN-5355 bits into a few
> clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
> is off.
>
> Best,
> Haibo
>
> On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:
>
> > Thank you very much Vrushali, Rohith, Varun and other folks who made this
> > happen. Great work, really appreciate the same!!
> >
> > +1 (binding) from my side:
> >
> > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> > fine.
> > # Based on code, looks like all apis are compatible.
> > # REST api docs looks fine as well, I guess we could improve that a bit
> > more post merge as well.
> > # Adding to additional thoughts which are discussed here, native service
> > also could publish events to atsv2. I think that work is also happened in
> > branch.
> >
> > Looking forward to a much wider adoption of ATSv2 with more projects.
> >
> > Thanks
> > Sunil
> >
> >
> > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > vrushalic2...@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> 383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Haibo Chen]

+1 from my side.

More from the perspective of ensuring there is no impact of ATSv2 when it
is off (by default), I deployed the latest YARN-5355 bits into a few
clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
is off.

Best,
Haibo

On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:

> Thank you very much Vrushali, Rohith, Varun and other folks who made this
> happen. Great work, really appreciate the same!!
>
> +1 (binding) from my side:
>
> # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> fine.
> # Based on code, looks like all apis are compatible.
> # REST api docs looks fine as well, I guess we could improve that a bit
> more post merge as well.
> # Adding to additional thoughts which are discussed here, native service
> also could publish events to atsv2. I think that work is also happened in
> branch.
>
> Looking forward to a much wider adoption of ATSv2 with more projects.
>
> Thanks
> Sunil
>
>
> On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/

[Aug 24, 2017 7:26:37 AM] (jzhuge) HDFS-12318. Fix IOException condition for 
openInfo in DFSInputStream.
[Aug 24, 2017 11:04:38 AM] (bibinchundatt) YARN-7074. Fix NM state store update 
comment. Contributed by Botong
[Aug 24, 2017 3:17:30 PM] (stevel) HDFS-12344. LocatedFileStatus regression: no 
longer accepting null
[Aug 24, 2017 8:36:49 PM] (junping_du) YARN-6876. Create an abstract log writer 
for extendability. Contributed
[Aug 25, 2017 12:52:41 AM] (yufei) YARN-7049. FSAppAttempt preemption related 
fields have confusing names.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 

Timed out junit tests :

   org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands 
   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [304K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/503/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Jason Lowe]

Andrew Wang wrote:


> This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.


I can see a need for branch-3.0, but please do not create branch-3.  Doing
so will relegate trunk back to the "patch purgatory" branch, a place where
patches won't see a release for years.  Unless something is imminently
going in that will break backwards compatibility and warrant a new 4.x
release, I don't see the need to distinguish trunk from the 3.x line.
Leaving trunk as the 3.x line means less branches to commit patches through
and more testing of every patch since trunk would remain an active area for
testing and releasing.  If we separate trunk and branch-3 then it's almost
certain only-trunk patches will start to accumulate and never get any
"real" testing until someone eventually decides it's time to go to Hadoop
4.x.  Looking back at trunk-as-3.x for an example, patches committed there
in the early days after branch-2 was cut didn't see a release for almost 6
years.

My apologies if I've missed a feature that is just going to miss the 3.0
release and will break compatibility when it goes in.  If so then we need
to cut branch-3, but if not then here's my plea to hold off until we do
need it.

Jason


On Thu, Aug 24, 2017 at 3:33 PM, Andrew Wang 
wrote:

> Glad to see the discussion continued in my absence :)
>
> From a release management perspective, it's *extremely* reasonable to block
> the inclusion of new features a month from the planned release date. A
> typical software development lifecycle includes weeks of feature freeze and
> weeks of code freeze. It is no knock on any developer or any feature to say
> that we should not include something in 3.0.0.
>
> I've been very open and clear about the goals, schedule, and scope of 3.0.0
> over the last year plus. The point of the extended alpha process was to get
> all our features in during alpha, and the alpha merge window has been open
> for a year. I'm unmoved by arguments about how long a feature has been
> worked on. None of these were not part of the original 3.0.0 scope, and our
> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
> HDFS EC that were part of the discussed scope.
>
> I see that two VOTEs have gone out since I was out. I still plan to follow
> the proposal in my original email. This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.
>
> I'm reaching out to the lead contributor of each of these features
> individually to discuss. We need to close on this quickly, and email is too
> low bandwidth at this stage.
>
> Best,
> Andrew
>

[jira] [Created] (MAPREDUCE-6947) Moving logging APIs over to slf4j in hadoop-mapreduce-examples

[JIRA]

Gergely Novák created MAPREDUCE-6947:


 Summary:   Moving logging APIs over to slf4j in 
hadoop-mapreduce-examples
 Key: MAPREDUCE-6947
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6947
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Gergely Novák
Assignee: Gergely Novák






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Rohith Sharma K S]

Hi Andrew

Thanks for update on release plan!

I would like to discuss specifically regarding compatibility of releases.
What is the compatibility to be maintained for GA if we don't merge to
beta1 release? IIUC, till now all the releases were alpha where
compatibility was not that important. All the public interfaces are
subjected to modifications. Once we release beta, compatibility would be a
matter.
During this gap i.e between beta-GA release, should we maintain
compatibility ?
If my understanding is right then TSv2 have to be merged with beta1
release. In TSv2 phase-2, we have compatibility changes from phase-1.


Thanks & Regards
Rohith Sharma K S

On 25 August 2017 at 02:03, Andrew Wang  wrote:

> Glad to see the discussion continued in my absence :)
>
> From a release management perspective, it's *extremely* reasonable to block
> the inclusion of new features a month from the planned release date. A
> typical software development lifecycle includes weeks of feature freeze and
> weeks of code freeze. It is no knock on any developer or any feature to say
> that we should not include something in 3.0.0.
>
> I've been very open and clear about the goals, schedule, and scope of 3.0.0
> over the last year plus. The point of the extended alpha process was to get
> all our features in during alpha, and the alpha merge window has been open
> for a year. I'm unmoved by arguments about how long a feature has been
> worked on. None of these were not part of the original 3.0.0 scope, and our
> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
> HDFS EC that were part of the discussed scope.
>
> I see that two VOTEs have gone out since I was out. I still plan to follow
> the proposal in my original email. This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.
>
> I'm reaching out to the lead contributor of each of these features
> individually to discuss. We need to close on this quickly, and email is too
> low bandwidth at this stage.
>
> Best,
> Andrew
>