Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

[Subramaniam V K]

+1.

Skimmed through the design doc and uber patch and seems to be reasonable.

This is a welcome addition especially w.r.t. cloud deployments so thanks to
everyone who worked on this.

On Mon, Dec 4, 2017 at 8:18 PM, Rohith Sharma K S  wrote:

> +1
>
> On Nov 30, 2017 7:26 AM, "Sunil G"  wrote:
>
> > Hi All,
> >
> >
> > Based on the discussion at [1], I'd like to start a vote to merge feature
> > branch
> >
> > YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
> > 6:00PM PDT.
> >
> >
> > This branch adds support to configure queue capacity as absolute resource
> > in
> >
> > capacity scheduler. This will help admins who want fine control of
> > resources of queues.
> >
> >
> > Feature development is done at YARN-5881 [2], jenkins build is here
> > (YARN-7510 [3]).
> >
> > All required tasks for this feature are committed. This feature changes
> > RM’s Capacity Scheduler only,
> >
> > and we did extensive tests for the feature in the last couple of months
> > including performance tests.
> >
> >
> > Key points:
> >
> > - The feature is turned off by default, and have to configure absolute
> > resource to enable same.
> >
> > - Detailed documentation about how to use this feature is done as part of
> > [4].
> >
> > - No major performance degradation is observed with this branch work. SLS
> > and UT performance
> >
> > tests are done.
> >
> >
> > There were 11 subtasks completed for this feature.
> >
> >
> > Huge thanks to everyone who helped with reviews, commits, guidance, and
> >
> > technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
> > Rohith Sharma K S, Eric Payne .
> >
> >
> > [1] :
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%
> > 3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%
> > 40mail.gmail.com%3E
> >
> > [2] : https://issues.apache.org/jira/browse/YARN-5881
> >
> > [3] : https://issues.apache.org/jira/browse/YARN-7510
> >
> > [4] : https://issues.apache.org/jira/browse/YARN-7533
> >
> >
> > Regards
> >
> > Sunil and Wangda
> >
>

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Subramaniam V K]

Allen, can we bump up the maven surefire heap size to max (if it already is
not) for the branch-2 nightly build and see if it helps?

Thanks,
Subru

On Tue, Oct 24, 2017 at 4:22 PM, Allen Wittenauer 
wrote:

>
> > On Oct 24, 2017, at 4:10 PM, Andrew Wang 
> wrote:
> >
> > FWIW we've been running branch-3.0 unit tests successfully internally,
> though we have separate jobs for Common, HDFS, YARN, and MR. The failures
> here are probably a property of running everything in the same JVM, which
> I've found problematic in the past due to OOMs.
>
> Last time I looked, surefire was configured to launch unit tests
> in different JVMs.  But that might only be true in trunk.  Or maybe only
> for some of the subprojects.
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Subramaniam V K]

Hi Allen,

I had set up the build (or intended to) in anticipation 2.9 release. Thanks
for fixing the configuration!

We did face HDFS tests timeouts in branch-2 when run together but
individually the tests pass:
https://issues.apache.org/jira/browse/HDFS-12620

Folks in HDFS, can you please take a look at HDFS tests in branch-2 as we
are not able to get even a single Yetus run to complete due to multiple
test failures/timeout.

Thanks,
Subru

On Mon, Oct 23, 2017 at 11:26 AM, Vrushali C 
wrote:

> Hi Allen,
>
> I have filed https://issues.apache.org/jira/browse/YARN-7380 for the
> timeline service findbugs warnings.
>
> thanks
> Vrushali
>
>
> On Mon, Oct 23, 2017 at 11:14 AM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > I’m really confused why this causes the Yahoo! QA boxes to go catatonic
> > (!?!) during the run.  As in, never come back online, probably in a
> kernel
> > panic. It’s pretty consistently in hadoop-hdfs, so something is going
> wrong
> > there… is branch-2 hdfs behaving badly?  Someone needs to run the
> > hadoop-hdfs unit tests to see what is going on.
> >
> > It’s probably worth noting that findbugs says there is a problem in the
> > timeline server hbase code.Someone should probably verify + fix that
> > issue.
> >
> >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>

2.9.0 status update (10/6/2017)

[Subramaniam V K]

We have pushed out the release to 10th November 2017:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

The *final *feature freeze date is now 20th October 2017:

   - Merge vote is ongoing for API based (Capacity) Scheduler configuration
   (YARN-5734) and HDFS Router based federation (HDFS-10467
   ). Both of them seem
   on track to pass.
   - Backport of ATS v2 (YARN-2928) and YARN Web UI v2 (YARN-3368) are in
   progress and should be done given the extension.


There are 21 outstanding blockers to be addressed by 27th October 2017:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

Andrew is also tracking most of these as they are common for 3.0.0 also.
We'll follow up additionally on those specifically targeted for 2.9.0.

-Subru/Arun

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Subramaniam V K]

Good to see this merged. I have initiated a separate thread with a smaller
set of stakeholders to discuss inclusion in 2.9. We'll report back to the
2.9 release thread as soon as we reach consensus.

On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash  wrote:

> +1 to maintaining history.
>
> On Wed, Aug 30, 2017 at 11:38 PM, varunsax...@apache.org <
> varun.saxena.apa...@gmail.com> wrote:
>
> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
> > Maintaining history I believe can be useful as it can make reverts
> > easier if at all required.
> > And can be an easy reference point to look at who had contributed what
> > without having to go back to the branch.
> >
> > Regards,
> > Varun Saxena.
> >
> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C 
> > wrote:
> >
> > > Thanks Sangjin for the link to the previous discussions on this! I
> think
> > > that helps answer Steve's questions.
> > >
> > > As decided on that thread [1], YARN-5355 as a feature branch was merged
> > to
> > > trunk via "git merge --no-ff" .
> > >
> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > > chose to develop on a feature branch YARN-5355 so that we could control
> > > when changes went into trunk and didn't inadvertently disrupt trunk.
> > >
> > > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> > >
> > > thanks
> > > Vrushali
> > > [1] https://lists.apache.org/thread.html/
> 43cd65c6b6c3c0e8ac2b3c76afd9ef
> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
> %3E
> > >
> > >
> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:
> > >
> > >> I recall this discussion about a couple of years ago:
> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> > >> dev.hadoop.apache.org%3E
> > >>
> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
> ste...@hortonworks.com
> > >
> > >> wrote:
> > >>
> > >>> I'd have assumed it would have gone in as one single patch, rather
> than
> > >>> a full history. I don't see why the trunk needs all the evolutionary
> > >>> history of a build.
> > >>>
> > >>> What should our policy/process be here?
> > >>>
> > >>> I do currently plan to merge the s3guard in as one single squashed
> > >>> patch; just getting HADOOP-14809 sorted first.
> > >>>
> > >>>
> > >>> > On 30 Aug 2017, at 07:09, Vrushali C 
> > wrote:
> > >>> >
> > >>> > I'm adding my +1 (binding) to conclude the vote.
> > >>> >
> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
> on
> > >>> with
> > >>> > the merge to trunk shortly. Thanks everyone!
> > >>> >
> > >>> > Regards
> > >>> > Vrushali
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > >>> > varun.saxena.apa...@gmail.com> wrote:
> > >>> >
> > >>> >> +1 (binding).
> > >>> >>
> > >>> >> Kudos to all the team members for their great work!
> > >>> >>
> > >>> >> Being part of the ATSv2 team, I have been involved with either
> > >>> development
> > >>> >> or review of most of the JIRAs'.
> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
> that
> > >>> there
> > >>> >> is no impact when ATSv2 is turned off.
> > >>> >>
> > >>> >> Regards,
> > >>> >> Varun Saxena.
> > >>> >>
> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >>> >> vrushalic2...@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi folks,
> > >>> >>>
> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> > merge
> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > >>> vote
> > >>> >>> will
> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>> >>>
> > >>> >>> We have previously completed one merge onto trunk [3] and
> Timeline
> > >>> Service
> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >>>
> > >>> >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
> > reasonably
> > >>> >>> confident that the state of the feature meets the criteria to be
> > >>> merged
> > >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> > >>> capacity
> > >>> >>> and provide valuable feedback so that we can make it
> > >>> production-ready.
> > >>> >>>
> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> > scalability
> > >>> and
> > >>> >>> usability improvements based on a new architecture. What we would
> > >>> like to
> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
> has a
> > >>> >>> complete end-to-end read/write flow with security and read level
> > >>> >>> authorization via whitelists. You should be able to start setting
> > it
> > >>> up
> > >>> >>> and
> > >>> >>> testing it.
> > >>> >>>
> > >>> >>> At a high level, the 

Re: [DISCUSS] Branches and versions for Hadoop 3

[Subramaniam V K]

Andrew,

First up thanks for tirelessly pushing on 3.0 release.

I am confused about your comment on creating 2 branches as my understanding
of Jason's (and Vinod's) comments are that we defer creating branch-3?

IMHO, we should consider creating branch-3 (necessary but not sufficient)
only when we have:

   1. a significant incompatible change.
   2. a new feature that cannot be turned off without affecting core
   components.

In summary, I feel we should follow a lazy rather than eager approach
towards creating mainline branches.

Thanks,
Subru



On Tue, Aug 29, 2017 at 11:45 AM, Wangda Tan  wrote:

> Gotcha, make sense, so I will hold commit until you cut the two branches
> and TSv2 get committed.
>
> Thanks,
> Wangda
>
> On Tue, Aug 29, 2017 at 11:25 AM, Andrew Wang 
> wrote:
>
> > Hi Wangda,
> >
> > I'll cut two branches: branch-3.0 (3.0.0-SNAPSHOT) and branch-3.0.0-beta1
> > (3.0.0-beta1-SNAPSHOT). This way we can merge GA features to branch-3.0
> but
> > not branch-3.0.0-beta1.
> >
> > Best,
> > Andrew
> >
> > On Tue, Aug 29, 2017 at 11:18 AM, Wangda Tan 
> wrote:
> >
> >> Vrushali,
> >>
> >> Sure we can wait TSv2 merged before merge resource profile branch.
> >>
> >> Andrew,
> >>
> >> My understanding is you're going to cut branch-3.0 for 3.0-beta1, and
> the
> >> same branch (branch-3.0) will be used for 3.0-GA as well. So my question
> >> is, there're several features (TSv2, resource profile, YARN-5734) are
> >> targeted to merge to 3.0-GA but not 3.0-beta1, which branch we should
> >> commit to, and when we can commit? Also, similar to 3.0.0-alpha1 to 4,
> you
> >> will cut branch-3.0.0-beta1, correct?
> >>
> >> Thanks,
> >> Wangda
> >>
> >>
> >> On Tue, Aug 29, 2017 at 11:05 AM, Andrew Wang  >
> >> wrote:
> >>
> >>> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
> >>>
> >>> We're still waiting on the native services and S3Guard merges, but I
> >>> don't want to hold branching to the last minute.
> >>>
> >>> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
> >>> wrote:
> >>>
>  Hi Andrew,
>  As Rohith mentioned, if you are good with it, from the TSv2 side, we
>  are ready to go for merge tonight itself (Pacific time)  right after
> the
>  voting period ends. Varun Saxena has been diligently rebasing up
> until now
>  so most likely our merge should be reasonably straightforward.
> 
>  @Wangda: your resource profile vote ends tomorrow, could we please
>  coordinate our merges?
> 
>  thanks
>  Vrushali
> 
> 
>  On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>  rohithsharm...@apache.org> wrote:
> 
> > On 29 August 2017 at 06:24, Andrew Wang 
> > wrote:
> >
> > > So far I've seen no -1's to the branching proposal, so I plan to
> > execute
> > > this tomorrow unless there's further feedback.
> > >
> > For on going branch merge threads i.e TSv2, voting will be closing
> > tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
> > branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait
> > for
> > couple of more days before creating branch-3.0 so that TSv2 branch
> > merge
> > would be done directly to trunk?
> >
> >
> >
> > >
> > > Regarding the above discussion, I think Jason and I have
> essentially
> > the
> > > same opinion.
> > >
> > > I hope that keeping trunk a release branch means a higher bar for
> > merges
> > > and code review in general. In the past, I've seen some patches
> > committed
> > > to trunk-only as a way of passing responsibility to a future user
> or
> > > reviewer. That doesn't help anyone; patches should be committed
> with
> > the
> > > intent of running them in production.
> > >
> > > I'd also like to repeat the above thanks to the many, many
> > contributors
> > > who've helped with release improvements. Allen's work on
> > create-release and
> > > automated changes and release notes were essential, as was Xiao's
> > work on
> > > LICENSE and NOTICE files. I'm also looking forward to Marton's site
> > > improvements, which addresses one of the remaining sore spots in
> the
> > > release process.
> > >
> > > Things have gotten smoother with each alpha we've done over the
> last
> > year,
> > > and it's a testament to everyone's work that we have a good
> > probability of
> > > shipping beta and GA later this year.
> > >
> > > Cheers,
> > > Andrew
> > >
> > >
> >
> 
> 
> >>>
> >>
> >
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Subramaniam V K]

+1 (binding).

I have been following the effort and had few design discussions around the
team especially about how it integrates with Federation. Overall I feel
it's a welcome improvement to YARN.

What are the timelines you are looking for getting this into branch-2?

Thanks,
Subru

On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee  wrote:

> +1 (binding)
>
> I've built the current branch, and checked out a few basic areas including
> documentation. Also perused the most recent changes that went in.
>
> Thanks much for the great team work! I look forward to seeing it in action.
>
> Regards,
> Sangjin
>
> On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen 
> wrote:
>
> > +1 from my side.
> >
> > More from the perspective of ensuring there is no impact of ATSv2 when it
> > is off (by default), I deployed the latest YARN-5355 bits into a few
> > clusters and ran internal Smoke tests. The tests shows no impact when
> ATSv2
> > is off.
> >
> > Best,
> > Haibo
> >
> > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:
> >
> > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> this
> > > happen. Great work, really appreciate the same!!
> > >
> > > +1 (binding) from my side:
> > >
> > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > # Accessed new YARN UI which shows various flows/flow activity etc.
> Seems
> > > fine.
> > > # Based on code, looks like all apis are compatible.
> > > # REST api docs looks fine as well, I guess we could improve that a bit
> > > more post merge as well.
> > > # Adding to additional thoughts which are discussed here, native
> service
> > > also could publish events to atsv2. I think that work is also happened
> in
> > > branch.
> > >
> > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > >
> > > Thanks
> > > Sunil
> > >
> > >
> > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > vrushalic2...@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> vote
> > > will
> > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > >
> > > > We have previously completed one merge onto trunk [3] and Timeline
> > > Service
> > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >
> > > > Since then, we have been working on extending the capabilities of
> > > Timeline
> > > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > > confident that the state of the feature meets the criteria to be
> merged
> > > > onto trunk and we'd love folks to get their hands on it in a test
> > > capacity
> > > > and provide valuable feedback so that we can make it
> production-ready.
> > > >
> > > > In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > > > usability improvements based on a new architecture. What we would
> like
> > to
> > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > complete end-to-end read/write flow with security and read level
> > > > authorization via whitelists. You should be able to start setting it
> up
> > > and
> > > > testing it.
> > > >
> > > > At a high level, the following are the key features that have been
> > > > implemented since alpha1:
> > > > - Security via Kerberos Authentication and delegation tokens
> > > > - Read side simple authorization via whitelist
> > > > - Client configurable entity sort ordering
> > > > - Richer REST APIs for apps, app attempts, containers, fetching
> metrics
> > > by
> > > > timerange, pagination, sub-app entities
> > > > - Support for storing sub-application entities (entities that exist
> > > outside
> > > > the scope of an application)
> > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > prefixes,
> > > > configurable hbase cluster
> > > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > > - Uses latest stable HBase release 1.2.6
> > > >
> > > > There are a total of 82 subtasks that were completed as part of this
> > > > effort.
> > > >
> > > > We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > > > does not impact existing functionality when disabled (by default).
> > > >
> > > > Special thanks to a team of folks who worked hard and contributed
> > towards
> > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >
> > > > Regards,
> > > > Vrushali
> > > >
> > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > 383.html
> > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > [4] 

Re: [Release thread] 2.8.0 release activities

[Subramaniam V K]

Vinod,

Thanks for initiating the 2.8 release thread. We are in late review stages
for YARN-4420 (Add REST API for listing reservations) and YARN-2575 (Adding
ACLs for reservation system), hoping to get them by next week. Any chance
you can put off cutting 2.8 by a week as we are planning to deploy
ReservationSystem and these are critical for that?

Cheers,
Subru

On Thu, Feb 4, 2016 at 3:17 PM, Chris Nauroth 
wrote:

> FYI, I've just needed to raise HDFS-9761 to blocker status for the 2.8.0
> release.
>
> --Chris Nauroth
>
>
>
>
> On 2/3/16, 6:19 PM, "Karthik Kambatla"  wrote:
>
> >Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me.
> >Let us not call it alpha or beta though, it is quite confusing. :)
> >
> >On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma  >
> >wrote:
> >
> >> Thanks Vinod. +1 for 2.8 release start.
> >>
> >> Regards,
> >> Uma
> >>
> >> On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" 
> >>wrote:
> >>
> >> >Seems like all the features listed in the Roadmap wiki are in. I¹m
> >>going
> >> >to try cutting an RC this weekend for a first/non-stable release off of
> >> >branch-2.8.
> >> >
> >> >Let me know if anyone has any objections/concerns.
> >> >
> >> >Thanks
> >> >+Vinod
> >> >
> >> >> On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli
> >> >> wrote:
> >> >>
> >> >> Branch-2.8 is created.
> >> >>
> >> >> As mentioned before, the goal on branch-2.8 is to put improvements /
> >> >>fixes to existing features with a goal of converging on an alpha
> >>release
> >> >>soon.
> >> >>
> >> >> Thanks
> >> >> +Vinod
> >> >>
> >> >>
> >> >>> On Nov 25, 2015, at 5:30 PM, Vinod Kumar Vavilapalli
> >> >>> wrote:
> >> >>>
> >> >>> Forking threads now in order to track all things related to the
> >> >>>release.
> >> >>>
> >> >>> Creating the branch now.
> >> >>>
> >> >>> Thanks
> >> >>> +Vinod
> >> >>>
> >> >>>
> >>  On Nov 25, 2015, at 11:37 AM, Vinod Kumar Vavilapalli
> >>  wrote:
> >> 
> >>  I think we¹ve converged at a high level w.r.t 2.8. And as I just
> >>sent
> >> out an email, I updated the Roadmap wiki reflecting the same:
> >> https://wiki.apache.org/hadoop/Roadmap
> >> 
> >> 
> >>  I plan to create a 2.8 branch EOD today.
> >> 
> >>  The goal for all of us should be to restrict improvements & fixes
> >>to
> >> only (a) the feature-set documented under 2.8 in the RoadMap wiki
> >>and
> >> (b) other minor features that are already in 2.8.
> >> 
> >>  Thanks
> >>  +Vinod
> >> 
> >> 
> >> > On Nov 11, 2015, at 12:13 PM, Vinod Kumar Vavilapalli
> >> >> wrote:
> >> >
> >> > - Cut a branch about two weeks from now
> >> > - Do an RC mid next month (leaving ~4weeks since branch-cut)
> >> > - As with 2.7.x series, the first release will still be called as
> >> >early / alpha release in the interest of
> >> >   ‹ gaining downstream adoption
> >> >   ‹ wider testing,
> >> >   ‹ yet reserving our right to fix any inadvertent
> >>incompatibilities
> >> >introduced.
> >> 
> >> >>>
> >> >>
> >> >
> >>
> >>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.0

[Subramaniam V K]

+1 (non-binding)

- Deployed single node cluster
- Verified reservation system works (YARN-1051) by making reservations and
submitting sample MR jobs against the reservations
- Verified move apps between capacity scheduler queues (YARN-2378)

Thanks,
Subru

On Mon, Nov 17, 2014 at 9:55 AM, Charles Lamb cl...@cloudera.com wrote:

 I built from the src package, started a cluster, created an encryption
 zone and read/wrote data from/to it.

 +1 (non-binding)

 Charles

Re: Thinking ahead to hadoop-2.6

[Subramaniam V K]

+1 for end of next week.

We have got all the patches for YARN-1051 committed to the branch. We are
currently fixing test-patch  plan to call a merge vote soon. Hopefully it
will go through by end of next week.

Thanks,
Subru

On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla ka...@cloudera.com
wrote:

 Looking at the patches, we might be able to get most of YARN-1492 in by the
 end of next week. There would be a couple of security items still
 remaining, but we can may be call the feature alpha-ready without them.

 On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo ctre...@gmail.com wrote:

  I would like to see the shared cache (YARN-1492) make it in as well. We
 are
  going through the final review process (with Karthik and Vinod) and
 should
  be fairly close to complete. This is another feature that has been under
  development for a while.
 
  Thanks,
  Chris
 
  On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas sur...@hortonworks.com
 
  wrote:
 
   I actually would like to see both archival storage and single replica
   memory writes to be in 2.6 release. Archival storage is in the final
  stages
   of getting ready for branch-2 merge as Nicholas has already indicated
 on
   the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of
  these
   features are being in development for sometime.
  
   On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com
 
   wrote:
  
Hey Arun,
   
Maybe we could do a quick run through of the Roadmap wiki and
   add/retarget
things accordingly?
   
I think the KMS and transparent encryption are ready to go. We've
 got a
very few further bug fixes pending, but that's it.
   
Two HDFS things that I think probably won't make the end of the week
  are
archival storage (HDFS-6584) and single replica memory writes
   (HDFS-6581),
which I believe are under the HSM banner. HDFS-6484 was just merged
 to
trunk and I think needs a little more work before it goes into
  branch-2.
HDFS-6581 hasn't even been merged to trunk yet, so seems a bit
 further
   off
yet.
   
Just my 2c as I did not work directly on these features. I just
  generally
shy away from shipping bits quite this fresh.
   
Thanks,
Andrew
   
On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com
   wrote:
   
 Looks like most of the content is in and hadoop-2.6 is shaping up
   nicely.

 I'll create branch-2.6 by end of the week and we can go from there
 to
 stabilize it - hopefully in the next few weeks.

 Thoughts?

 thanks,
 Arun

 On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy 
 a...@hortonworks.com
 wrote:

  Folks,
 
   With hadoop-2.5 nearly done, it's time to start thinking ahead
 to
  hadoop-2.6.
 
   Currently, here is the Roadmap per the wiki:
 
  • HADOOP
  • Credential provider HADOOP-10607
  • HDFS
  • Heterogeneous storage (Phase 2) - Support APIs
  for
 using
  storage tiers by the applications HDFS-5682
  • Memory as storage tier HDFS-5851
  • YARN
  • Dynamic Resource Configuration YARN-291
  • NodeManager Restart YARN-1336
  • ResourceManager HA Phase 2 YARN-556
  • Support for admin-specified labels in YARN
  YARN-796
  • Support for automatic, shared cache for YARN
 application
  artifacts YARN-1492
  • Support NodeGroup layer topology on YARN
 YARN-18
  • Support for Docker containers in YARN YARN-1964
  • YARN service registry YARN-913
 
   My suspicion is, as is normal, some will make the cut and some
   won't.
  Please do add/subtract from the list as appropriate. Ideally, it
   would
be
  good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up
 a
 cadence.
 
   More importantly, as we discussed previously, we'd like
 hadoop-2.6
   to
be
  the *last* Apache Hadoop 2.x release which support JDK6. I'll
  start a
  discussion with other communities (HBase, Pig, Hive, Oozie etc.)
  and
see
  how they feel about this.
 
  thanks,
  Arun
 
 


 --

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
   entity
to
 which it is addressed and may contain information that is
  confidential,
 privileged and exempt from disclosure under applicable law. If the
   reader
 of this message is not the intended recipient, you are hereby
  notified
that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you
 have
 received this