Re: Apache Hadoop 3.0.3 Release plan

2018-05-08 Thread Vrushali C 
+1 for including the YARN-7190 patch in 3.0.3 release. This is a fix that
will enable HBase to use Hadoop 3.0.x in the production line.

thanks
Vrushali


On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang  wrote:

> Thanks Wei-Chiu and Haibo for the feedback!
>
> Good thing is that I have made the following note couple of days ago when I
> looked the at branch diff, so we are on the same page:
>
>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x gets TSv2
> related hbase jars, not the user classpath. Contributed by Varun Saxena."
>
> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the comment
> below:
> https://issues.apache.org/jira/browse/YARN-7190?focusedCommentId=16457649;
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16457649
>
>
> In addition, I will revert   https://issues.apache.org/
> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>
> Best,
>
> --Yongjun
>
> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen  wrote:
>
> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it is
> > technically incompatible.
> > It is critical enough to justify being an exception, IMO.
> >
> > Added Rohith and Vrushali
> >
> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang 
> > wrote:
> >
> >> Thanks Yongjun for driving 3.0.3 release!
> >>
> >> IMHO, could we consider adding YARN-7190
> >>  into the list?
> >> I understand that it is listed as an incompatible change, however,
> because
> >> of this bug, HBase considers the entire Hadoop 3.0.x line not production
> >> ready. I feel there's not much point releasing any more 3.0.x releases
> if
> >> downstream projects can't pick it up (after the fact that HBase is one
> of
> >> the most important projects around Hadoop).
> >>
> >> On Mon, May 7, 2018 at 1:19 PM, Yongjun Zhang 
> >> wrote:
> >>
> >> > Hi Eric,
> >> >
> >> > Thanks for the feedback, good point. I will try to clean up things,
> then
> >> > cut branch before the release production and vote.
> >> >
> >> > Best,
> >> >
> >> > --Yongjun
> >> >
> >> > On Mon, May 7, 2018 at 8:39 AM, Eric Payne  >> > invalid
> >> > > wrote:
> >> >
> >> > > >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
> >> vote
> >> > > for RC on May 30th
> >> > > I much prefer to wait to cut the branch until just before the
> >> production
> >> > > of the release and the vote. With so many branches, we sometimes
> miss
> >> > > putting critical bug fixes in unreleased branches if the branch is
> cut
> >> > too
> >> > > early.
> >> > >
> >> > > My 2 cents...
> >> > > Thanks,
> >> > > -Eric Payne
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Monday, May 7, 2018, 12:09:00 AM CDT, Yongjun Zhang <
> >> > > yjzhan...@apache.org> wrote:
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Hi All,
> >> > >
> >> > > >
> >> > > We have released Apache Hadoop 3.0.2 in April of this year [1].
> Since
> >> > then,
> >> > > there are quite some commits done to branch-3.0. To further improve
> >> the
> >> > > quality of release, we plan to do 3.0.3 release now. The focus of
> >> 3.0.3
> >> > > will be fixing blockers (3), critical bugs (17) and bug fixes
> (~130),
> >> see
> >> > > [2].
> >> > >
> >> > > Usually no new feature should be included for maintenance releases,
> I
> >> > > noticed we have https://issues.apache.org/jira/browse/HADOOP-13055
> in
> >> > the
> >> > > branch classified as new feature. I will talk with the developers to
> >> see
> >> > if
> >> > > we should include it in 3.0.3.
> >> > >
> >> > > I also noticed that there are more commits in the branch than can be
> >> > found
> >> > > by query [2], also some commits committed to 3.0.3 do not have their
> >> jira
> >> > > target release field filled in accordingly. I will go through them
> to
> >> > > update the jira.
> >> > >
> >> > > >
> >> > > We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
> vote
> >> > for
> >> > > RC on May 30th, targeting for Jun 8th release.
> >> > >
> >> > > >
> >> > > Your insights are welcome.
> >> > >
> >> > > >
> >> > > [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07
> >> 790.html
> >> > >
> >> > > > [2] https://issues.apache.org/jira/issues/?filter=12343874  See
> >> Note
> >> > > below
> >> > > Note: seems I need some admin change so that I can make the filter
> in
> >> [2]
> >> > > public, I'm working on that. For now, you can use jquery
> >> > > (project = hadoop OR project = "Hadoop HDFS" OR project = "Hadoop
> >> YARN"
> >> > OR
> >> > > project = "Hadoop Map/Reduce") AND fixVersion in (3.0.3) ORDER BY
> >> > priority
> >> > > DESC
> >> > >
> >> > > Thanks and best regards,
> >> > >
> >> > > --Yongjun
> >> > >
> >> > > 
> -
> >> > > To unsubscribe, e-mail:

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

2017-11-21 Thread Vrushali C 
Hi Vinod,

bq. (b) We need to figure out if this V1 TimelineService should even be
support given ATSv2.

Yes, I am following this discussion. Let me chat with Rohith and Varun
about this and we will respond on this thread. As such, my preliminary
thoughts are that we should continue to support Timeline Service V1 till we
have the detailed entity level ACLs in V2 and perhaps also a proposal
around upgrade/migration paths from TSv1 to TSv2.

But in any case, we do need to work towards phasing out Timeline Service V1.

thanks
Vrushali


On Tue, Nov 21, 2017 at 1:16 PM, Vinod Kumar Vavilapalli  wrote:

> >> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
> even work. Not just deprecated in favor of timelineserver as was advertised.
> >
> >   This works for me in trunk and the bash code doesn’t appear to
> have changed in a very long time.  Probably something local to your
> install.  (I do notice that the deprecation message says “starting” which
> is awkward when the stop command is given though.)  Also: is the
> deprecation message even true at this point?
>
>
> Sorry, I mischaracterized the problem.
>
> The real issue is that I cannot use this command line when the MapReduce
> JobHistoryServer is already started on the same machine.
>
> ~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
> WARNING: Use of this script to start YARN daemons is deprecated.
> WARNING: Attempting to execute replacement "yarn --daemon start" instead.
> DEPRECATED: Use of this command to start the timeline server is deprecated.
> Instead use the timelineserver command for it.
> Starting the History Server anyway...
> historyserver is running as process 86156.  Stop it first.
>
> So, it looks like in shell-scripts, there can ever be only one daemon of a
> given name, irrespective of which daemon scripts are invoked.
>
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the
> MapReduce JHS - only one of them can be started on the same node.
>  (b) We need to figure out if this V1 TimelineService should even be
> support given ATSv2.
>
> @Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please
> comment on (b).
>
> Thanks
> +Vinod

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2017-10-23 Thread Vrushali C 
Hi Allen,

I have filed https://issues.apache.org/jira/browse/YARN-7380 for the
timeline service findbugs warnings.

thanks
Vrushali


On Mon, Oct 23, 2017 at 11:14 AM, Allen Wittenauer  wrote:

>
> I’m really confused why this causes the Yahoo! QA boxes to go catatonic
> (!?!) during the run.  As in, never come back online, probably in a kernel
> panic. It’s pretty consistently in hadoop-hdfs, so something is going wrong
> there… is branch-2 hdfs behaving badly?  Someone needs to run the
> hadoop-hdfs unit tests to see what is going on.
>
> It’s probably worth noting that findbugs says there is a problem in the
> timeline server hbase code.Someone should probably verify + fix that
> issue.
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-10-16 Thread Vrushali C 
Timeline Service v2 should be landing on branch2 shortly.

thanks
Vrushali

On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vrushalic2...@gmail.com> wrote:

> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <subru...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ravihad...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsax...@apache.org <
>>> varun.saxena.apa...@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vrushalic2...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> ste...@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vrushalic2...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> >

Re: [DISCUSS] Looking to a 2.9.0 release

2017-09-07 Thread Vrushali C 
As I mentioned in the [VOTE] thread at [1],  for Timeline Service v2, we
are thinking about merging to branch2 some time in the next couple of weeks.


So far, we have been maintaining a branch2 based YARN-5355_branch2 along
with our trunk based feature branch YARN-5355. Varun Saxena has been
diligently rebasing it to stay current with branch2.

Currently, we are in the process of testing it just like we did our due
diligence with the trunk based YARN-5355 branch and will ensure the TSv2
branch2 code is a stable state to be merged.

We are also looking into back porting the new yarn-ui to branch2 along with
Sunil Govind. This is work in progress [2] and needs some testing as well.

thanks

Vrushali

[1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27734.html

[2] https://issues.apache.org/jira/browse/YARN-7169

On Thu, Sep 7, 2017 at 3:25 PM, Iñigo Goiri  wrote:

> Hi Subru,
> We are also discussing the merge of HDFS-10467 (Router-based federation)
> and we would like to target 2.9 to do a full release together with YARN
> federation.
> Chris Douglas already arranged the integration into trunk for 3.0.0 GA.
>
> Regarding the points to cover:
> 1. API compatibility: we just extend ClientProtocol so no changes in the
> API.
> 2. Turning feature off: if the Router is not started, the feature is
> disabled completely.
> 3. Stability/testing: the internal version is heavily tested. We will
> start testing the OSS version soon. In any case, the feature is isolated
> and minor bugs will not affect anybody else other than the users of the
> feature.
> 4. Deployment: we are currently using 2.7.1 and we would like to switch to
> 2.9 when available.
> 5. Timeline: finishing the UI and the security JIRAs in HDFS-10467 should
> give us a ready to use version. There will be small features added but
> nothing major. There are a couple minor issues with the merge
> (e.g., HDFS-12384) but should be worked out soon.
>
> Thanks,
> Inigo
>
>
> On Tue, Sep 5, 2017 at 4:26 PM, Jonathan Hung 
> wrote:
>
>> Hi Subru,
>>
>> Thanks for starting the discussion. We are targeting merging YARN-5734
>> (API-based scheduler configuration) to branch-2 before the release of
>> 2.9.0, since the feature is close to complete. Regarding the requirements
>> for merge,
>>
>> 1. API compatibility - this feature adds new APIs, does not modify any
>> existing ones.
>> 2. Turning feature off - using the feature is configurable and is turned
>> off by default.
>> 3. Stability/testing - this is an RM-only change, so we plan on deploying
>> this feature to a test RM and verifying configuration changes for capacity
>> scheduler. (Right now fair scheduler is not supported.)
>> 4. Deployment - we want to get this feature in to 2.9.0 since we want to
>> use this feature and 2.9 version in our next upgrade.
>> 5. Timeline - we have one main blocker which we are planning to resolve by
>> end of week. The rest of the month will be testing then a merge vote on
>> the
>> last week of Sept.
>>
>> Please let me know if you have any concerns. Thanks!
>>
>>
>> Jonathan Hung
>>
>> On Wed, Jul 26, 2017 at 11:23 AM, J. Rottinghuis 
>> wrote:
>>
>> > Thanks Vrushali for being entirely open as to the current status of
>> ATSv2.
>> > I appreciate that we want to ensure things are tested at scale, and as
>> you
>> > said we are working on that right now on our clusters.
>> > We have tested the feature to demonstrate it works at what we consider
>> > moderate scale.
>> >
>> > I think the criteria for including this feature in the 2.9 release
>> should
>> > be if it can be safely turned off and not cause impact to anybody not
>> using
>> > the new feature. The confidence for this is high for timeline service
>> v2.
>> >
>> > Therefore, I think timeline service v2 should definitely be part of 2.9.
>> > That is the big draw for us to work on stabilizing a 2.9 release rather
>> > than just going to 2.8 and back-porting things ourselves.
>> >
>> > Thanks,
>> >
>> > Joep
>> >
>> > On Tue, Jul 25, 2017 at 11:39 PM, Vrushali Channapattan <
>> > vrushalic2...@gmail.com> wrote:
>> >
>> > > Thanks Subru for initiating this discussion.
>> > >
>> > > Wanted to share some thoughts in the context of Timeline Service v2.
>> The
>> > > current status of this module is that we are ramping up for a second
>> > merge
>> > > to trunk. We still have a few merge blocker jiras outstanding, which
>> we
>> > > think we will finish soon.
>> > >
>> > > While we have done some testing, we are yet to test at scale. Given
>> all
>> > > this, we were thinking of initially targeting a beta release vehicle
>> > rather
>> > > than a stable release.
>> > >
>> > > As such, timeline service v2 has branch-2 branch called as
>> > > YARN-5355-branch-2 in case anyone wants to try it out. Timeline
>> service
>> > v2
>> > > can be turned off and should not affect the cluster.
>> > >
>> > > thanks
>> > > Vrushali
>> > >
>> > >
>> > >
>> >

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Vrushali C 
Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.

As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .

Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.

Is the latest merge causing any conflicts or issues for s3guard, Steve?

thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:

> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <ste...@hortonworks.com>
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C <vrushalic2...@gmail.com> wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
>> > varun.saxena.apa...@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2...@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application ent

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Vrushali C 
I'm adding my +1 (binding) to conclude the vote.

With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!

Regards
Vrushali


On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>

Re: Branch merges and 3.0.0-beta1 scope

2017-08-29 Thread Vrushali C 
bq. I don't think it's a lot to ask that feature leads shoot an email to
the release manager of their target version. DISCUSS emails right before a
proposed merge VOTE are way too late, it ends up being a fire drill where
we need to scramble on many fronts.

Yes, I have been thinking about this too. Based on what I have observed,
[DISCUSS] is more of a late-stage communication typically just like 8-10
days before the [VOTE] goes out. Most the leads send out [DISCUSS] emails
when the feature is extremely close to completion. So it seems to me that
there is an inherent hesitation in sending out [DISCUSS] emails when things
are not close to completion, so the Release Manager does not hear about
these early on.

Perhaps we want to add another step like [HEADS-UP] which goes out to at
least the Release Manager and known stakeholders or perhaps even the entire
dev community well before [DISCUSS].

The  [HEADS-UP] could be a very simple note that basically says we are
still working on this but we wish to aim for so-and-so release. It could go
out as soon as release idea is floated but perhaps no later than 7-8 weeks
before planned release date. [HEADS-UP] indicates that despite
contributors' best efforts, there exists a possibility that that feature
may not finish in time to make it into the release but at least the RM is
aware of the intentions and can track these.

Perhaps a time frame like the following may be good to follow:
[HEADS-UP]: typically at least 2 months before release date or as early as
anyone wants to communicate to RM.
[DISCUSS]: approx 4-6 weeks before release date
[VOTE]: closes before 2 (or 3?) weeks before release date

While I do think some timeframes like these should become the norm, we may
not want to be too rigidly pedantic as well. It may be a good idea to keep
ourselves open to making exceptions on a per case basis.

thanks
Vrushali



On Tue, Aug 29, 2017 at 2:25 PM, Andrew Wang 
wrote:

> Hi Vinod,
>
> On Fri, Aug 25, 2017 at 2:42 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> > From a release management perspective, it's *extremely* reasonable to
>> block the inclusion of new features a month from the planned release date.
>> A typical software development lifecycle includes weeks of feature freeze
>> and weeks of code freeze. It is no knock on any developer or any feature to
>> say that we should not include something in 3.0.0.
>>
>>
>> We have never followed the ‘typical' lifecycle that I am guessing you are
>> referring to. If we are, you'll need to publish some of the following: a
>> feature freeze date, blockers-criticals-only-from-now date,
>> testing-finish date, documentation-finish date, final release date and so
>> on.
>>
>
> We discussed this as part of the 3.0 alpha/beta/GA plan. The point of the
> extended alpha/beta process was to release on a schedule. Things that
> weren't ready could be merged for the next alpha. I also advertised alpha4
> as feature complete and beta1 as code complete so we could quickly move on
> to GA.
>
>
>> What we do with Apache releases typically is instead we say ‘this' is
>> roughly when we want to release, and roughly what features must land and
>> let the rest figure out itself.
>>
>> We did this too. We defined the original scope for 3.0.0 GA way back when
> we started the 3.0.0 release process. I've been writing status updates on
> the wiki and tracking targeted features and release blockers throughout.
>
> The target versions of this recent batch of features were not discussed
> with me, the release manager, until just recently. After some discussion, I
> think we've arrived at a release plan that everyone's happy with. But, I
> want to be clear that late-breaking inclusion of additional scope should be
> considered the exception rather than the norm. Merging code so close to
> release means less time for testing and validation, which means lower
> quality releases.
>
> I don't think it's a lot to ask that feature leads shoot an email to the
> release manager of their target version. DISCUSS emails right before a
> proposed merge VOTE are way too late, it ends up being a fire drill where
> we need to scramble on many fronts.
>
>
>> Neither is right or wrong. If we want to change the process, we should
>> communicate as such.
>>
>> Proposing a feature freeze date on the fly is only going to confuse
>> people.
>>
>
>> > I've been very open and clear about the goals, schedule, and scope of
>> 3.0.0 over the last year plus. The point of the extended alpha process was
>> to get all our features in during alpha, and the alpha merge window has
>> been open for a year. I'm unmoved by arguments about how long a feature has
>> been worked on. None of these were not part of the original 3.0.0 scope,
>> and our users have been waiting even longer for big-ticket 3.0 items like
>> JDK8 and HDFS EC that were part of the discussed scope.
>>
>>
>> Except our schedule is so fluid (not due to the release

Re: [DISCUSS] Branches and versions for Hadoop 3

2017-08-29 Thread Vrushali C 
Hi Andrew,
As Rohith mentioned, if you are good with it, from the TSv2 side, we are
ready to go for merge tonight itself (Pacific time)  right after the voting
period ends. Varun Saxena has been diligently rebasing up until now so most
likely our merge should be reasonably straightforward.

@Wangda: your resource profile vote ends tomorrow, could we please
coordinate our merges?

thanks
Vrushali


On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> On 29 August 2017 at 06:24, Andrew Wang  wrote:
>
> > So far I've seen no -1's to the branching proposal, so I plan to execute
> > this tomorrow unless there's further feedback.
> >
> For on going branch merge threads i.e TSv2, voting will be closing
> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
> couple of more days before creating branch-3.0 so that TSv2 branch merge
> would be done directly to trunk?
>
>
>
> >
> > Regarding the above discussion, I think Jason and I have essentially the
> > same opinion.
> >
> > I hope that keeping trunk a release branch means a higher bar for merges
> > and code review in general. In the past, I've seen some patches committed
> > to trunk-only as a way of passing responsibility to a future user or
> > reviewer. That doesn't help anyone; patches should be committed with the
> > intent of running them in production.
> >
> > I'd also like to repeat the above thanks to the many, many contributors
> > who've helped with release improvements. Allen's work on create-release
> and
> > automated changes and release notes were essential, as was Xiao's work on
> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
> > improvements, which addresses one of the remaining sore spots in the
> > release process.
> >
> > Things have gotten smoother with each alpha we've done over the last
> year,
> > and it's a testament to everyone's work that we have a good probability
> of
> > shipping beta and GA later this year.
> >
> > Cheers,
> > Andrew
> >
> >
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-25 Thread Vrushali C 
Hi Subru,

Thanks for your vote and your response!

Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.

thanks
Vrushali







On Fri, Aug 25, 2017 at 2:28 PM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K 
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee  wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen 
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G  wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2...@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > >

[jira] [Created] (MAPREDUCE-5550) Job Status message not shown in UI with Hadoop 2.0

2013-09-30 Thread Vrushali C (JIRA) 
Vrushali C created MAPREDUCE-5550:
-

 Summary: Job Status message not shown in UI with Hadoop 2.0
 Key: MAPREDUCE-5550
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5550
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.5-alpha
Reporter: Vrushali C





Hadoop 1.0 JobTracker UI displays Job Status message when list of mapper or 
reduce tasks are listed. This give an idea of how that task is making progress.

Hadoop 2.0 ResourceManager UI does not have this. It would be good to have this 
on RM UI.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (MAPREDUCE-5502) History link in resource manager is broken for KILLED jobs

2013-09-10 Thread Vrushali C (JIRA) 
Vrushali C created MAPREDUCE-5502:
-

 Summary: History link in resource manager is broken for KILLED jobs
 Key: MAPREDUCE-5502
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5502
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.0.5-alpha
Reporter: Vrushali C



History link in resource manager is broken for KILLED jobs.

Seems to happen with jobs with State 'KILLED' and FinalStatus 'KILLED'. If the 
State is 'FINISHED' and FinalStatus is 'KILLED', then the History link is 
fine.

It isn't easy to reproduce the problem since the time at which the app is 
killed determines the state it ends up in, which is hard to guess. these 
particular jobs seem to get a Diagnostics message of Application killed by 
user. where as the other killed jobs get  Kill Job received from client 
job_1378766187901_0002
Job received Kill while in RUNNING state. 


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5432) JobHistoryParser does not fetch clockSplits, cpuUsages, vMemKbytes, physMemKbytes from history file

2013-07-29 Thread Vrushali C (JIRA) 
Vrushali C created MAPREDUCE-5432:
-

 Summary: JobHistoryParser does not fetch clockSplits, cpuUsages, 
vMemKbytes, physMemKbytes from history file
 Key: MAPREDUCE-5432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.0.4-alpha
Reporter: Vrushali C



JobHistoryParser's handleMapAttemptFinishedEvent() function does not look at 
MapAttemptFinishedEvent's   
  int[] clockSplits;
  int[] cpuUsages;
  int[] vMemKbytes;
  int[] physMemKbytes;

JobHistoryParser's inner class TaskAttemptInfo also needs to be enhanced to 
have these as members so that handleMapAttemptFinishedEvent() can get them and 
store them.




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5309) 2.0.4 JobHistoryParser can't parse certain failed job history files generated by 2.0.3 history server

2013-06-07 Thread Vrushali C (JIRA) 
Vrushali C created MAPREDUCE-5309:
-

 Summary: 2.0.4 JobHistoryParser can't parse certain failed job 
history files generated by 2.0.3 history server
 Key: MAPREDUCE-5309
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5309
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.0.4-alpha
Reporter: Vrushali C



When the 2.0.4 JobHistoryParser tries to parse a job history file generated by 
hadoop 2.0.3, the jobhistoryparser throws as an error as

java.lang.ClassCastException: org.apache.avro.generic.GenericData$Array cannot 
be cast to org.apache.hadoop.mapreduce.jobhistory.JhCounters
at 
org.apache.hadoop.mapreduce.jobhistory.TaskAttemptUnsuccessfulCompletion.put(TaskAttemptUnsuccessfulCompletion.java:58)
at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at 
org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:93)
at 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:111)
at 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:156)
at 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:142)
at 
com.twitter.somepackage.Test20JobHistoryParsing.testFileAvro(Test20JobHistoryParsing.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at 
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at 
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)


Test code and the job history file are attached.

Test code:
package com.twitter.somepackagel;

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser;
import org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.JobInfo;
import org.junit.Test;
import org.apache.hadoop.yarn.YarnException;

public class Test20JobHistoryParsing {
   
  @Test
  public void testFileAvro() throws IOException
  {
  Path local_path2 = new Path(/tmp/job_2_0_3-KILLED.jhist);
 JobHistoryParser parser2 = new JobHistoryParser(FileSystem.getLocal(new 
Configuration()), local_path2);
 try {
   JobInfo ji2 = parser2.parse();
   System.out.println( job info:  + ji2.getJobname() +  
 + ji2.getFinishedMaps() +  
 + ji2.getTotalMaps() +  
 + ji2.getJobId() ) ;
 }
 catch (IOException e) {
throw new YarnException(Could not load history file 
   + local_path2.getName(), e);
 }
  }
}

This seems to stem from the fix in 
https://issues.apache.org/jira/browse