Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Vrushali C]

Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.

As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .

Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.

Is the latest merge causing any conflicts or issues for s3guard, Steve?

thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:

> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
>> > varun.saxena.apa...@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2...@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application entities (entities that exist
>> >>> outside
>> >>> the scope of an application)
>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>> >>> prefixes,
>> >>> configurable hbase cluster
>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>> >>> - Uses latest stable HBase release 1.2.6
>> >>>
>> >>> There are a total of 82 subtasks that were completed as part of this
>> >>> effort.
>> >>>
>> >>> We paid close attention to ensure that once disabled Timeline Service
>> v.2
>> >>> does not impact existing functionality when disabled (by default).
>> >>>
>> >>> Special thanks to a team of folks who worked hard and contributed
>> towards
>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>> Varun
>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >>>
>> >>> Regards,
>> >>> Vrushali
>> >>>
>> >>> [1] 

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Colin McCabe]

The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.

Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.

The svn workflow evolved because merging feature branches back to trunk
was really painful in svn.  So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.

P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging

P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?

Colin


On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> 
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
> wrote:
> 
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > > varun.saxena.apa...@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2...@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance: 

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Sangjin Lee]

I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E

On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
wrote:

> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > varun.saxena.apa...@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2...@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Steve Loughran]

I'd have assumed it would have gone in as one single patch, rather than a full 
history. I don't see why the trunk needs all the evolutionary history of a 
build.

What should our policy/process be here?

I do currently plan to merge the s3guard in as one single squashed patch; just 
getting HADOOP-14809 sorted first.


> On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> 
> I'm adding my +1 (binding) to conclude the vote.
> 
> With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> the merge to trunk shortly. Thanks everyone!
> 
> Regards
> Vrushali
> 
> 
> On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> varun.saxena.apa...@gmail.com> wrote:
> 
>> +1 (binding).
>> 
>> Kudos to all the team members for their great work!
>> 
>> Being part of the ATSv2 team, I have been involved with either development
>> or review of most of the JIRAs'.
>> Tested ATSv2 in both secure and non-secure mode. Also verified that there
>> is no impact when ATSv2 is turned off.
>> 
>> Regards,
>> Varun Saxena.
>> 
>> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> vrushalic2...@gmail.com> wrote:
>> 
>>> Hi folks,
>>> 
>>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>>> will
>>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> 
>>> We have previously completed one merge onto trunk [3] and Timeline Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> 
>>> Since then, we have been working on extending the capabilities of Timeline
>>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> confident that the state of the feature meets the criteria to be merged
>>> onto trunk and we'd love folks to get their hands on it in a test capacity
>>> and provide valuable feedback so that we can make it production-ready.
>>> 
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>>> usability improvements based on a new architecture. What we would like to
>>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> complete end-to-end read/write flow with security and read level
>>> authorization via whitelists. You should be able to start setting it up
>>> and
>>> testing it.
>>> 
>>> At a high level, the following are the key features that have been
>>> implemented since alpha1:
>>> - Security via Kerberos Authentication and delegation tokens
>>> - Read side simple authorization via whitelist
>>> - Client configurable entity sort ordering
>>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>>> timerange, pagination, sub-app entities
>>> - Support for storing sub-application entities (entities that exist
>>> outside
>>> the scope of an application)
>>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> prefixes,
>>> configurable hbase cluster
>>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> - Uses latest stable HBase release 1.2.6
>>> 
>>> There are a total of 82 subtasks that were completed as part of this
>>> effort.
>>> 
>>> We paid close attention to ensure that once disabled Timeline Service v.2
>>> does not impact existing functionality when disabled (by default).
>>> 
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> 
>>> Regards,
>>> Vrushali
>>> 
>>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> 
>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6949) yarn.app.mapreduce.am.log.level is not documented in mapred-default.xml

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen resolved MAPREDUCE-6949.
---
Resolution: Duplicate

> yarn.app.mapreduce.am.log.level is not documented in mapred-default.xml
> ---
>
> Key: MAPREDUCE-6949
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6949
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.0.0-alpha4
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6949) yarn.app.mapreduce.am.log.level is not documented in mapred-default.xml

[Haibo Chen (JIRA)]

Haibo Chen created MAPREDUCE-6949:
-

 Summary: yarn.app.mapreduce.am.log.level is not documented in 
mapred-default.xml
 Key: MAPREDUCE-6949
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6949
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: documentation
Affects Versions: 3.0.0-alpha4
Reporter: Haibo Chen
Assignee: Haibo Chen
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/508/

[Aug 29, 2017 11:21:14 AM] (sunilg) YARN-6386. Show decommissioning nodes in 
new YARN UI. Contributed by
[Aug 29, 2017 11:36:22 AM] (aajisaka) HADOOP-14671. Upgrade to Apache Yetus 
0.5.0.
[Aug 29, 2017 2:52:24 PM] (jlowe) YARN-5816.
[Aug 29, 2017 6:02:43 PM] (stevel) HADOOP-14802. Add support for using 
container saskeys for all accesses.
[Aug 29, 2017 6:03:44 PM] (stevel) HADOOP-14583. wasb throws an exception if 
you try to create a file and
[Aug 29, 2017 6:27:47 PM] (manojpec) HDFS-11912. Snapshot functionality test 
along with randomized file IO
[Aug 29, 2017 9:53:09 PM] (Carlo Curino) YARN-7010. Federation: routing REST 
invocations transparently to
[Aug 29, 2017 10:16:23 PM] (junping_du) YARN-7037. Optimize data transfer with 
zero-copy approach for
[Aug 29, 2017 10:29:22 PM] (wangda) YARN-5219. When an export var command fails 
in launch_container.sh, the
[Aug 29, 2017 11:33:51 PM] (yufei) YARN-7077. TestAMSimulator and 
TestNMSimulator fail. (Contributed by
[Aug 29, 2017 11:57:03 PM] (yzhang) HDFS-12191. Provide option to not capture 
the accessTime change of a
[Aug 30, 2017 3:35:53 AM] (jianhe) YARN-6756. 
ContainerRequest#executionTypeRequest causes NPE. Contributed
[Aug 30, 2017 4:30:02 AM] (xiao) HDFS-12336. Listing encryption zones still 
fails when deleted EZ is not
[Aug 30, 2017 5:15:11 AM] (rohithsharmaks) YARN-6982. Potential issue on 
setting AMContainerSpec#tokenConf to null
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-5156. YARN_CONTAINER_FINISHED of 
YARN_CONTAINERs will always have
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-5229. Refactor 
#isApplicationEntity and #getApplicationEvent from
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-5638. Introduce a collector 
timestamp to uniquely identify
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-3649. Allow configurable prefix 
for hbase table names like prod,
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5715. Introduce entity prefix for 
return and sort order.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5265. Make HBase configuration for 
the timeline service
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-3359. Recover collector list when 
RM fails over (Li Lu via Varun
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-3359. Addendum for Recover 
collector list when RM fails over (Li Lu
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5792. Adopt the id prefix for 
YARN, MR, and DS entities.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5739. Provide timeline reader API 
to list available timeline entity
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5922. Remove direct references of 
HBaseTimelineWriter/Reader in
[Aug 30, 2017 5:59:51 AM] (varunsaxena) MAPREDUCE-6818. Remove direct reference 
to TimelineClientImpl.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5925. Extract 
hbase-backend-exclusive utility methods from
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5585. [Atsv2] Reader side changes 
for entity prefix and support for
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5980. Update documentation for 
single node hbase deploy.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5378. Accommodate app-id->cluster 
mapping (Sangjin Lee via Varun
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-6064. Support fromId for flowRuns 
and flow/flowRun apps REST API's
[Aug 30, 2017 5:59:52 AM] (varunsaxena) Addendum for YARN-6064. Support fromId 
for flowRuns and flow/flowRun
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6094. Update the coprocessor to be 
a dynamically loaded one.
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-5928. Move ATSv2 HBase backend 
code into a new module that is only
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-4675. Reorganize TimelineClient 
and TimelineClientImpl into
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6069. CORS support in timeline v2 
(Rohith Sharma K S via Varun
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6253. 
FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6027. Support fromid(offset) 
filter for /flows API (Rohith Sharma K
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6256. Add FROM_ID info key for 
timeline entities in reader response
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6237. Move UID constant to 
TimelineReaderUtils (Rohith Sharma K S
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6318. timeline service schema 
creator fails if executed from a
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6146. Add Builder methods for 
TimelineEntityFilters (Haibo Chen via
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6555. Store application flow 
context in NM state store for
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6658. Remove columnFor() methods 
of Columns in HBaseTimeline
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6316 Provide help information and 
documentation for
[Aug 30, 2017 5:59:53 AM] (varunsaxena) YARN-5647. [ATSv2 Security] Collector 
side changes for loading auth

[jira] [Created] (MAPREDUCE-6948) TestJobImpl.testUnusableNodeTransition failed

[Haibo Chen (JIRA)]

Haibo Chen created MAPREDUCE-6948:
-

 Summary: TestJobImpl.testUnusableNodeTransition failed
 Key: MAPREDUCE-6948
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6948
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 3.0.0-alpha4
Reporter: Haibo Chen


*Error Message*
expected: but was:

*Stacktrace*
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.assertJobState(TestJobImpl.java:1041)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testUnusableNodeTransition(TestJobImpl.java:615)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Thanks Brahma for comment on this thread. To be clear, I always update branch 
version just before RC kicking off.

For 2.8.2 release, I don't have plan to involve big top or other third-party 
test tools. As always, we will rely on test/verify efforts from community 
especially from large deployed production cluster - as far as I know,  there 
are already several companies. like: Yahoo!, Alibaba, etc. already deploy 2.8 
release in large production clusters for months which give me more confidence 
on 2.8.2.


Here is more update on 2.8.2 release:

Blocker issues:

-  A new blocker YARN-7076 get reported and fixed by Jian He through last 
weekend.

-  Another new blocker - HADOOP-14814 get identified from my latest jdiff 
run against 2.7.4. The simple fix on an incompatible API change should get 
commit soon.


Critical issues:

-  YARN-7083 already get committed. Thanks Jason for reporting the issue 
and delivering the fix.

-  YARN-6091 get push out from 2.8.2 as issue is not a regression and 
pending for a while.

-  Daryn is actively working on HADOOP-9747 for a while, and the patch are 
getting close to be committed. However, according to Daryn, the patch seems to 
cause some regression in some corner cases in secured environment (Storm auto 
tgt, etc.). May need some additional watch/review on this JIRA's fixes.



My monitoring JACC report between 2.8.2 and 2.7.4 will get finish tomorrow. If 
everything is going smoothly, I am planning to kick off RC0 around holiday 
(this weekend).



Thanks,



​Junping




From: Brahma Reddy Battula 
Sent: Tuesday, August 29, 2017 8:42 AM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du,Please correct me,if I am wrong.


--Brahma Reddy Battula

From: Junping Du 
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du 
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If 

Re: [DISCUSS] Branches and versions for Hadoop 3

[varunsax...@apache.org]

Hi Andrew,

We have completed the merge of TSv2 to trunk.
You can now go ahead with the branching.

Regards,
Varun Saxena.

On Tue, Aug 29, 2017 at 11:35 PM, Andrew Wang 
wrote:

> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
>
> We're still waiting on the native services and S3Guard merges, but I don't
> want to hold branching to the last minute.
>
> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
> wrote:
>
>> Hi Andrew,
>> As Rohith mentioned, if you are good with it, from the TSv2 side, we are
>> ready to go for merge tonight itself (Pacific time)  right after the voting
>> period ends. Varun Saxena has been diligently rebasing up until now so most
>> likely our merge should be reasonably straightforward.
>>
>> @Wangda: your resource profile vote ends tomorrow, could we please
>> coordinate our merges?
>>
>> thanks
>> Vrushali
>>
>>
>> On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>>
>>> On 29 August 2017 at 06:24, Andrew Wang 
>>> wrote:
>>>
>>> > So far I've seen no -1's to the branching proposal, so I plan to
>>> execute
>>> > this tomorrow unless there's further feedback.
>>> >
>>> For on going branch merge threads i.e TSv2, voting will be closing
>>> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
>>> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
>>> couple of more days before creating branch-3.0 so that TSv2 branch merge
>>> would be done directly to trunk?
>>>
>>>
>>>
>>> >
>>> > Regarding the above discussion, I think Jason and I have essentially
>>> the
>>> > same opinion.
>>> >
>>> > I hope that keeping trunk a release branch means a higher bar for
>>> merges
>>> > and code review in general. In the past, I've seen some patches
>>> committed
>>> > to trunk-only as a way of passing responsibility to a future user or
>>> > reviewer. That doesn't help anyone; patches should be committed with
>>> the
>>> > intent of running them in production.
>>> >
>>> > I'd also like to repeat the above thanks to the many, many contributors
>>> > who've helped with release improvements. Allen's work on
>>> create-release and
>>> > automated changes and release notes were essential, as was Xiao's work
>>> on
>>> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
>>> > improvements, which addresses one of the remaining sore spots in the
>>> > release process.
>>> >
>>> > Things have gotten smoother with each alpha we've done over the last
>>> year,
>>> > and it's a testament to everyone's work that we have a good
>>> probability of
>>> > shipping beta and GA later this year.
>>> >
>>> > Cheers,
>>> > Andrew
>>> >
>>> >
>>>
>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Vrushali C]

I'm adding my +1 (binding) to conclude the vote.

With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!

Regards
Vrushali


On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>