[jira] [Created] (MAPREDUCE-6950) Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx

2017-08-31 Thread zhengchenyu (JIRA) 
zhengchenyu created MAPREDUCE-6950:
--

 Summary: Error Launching job : java.io.IOException: Unknown Job 
job_xxx_xxx
 Key: MAPREDUCE-6950
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6950
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mr-am
Affects Versions: 2.7.1
Reporter: zhengchenyu
 Fix For: 2.7.5


some job report error, like this:
{code}
hadoop.mapreduce.Job.monitorAndPrintJob(Job.java 1367) [main] :  map 100% 
reduce 100%
[2017-08-31T20:27:12.591+08:00] [INFO] 
hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java 277) 
[main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. 
Redirecting to job history server
[2017-08-31T20:27:12.821+08:00] [INFO] 
hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java 277) 
[main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. 
Redirecting to job history server
[2017-08-31T20:27:13.039+08:00] [INFO] 
hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java 277) 
[main] : Application state is completed. FinalApplicationStatus=SUCCEEDED. 
Redirecting to job history server
[2017-08-31T20:27:13.256+08:00] [ERROR] 
hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java 1034) [main] : 
Error Launching job : java.io.IOException: Unknown Job job_xxx_xxx
{code}

I found the am container log, like below. Here we know error happened in 
pipeline, maybe some dn error. And I also found some other reason which close 
the JobHistoryEventHandler. So MR AM can't write the information for JH. So 
client counldn't know whether the appplication is finished. 

{code}
2017-08-31 20:27:10,813 INFO [Thread-1968] 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing 
event MAP_ATTEMPT_STARTED
2017-08-31 20:27:10,814 ERROR [Thread-1968] 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Error writing 
History Event: 
org.apache.hadoop.mapreduce.jobhistory.TaskAttemptStartedEvent@2055ea0a
java.io.EOFException: Premature EOF: no length prefix available
at 
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2292)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1317)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
2017-08-31 20:27:10,814 INFO [Thread-1968] 
org.apache.hadoop.service.AbstractService: Service JobHistoryEventHandler 
failed in state STOPPED; cause: 
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.EOFException: 
Premature EOF: no length prefix available
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.EOFException: 
Premature EOF: no length prefix available
at 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:580)
at 
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:374)
 
at 
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at 
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at 
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at 
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at 
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
{code}

This problem is serious , especially for hive. Job must rerun meaninglessly!  
So I think we need to retry the operation of writing history event. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-31 Thread Subramaniam V K 
Good to see this merged. I have initiated a separate thread with a smaller
set of stakeholders to discuss inclusion in 2.9. We'll report back to the
2.9 release thread as soon as we reach consensus.

On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash  wrote:

> +1 to maintaining history.
>
> On Wed, Aug 30, 2017 at 11:38 PM, varunsax...@apache.org <
> varun.saxena.apa...@gmail.com> wrote:
>
> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
> > Maintaining history I believe can be useful as it can make reverts
> > easier if at all required.
> > And can be an easy reference point to look at who had contributed what
> > without having to go back to the branch.
> >
> > Regards,
> > Varun Saxena.
> >
> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C 
> > wrote:
> >
> > > Thanks Sangjin for the link to the previous discussions on this! I
> think
> > > that helps answer Steve's questions.
> > >
> > > As decided on that thread [1], YARN-5355 as a feature branch was merged
> > to
> > > trunk via "git merge --no-ff" .
> > >
> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > > chose to develop on a feature branch YARN-5355 so that we could control
> > > when changes went into trunk and didn't inadvertently disrupt trunk.
> > >
> > > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> > >
> > > thanks
> > > Vrushali
> > > [1] https://lists.apache.org/thread.html/
> 43cd65c6b6c3c0e8ac2b3c76afd9ef
> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
> %3E
> > >
> > >
> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:
> > >
> > >> I recall this discussion about a couple of years ago:
> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> > >> dev.hadoop.apache.org%3E
> > >>
> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
> ste...@hortonworks.com
> > >
> > >> wrote:
> > >>
> > >>> I'd have assumed it would have gone in as one single patch, rather
> than
> > >>> a full history. I don't see why the trunk needs all the evolutionary
> > >>> history of a build.
> > >>>
> > >>> What should our policy/process be here?
> > >>>
> > >>> I do currently plan to merge the s3guard in as one single squashed
> > >>> patch; just getting HADOOP-14809 sorted first.
> > >>>
> > >>>
> > >>> > On 30 Aug 2017, at 07:09, Vrushali C 
> > wrote:
> > >>> >
> > >>> > I'm adding my +1 (binding) to conclude the vote.
> > >>> >
> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
> on
> > >>> with
> > >>> > the merge to trunk shortly. Thanks everyone!
> > >>> >
> > >>> > Regards
> > >>> > Vrushali
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > >>> > varun.saxena.apa...@gmail.com> wrote:
> > >>> >
> > >>> >> +1 (binding).
> > >>> >>
> > >>> >> Kudos to all the team members for their great work!
> > >>> >>
> > >>> >> Being part of the ATSv2 team, I have been involved with either
> > >>> development
> > >>> >> or review of most of the JIRAs'.
> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
> that
> > >>> there
> > >>> >> is no impact when ATSv2 is turned off.
> > >>> >>
> > >>> >> Regards,
> > >>> >> Varun Saxena.
> > >>> >>
> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >>> >> vrushalic2...@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi folks,
> > >>> >>>
> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> > merge
> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > >>> vote
> > >>> >>> will
> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>> >>>
> > >>> >>> We have previously completed one merge onto trunk [3] and
> Timeline
> > >>> Service
> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >>>
> > >>> >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
> > reasonably
> > >>> >>> confident that the state of the feature meets the criteria to be
> > >>> merged
> > >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> > >>> capacity
> > >>> >>> and provide valuable feedback so that we can make it
> > >>> production-ready.
> > >>> >>>
> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> > scalability
> > >>> and
> > >>> >>> usability improvements based on a new architecture. What we would
> > >>> like to
> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
> has a
> > >>> >>> complete end-to-end read/write flow with security and read level
> > >>> >>> authorization via whitelists. You should be able to start setting
> > it
> > >>> up
> > >>> >>> and
> > >>> >>> testing it.
> > >>> >>>
> > >>> >>> At a high level, the

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-31 Thread Ravi Prakash 
+1 to maintaining history.

On Wed, Aug 30, 2017 at 11:38 PM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
> Maintaining history I believe can be useful as it can make reverts
> easier if at all required.
> And can be an easy reference point to look at who had contributed what
> without having to go back to the branch.
>
> Regards,
> Varun Saxena.
>
> On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C 
> wrote:
>
> > Thanks Sangjin for the link to the previous discussions on this! I think
> > that helps answer Steve's questions.
> >
> > As decided on that thread [1], YARN-5355 as a feature branch was merged
> to
> > trunk via "git merge --no-ff" .
> >
> > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > chose to develop on a feature branch YARN-5355 so that we could control
> > when changes went into trunk and didn't inadvertently disrupt trunk.
> >
> > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> >
> > thanks
> > Vrushali
> > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:
> >
> >> I recall this discussion about a couple of years ago:
> >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> >> dev.hadoop.apache.org%3E
> >>
> >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran  >
> >> wrote:
> >>
> >>> I'd have assumed it would have gone in as one single patch, rather than
> >>> a full history. I don't see why the trunk needs all the evolutionary
> >>> history of a build.
> >>>
> >>> What should our policy/process be here?
> >>>
> >>> I do currently plan to merge the s3guard in as one single squashed
> >>> patch; just getting HADOOP-14809 sorted first.
> >>>
> >>>
> >>> > On 30 Aug 2017, at 07:09, Vrushali C 
> wrote:
> >>> >
> >>> > I'm adding my +1 (binding) to conclude the vote.
> >>> >
> >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
> >>> with
> >>> > the merge to trunk shortly. Thanks everyone!
> >>> >
> >>> > Regards
> >>> > Vrushali
> >>> >
> >>> >
> >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> >>> > varun.saxena.apa...@gmail.com> wrote:
> >>> >
> >>> >> +1 (binding).
> >>> >>
> >>> >> Kudos to all the team members for their great work!
> >>> >>
> >>> >> Being part of the ATSv2 team, I have been involved with either
> >>> development
> >>> >> or review of most of the JIRAs'.
> >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> >>> there
> >>> >> is no impact when ATSv2 is turned off.
> >>> >>
> >>> >> Regards,
> >>> >> Varun Saxena.
> >>> >>
> >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >>> >> vrushalic2...@gmail.com> wrote:
> >>> >>
> >>> >>> Hi folks,
> >>> >>>
> >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> merge
> >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> >>> vote
> >>> >>> will
> >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>> >>>
> >>> >>> We have previously completed one merge onto trunk [3] and Timeline
> >>> Service
> >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >>>
> >>> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> >>> Service v2 in a feature branch [2] for a while, and we are
> reasonably
> >>> >>> confident that the state of the feature meets the criteria to be
> >>> merged
> >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> >>> capacity
> >>> >>> and provide valuable feedback so that we can make it
> >>> production-ready.
> >>> >>>
> >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> scalability
> >>> and
> >>> >>> usability improvements based on a new architecture. What we would
> >>> like to
> >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> >>> complete end-to-end read/write flow with security and read level
> >>> >>> authorization via whitelists. You should be able to start setting
> it
> >>> up
> >>> >>> and
> >>> >>> testing it.
> >>> >>>
> >>> >>> At a high level, the following are the key features that have been
> >>> >>> implemented since alpha1:
> >>> >>> - Security via Kerberos Authentication and delegation tokens
> >>> >>> - Read side simple authorization via whitelist
> >>> >>> - Client configurable entity sort ordering
> >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> >>> metrics by
> >>> >>> timerange, pagination, sub-app entities
> >>> >>> - Support for storing sub-application entities (entities that exist
> >>> >>> outside
> >>> >>> the scope of an application)
> >>> >>> -

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-08-31 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/

[Aug 30, 2017 7:28:56 AM] (rakeshr) HDFS-12258. ec -listPolicies should list 
all policies in system, no
[Aug 30, 2017 2:54:30 PM] (aajisaka) Revert "HADOOP-14671. Upgrade to Apache 
Yetus 0.5.0."
[Aug 30, 2017 4:14:59 PM] (haibochen) YARN-6868. Add test scope to certain 
entries in
[Aug 30, 2017 5:07:48 PM] (haibochen) MAPREDUCE-6892. Issues with the count of 
failed/killed tasks in the
[Aug 30, 2017 5:29:42 PM] (arp) HDFS-12356. Unit test for JournalNode sync 
during Rolling Upgrade.
[Aug 30, 2017 8:30:14 PM] (junping_du) HADOOP-14814. Fix incompatible API 
change on FsServerDefaults to
[Aug 30, 2017 9:07:23 PM] (shv) MAPREDUCE-6931. Remove TestDFSIO "Total 
Throughput" calculation.
[Aug 31, 2017 12:26:13 AM] (templedf) YARN-7115. Move BoundedAppender to 
org.hadoop.yarn.util pacakge
[Aug 31, 2017 1:04:55 AM] (rkanter) YARN-7094. Document the current known issue 
with server-side NM graceful
[Aug 31, 2017 4:47:24 AM] (aw) HADOOP-14670. Increase minimum cmake version for 
all platforms




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestLeaseRecoveryStriped 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncCheckerTimeout 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.hdfs.TestEncryptedTransfer 
   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes 
   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem 
   hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestWriteReadStripedFile 
   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/509/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

Re: [DISCUSS] Merge yarn-native-services branch into trunk

2017-08-31 Thread Jian He 
Update:
I’ve chatted with Andrew offline, we’ll proceed with merging 
yarn-native-services into trunk for beta.
We’ll advertise this feature as “alpha"
Currently, we have completed all the jiras for this merge - I’ve also moved out 
the subtasks that are not blocking this merge.

I’ve created YARN-7127 to run the entire patch against trunk, once that goes 
green, I plan to start a formal vote.

Thanks,
Jian

On Aug 18, 2017, at 2:48 PM, Andrew Wang 
> wrote:

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He 
> wrote:
Thanks Andrew for the comments. Answers below:

- There are no new APIs added in YARN/Hadoop core. In fact, all the new code 
are running outside of existing system and they are optional and require users 
to explicitly opt in. The new system’s own rest API is not stable and will be 
evolving.

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?

- We have been running/testing a version of the entire system internally for 
quite a while.

Do you mind elaborating on the level of testing? Number of nodes, types of 
applications, production or test workload, etc. It'd help us build confidence.

- I’d like to see this in hadoop3-beta1. Of course, we’ll take responsibility 
of moving fast and not block the potential timeline.

Few more questions:

How should we advertise this feature in the release? Since the APIs are 
unstable, I'd propose calling it "alpha" in the release notes, like we do the 
TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge? 
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-31 Thread varunsax...@apache.org 
Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
Maintaining history I believe can be useful as it can make reverts
easier if at all required.
And can be an easy reference point to look at who had contributed what
without having to go back to the branch.

Regards,
Varun Saxena.

On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C  wrote:

> Thanks Sangjin for the link to the previous discussions on this! I think
> that helps answer Steve's questions.
>
> As decided on that thread [1], YARN-5355 as a feature branch was merged to
> trunk via "git merge --no-ff" .
>
> Although trunk already had TSv2 code (alpha1) prior to this merge, we
> chose to develop on a feature branch YARN-5355 so that we could control
> when changes went into trunk and didn't inadvertently disrupt trunk.
>
> Is the latest merge causing any conflicts or issues for s3guard, Steve?
>
> thanks
> Vrushali
> [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:
>
>> I recall this discussion about a couple of years ago:
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> dev.hadoop.apache.org%3E
>>
>> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
>> wrote:
>>
>>> I'd have assumed it would have gone in as one single patch, rather than
>>> a full history. I don't see why the trunk needs all the evolutionary
>>> history of a build.
>>>
>>> What should our policy/process be here?
>>>
>>> I do currently plan to merge the s3guard in as one single squashed
>>> patch; just getting HADOOP-14809 sorted first.
>>>
>>>
>>> > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
>>> >
>>> > I'm adding my +1 (binding) to conclude the vote.
>>> >
>>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>>> with
>>> > the merge to trunk shortly. Thanks everyone!
>>> >
>>> > Regards
>>> > Vrushali
>>> >
>>> >
>>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
>>> > varun.saxena.apa...@gmail.com> wrote:
>>> >
>>> >> +1 (binding).
>>> >>
>>> >> Kudos to all the team members for their great work!
>>> >>
>>> >> Being part of the ATSv2 team, I have been involved with either
>>> development
>>> >> or review of most of the JIRAs'.
>>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>>> there
>>> >> is no impact when ATSv2 is turned off.
>>> >>
>>> >> Regards,
>>> >> Varun Saxena.
>>> >>
>>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> >> vrushalic2...@gmail.com> wrote:
>>> >>
>>> >>> Hi folks,
>>> >>>
>>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>>> vote
>>> >>> will
>>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> >>>
>>> >>> We have previously completed one merge onto trunk [3] and Timeline
>>> Service
>>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >>>
>>> >>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> >>> confident that the state of the feature meets the criteria to be
>>> merged
>>> >>> onto trunk and we'd love folks to get their hands on it in a test
>>> capacity
>>> >>> and provide valuable feedback so that we can make it
>>> production-ready.
>>> >>>
>>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>>> and
>>> >>> usability improvements based on a new architecture. What we would
>>> like to
>>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> >>> complete end-to-end read/write flow with security and read level
>>> >>> authorization via whitelists. You should be able to start setting it
>>> up
>>> >>> and
>>> >>> testing it.
>>> >>>
>>> >>> At a high level, the following are the key features that have been
>>> >>> implemented since alpha1:
>>> >>> - Security via Kerberos Authentication and delegation tokens
>>> >>> - Read side simple authorization via whitelist
>>> >>> - Client configurable entity sort ordering
>>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> metrics by
>>> >>> timerange, pagination, sub-app entities
>>> >>> - Support for storing sub-application entities (entities that exist
>>> >>> outside
>>> >>> the scope of an application)
>>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> >>> prefixes,
>>> >>> configurable hbase cluster
>>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> >>> - Uses latest stable HBase release 1.2.6
>>> >>>
>>> >>> There are a total of 82 subtasks that were completed as part of this
>>> >>> effort.
>>> >>>
>>> >>> We paid close attention to ensure