Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-10-17 Thread varunsax...@apache.org 
Hi All,

Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7

We will continue monitoring if everything works well post-merge.

Regards,
Varun Saxena.

On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C  wrote:

> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C 
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K 
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash 
>>> wrote:
>>>
 +1 to maintaining history.

 On Wed, Aug 30, 2017 at 11:38 PM, varunsax...@apache.org <
 varun.saxena.apa...@gmail.com> wrote:

 > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
 > Maintaining history I believe can be useful as it can make reverts
 > easier if at all required.
 > And can be an easy reference point to look at who had contributed what
 > without having to go back to the branch.
 >
 > Regards,
 > Varun Saxena.
 >
 > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C 
 > wrote:
 >
 > > Thanks Sangjin for the link to the previous discussions on this! I
 think
 > > that helps answer Steve's questions.
 > >
 > > As decided on that thread [1], YARN-5355 as a feature branch was
 merged
 > to
 > > trunk via "git merge --no-ff" .
 > >
 > > Although trunk already had TSv2 code (alpha1) prior to this merge,
 we
 > > chose to develop on a feature branch YARN-5355 so that we could
 control
 > > when changes went into trunk and didn't inadvertently disrupt trunk.
 > >
 > > Is the latest merge causing any conflicts or issues for s3guard,
 Steve?
 > >
 > > thanks
 > > Vrushali
 > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
 6afd9ef
 > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
 pache.org%3E
 > >
 > >
 > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee 
 wrote:
 > >
 > >> I recall this discussion about a couple of years ago:
 > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
 > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
 > >> dev.hadoop.apache.org%3E
 > >>
 > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
 ste...@hortonworks.com
 > >
 > >> wrote:
 > >>
 > >>> I'd have assumed it would have gone in as one single patch,
 rather than
 > >>> a full history. I don't see why the trunk needs all the
 evolutionary
 > >>> history of a build.
 > >>>
 > >>> What should our policy/process be here?
 > >>>
 > >>> I do currently plan to merge the s3guard in as one single squashed
 > >>> patch; just getting HADOOP-14809 sorted first.
 > >>>
 > >>>
 > >>> > On 30 Aug 2017, at 07:09, Vrushali C 
 > wrote:
 > >>> >
 > >>> > I'm adding my +1 (binding) to conclude the vote.
 > >>> >
 > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
 get on
 > >>> with
 > >>> > the merge to trunk shortly. Thanks everyone!
 > >>> >
 > >>> > Regards
 > >>> > Vrushali
 > >>> >
 > >>> >
 > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
 > >>> > varun.saxena.apa...@gmail.com> wrote:
 > >>> >
 > >>> >> +1 (binding).
 > >>> >>
 > >>> >> Kudos to all the team members for their great work!
 > >>> >>
 > >>> >> Being part of the ATSv2 team, I have been involved with either
 > >>> development
 > >>> >> or review of most of the JIRAs'.
 > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
 that
 > >>> there
 > >>> >> is no impact when ATSv2 is turned off.
 > >>> >>
 > >>> >>

Re: 2.9.0 status update (10/6/2017)

2017-10-17 Thread varunsax...@apache.org 
Hi All,

Please note that ATSv2 has been merged to branch-2.
Refer to
https://github.com/apache/hadoop/commit/54a694172d7162b2ff45d6e1780c0523cfda66e7

We will continue monitoring if everything works well post-merge.

Regards,
Varun Saxena.

On Sat, Oct 7, 2017 at 5:30 AM, Subramaniam V K  wrote:

> We have pushed out the release to 10th November 2017:
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
>
> The *final *feature freeze date is now 20th October 2017:
>
>- Merge vote is ongoing for API based (Capacity) Scheduler configuration
>(YARN-5734) and HDFS Router based federation (HDFS-10467
>). Both of them seem
>on track to pass.
>- Backport of ATS v2 (YARN-2928) and YARN Web UI v2 (YARN-3368) are in
>progress and should be done given the extension.
>
>
> There are 21 outstanding blockers to be addressed by 27th October 2017:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>
> Andrew is also tracking most of these as they are common for 3.0.0 also.
> We'll follow up additionally on those specifically targeted for 2.9.0.
>
> -Subru/Arun
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-10-17 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/

[Oct 16, 2017 3:01:47 PM] (stevel) HADOOP-14935. Azure: POSIX permissions are 
taking effect in access()
[Oct 16, 2017 4:51:10 PM] (xiao) HDFS-12603. Enable async edit logging by 
default. Contributed by Andrew
[Oct 16, 2017 4:55:22 PM] (xiao) HDFS-12637. Extend 
TestDistributedFileSystemWithECFile with a random EC
[Oct 16, 2017 5:34:06 PM] (xiao) HDFS-12642. Log block and datanode details in 
BlockRecoveryWorker.
[Oct 16, 2017 7:57:48 PM] (weichiu) HDFS-12485. expunge may fail to remove 
trash from encryption zone.
[Oct 16, 2017 8:12:15 PM] (Arun Suresh) YARN-7275. NM Statestore cleanup for 
Container updates. (Kartheek
[Oct 16, 2017 8:57:03 PM] (junping_du) YARN-7124. LogAggregationTFileController 
deletes/renames while file is
[Oct 16, 2017 9:19:31 PM] (xiao) HADOOP-14949. TestKMS#testACLs fails 
intermittently.
[Oct 16, 2017 10:00:38 PM] (nroberts) YARN-7333. container-executor fails to 
remove entries from a directory
[Oct 16, 2017 10:34:32 PM] (rkanter) YARN-7308. TestApplicationACLs fails with 
FairScheduler (rkanter)
[Oct 16, 2017 11:42:59 PM] (weichiu) HADOOP-14948. Document missing config key 
hadoop.treat.subject.external.
[Oct 17, 2017 12:42:41 AM] (manojpec) HDFS-12614. 
FSPermissionChecker#getINodeAttrs() throws NPE when
[Oct 17, 2017 2:44:30 AM] (lei) HDFS-12613. Native EC coder should implement 
release() as idempotent




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.server.router.webapp.TestRouterWebServicesREST 
   hadoop.tools.util.TestProducerConsumer 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-compile-javac-root.txt
  [284K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
  [424K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/560/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6984) MR AM to clean up temporary files from previous attempt

2017-10-17 Thread Gergo Repas (JIRA) 
Gergo Repas created MAPREDUCE-6984:
--

 Summary: MR AM to clean up temporary files from previous attempt
 Key: MAPREDUCE-6984
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6984
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: applicationmaster
Affects Versions: 3.0.0-beta1
Reporter: Gergo Repas


When the MR AM restarts, the {outputDir}/_temporary/{appAttemptNumber} 
directory remains on HDFS, even though this directory is not used during the 
next attempt if the restart has been done without recovery. So if recovery is 
not used for the AM restart, then the deletion of this directory can be done 
earlier (at the start of the next attempt). The benefit is that more free HDFS 
space is available for the next attempt.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6983) Moving logging APIs over to slf4j in hadoop-mapreduce-client-core

2017-10-17 Thread Jinjiang Ling (JIRA) 
Jinjiang Ling created MAPREDUCE-6983:


 Summary: Moving logging APIs over to slf4j in 
hadoop-mapreduce-client-core
 Key: MAPREDUCE-6983
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6983
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Jinjiang Ling
Assignee: Jinjiang Ling






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org