2017-09-19 Hadoop 3 release status update

2017-09-19 Thread Andrew Wang 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-09-19

Sorry for the late update. We're down to one blocker and one EC must do!
Made great progress over the last week and a bit.

We will likely cut RC0 this week.

Highlights:

   - Down to just two blocker issues!

Red flags:

   - HDFS unit tests are quite flaky. Some blockers were filed and then
   resolved or downgraded. More work to do here.

Previously tracked beta1 blockers that have been resolved or dropped:

   - HADOOP-14738  (Remove
   S3N and obsolete bits of S3A; rework docs): Committed!
   - HADOOP-14284  (Shade
   Guava everywhere): We resolved this since we decided it was unnecessary for
   beta1.
   - YARN-7162  (Remove
   XML excludes file format): Robert committed after review from Junping.
   - HADOOP-14847  (Remove
   Guava Supplier and change to java Supplier in AMRMClient and
   AMRMClientAysnc): Committed!
   - HADOOP-14238 
(Rechecking
   Guava's object is not exposed to user-facing API): We dropped this off the
   blocker list in the absence of other known issues
   - HADOOP-14835  (mvn
   site build throws SAX errors): I committed after further discussion and
   review with Sean Mackrory and Allen. Planning to switch to japicmp for
   later releases.
   - HDFS-12218  (Rename
   split EC / replicated block metrics in BlockManager): Committed.


beta1 blockers:

   - HADOOP-14771 
(hadoop-client
   does not include hadoop-yarn-client): This was committed but then reverted
   since it broke the build. Haibo and Sean are actively pressing towards a
   correct fix.


beta1 features:

   - Erasure coding
  - Resolved a number of must-dos
 - HDFS-7859 (fsimage changes) was committed!
 - HDFS-12395 (edit log changes) was also committed!
 - HDFS-12218 is discussed above.
  - Remaining blockers:
 - HDFS-12447 is to refactor some of the fsimage code, Andrew needs
 to review
  - Also been progress cleaning up the flaky unit tests, still more to
  do
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
   - Ray has gone through almost all the YARN protos and thinks we're okay
  to move forwards.
  - I think we'll move forward without this committed, given that Sean
  has run it successfully.
   - Classpath isolation (HADOOP-11656)
  - We have just HADOOP-14771 left.
   - Compat guide (HADOOP-13714
   )
  - This was committed! Some follow-on work filed for GA.
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

GA features:

   - Resource profiles (Wangda Tan)
  - Merge vote was sent out. Since branch-3.0 has been cut, this can be
  merged to trunk (3.1.0) and then backported once we've completed testing.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
  run in production at MSFT.
  - If it passes Cloudera internal integration testing, I'm fine
  putting this in for GA.
   - API-based scheduler configuration (Jonathan Hung)
  - Jonathan mentioned that his main goal is to get this in for 2.9.0,
  which seems likely to go out after 3.0.0 GA since there hasn't been any
  serious release planning yet. Jonathan said that delaying this
until 3.1.0
  is fine.
   - YARN native services
  - Still not 100% clear when this will land.

Re: qbt is failiing///RE: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-19 Thread Allen Wittenauer 

> On Sep 19, 2017, at 6:35 AM, Brahma Reddy Battula 
>  wrote:
> 
> qbt is failing from two days with following errors, any idea on this..?

Nothing to be too concerned about.

This is what it looks like when a build server gets bounced or crashed. 
 INFRA team knows our jobs take forever so they rarely wait for them to finish 
if they are doing upgrades.  They’ve been doing that work lately; you can 
follow the action on builds@.





-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

qbt is failiing///RE: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-19 Thread Brahma Reddy Battula 
qbt is failing from two days with following errors, any idea on this..?




cd 
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
/opt/maven/bin/mvn 
-Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-full-1 -Ptest-patch 
-Pparallel-tests -Pshelltest -Pnative -Drequire.snappy -Drequire.openssl 
-Drequire.fuse -Drequire.test.libhadoop -Pyarn-ui clean test -fae > 
/testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 2>&1
FATAL: command execution failed
Command close created at
at hudson.remoting.Command.(Command.java:60)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1123)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1121)
at hudson.remoting.Channel.close(Channel.java:1281)
at hudson.remoting.Channel.close(Channel.java:1263)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1128)
Caused: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1129)
at hudson.remoting.Channel$1.handle(Channel.java:527)
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused: java.io.IOException: Backing channel 'H10' is disconnected.
at 
hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
at 
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy125.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
at hudson.model.Run.execute(Run.java:1735)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Build step 'Execute shell' marked build as failure
ERROR: Step ?Publish Checkstyle analysis results? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Step ?Publish FindBugs analysis results? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Step ?Archive the artifacts? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Step ?Publish JUnit test result report? failed: no workspace for 
hadoop-qbt-trunk-java8-linux-x86 #526
ERROR: Build step failed with exception
java.lang.NullPointerException
at 
hudson.plugins.violations.ViolationsPublisher.perform(ViolationsPublisher.java:74)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:676)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:621)
at hudson.model.Run.execute(Run.java:1760)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Build step 'Report Violations' marked build as failure

--Brahma Reddy Battula

-Original Message-
From: Apache Jenkins Server [mailto:jenk...@builds.apache.org] 
Sent: 19 September 2017 15:07
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/527/

[Sep 18, 2017 10:07:12 AM] (kai.zheng) HDFS-12460. Make addErasureCodingPolicy 
an idempotent operation.
[Sep 18, 2017 3:16:09 PM] (jlowe) YARN-7192. Add a pluggable StateMachine 
Listener that is notified of NM [Sep 18, 2017 4:53:24 PM] (arp) HDFS-12470. 
DiskBalancer: Some tests create plan files under system [Sep 18, 2017 5:32:08 
PM] (rkanter) Revert "YARN-7162. Remove XML excludes file format (rkanter)" - 
wrong [Sep 18, 2017 5:40:06 PM] (rkanter) MAPREDUCE-6954. Disable erasure 
coding for files that are uploaded to [Sep 18, 20

[jira] [Created] (MAPREDUCE-6961) Pull up FileOutputCommitter.getOutputPath to PathOutputCommitter

2017-09-19 Thread Steve Loughran (JIRA) 
Steve Loughran created MAPREDUCE-6961:
-

 Summary: Pull up FileOutputCommitter.getOutputPath to 
PathOutputCommitter
 Key: MAPREDUCE-6961
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6961
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 3.0.0-beta1
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor


SPARK-21549 has shown that downstream code is relying on the internal property 

if we pulled {{FileOutputCommitter.getOutputPath}} to the 
{{PathOutputCommitter}} of MAPREDUCE-6956, then there'd be a public/stable way 
to get this. Admittedly, it does imply that the committer will always have 
*some* output path, but FileOutputFormat depends on that anyway.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-19 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/527/

[Sep 18, 2017 10:07:12 AM] (kai.zheng) HDFS-12460. Make addErasureCodingPolicy 
an idempotent operation.
[Sep 18, 2017 3:16:09 PM] (jlowe) YARN-7192. Add a pluggable StateMachine 
Listener that is notified of NM
[Sep 18, 2017 4:53:24 PM] (arp) HDFS-12470. DiskBalancer: Some tests create 
plan files under system
[Sep 18, 2017 5:32:08 PM] (rkanter) Revert "YARN-7162. Remove XML excludes file 
format (rkanter)" - wrong
[Sep 18, 2017 5:40:06 PM] (rkanter) MAPREDUCE-6954. Disable erasure coding for 
files that are uploaded to
[Sep 18, 2017 6:26:44 PM] (Arun Suresh) YARN-7199. Fix
[Sep 18, 2017 9:04:05 PM] (xgong) YARN-6570. No logs were found for running 
application, running
[Sep 18, 2017 9:25:35 PM] (haibochen) HADOOP-14771. hadoop-client does not 
include hadoop-yarn-client. (Ajay
[Sep 18, 2017 10:04:43 PM] (jlowe) MAPREDUCE-6958. Shuffle audit logger should 
log size of shuffle
[Sep 18, 2017 10:13:42 PM] (wang) HADOOP-14835. mvn site build throws SAX 
errors. Contributed by Andrew
[Sep 18, 2017 10:49:31 PM] (Arun Suresh) YARN-7203. Add container ExecutionType 
into ContainerReport. (Botong
[Sep 19, 2017 2:05:54 AM] (aajisaka) MAPREDUCE-6947. Moving logging APIs over 
to slf4j in


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org