Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2017-03-20 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/

[Mar 19, 2017 6:34:23 PM] (naganarasimha_gr) MAPREDUCE-6865. Fix typo in 
javadoc for DistributedCache. Contributed by




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.tracing.TestTracing 
   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncCheckerTimeout 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-compile-root.txt
  [136K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-compile-root.txt
  [136K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-compile-root.txt
  [136K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [352K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/263/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [44K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org

[jira] [Created] (YARN-6367) YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService

2017-03-20 Thread Xuan Gong (JIRA)
Xuan Gong created YARN-6367:
---

 Summary: YARN logs CLI needs alway check 
containerLogsInfo/containerLogInfo before parse the JSON object from 
NMWebService
 Key: YARN-6367
 URL: https://issues.apache.org/jira/browse/YARN-6367
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Xuan Gong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Created] (YARN-6368) Decommissioning an NM results in a -1 exit code

2017-03-20 Thread Miklos Szegedi (JIRA)
Miklos Szegedi created YARN-6368:


 Summary: Decommissioning an NM results in a -1 exit code
 Key: YARN-6368
 URL: https://issues.apache.org/jira/browse/YARN-6368
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Miklos Szegedi
Assignee: Miklos Szegedi
Priority: Minor


In NodeManager.java we should exit normally in case the RM shuts down the node:
{code}
} finally {
  if (shouldExitOnShutdownEvent
  && !ShutdownHookManager.get().isShutdownInProgress()) {
ExitUtil.terminate(-1);
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Created] (YARN-6366) Refactor the NodeManager DeletionService to support additional DeletionTask types.

2017-03-20 Thread Shane Kumpf (JIRA)
Shane Kumpf created YARN-6366:
-

 Summary: Refactor the NodeManager DeletionService to support 
additional DeletionTask types.
 Key: YARN-6366
 URL: https://issues.apache.org/jira/browse/YARN-6366
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager, yarn
Reporter: Shane Kumpf


The NodeManager's DeletionService only supports file based DeletionTask's. This 
makes sense as files (and directories) have been the primary concern for clean 
up to date. With the addition of the Docker container runtime, addition types 
of DeletionTask are likely to be required, such as deletion of docker container 
and images. See YARN-5366 and YARN-5670. This issue is to refactor the 
DeletionService to support additional DeletionTask's.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-20 Thread Marton Elek
+1 (non-binding)

Tested from the released binary package
* 5 node cluster running from dockerized containers (every 
namenode/datanode/nodemanager, etc. are running in separated containers) 
* Bitcoin bockchain data (~100Gb) parsed and imported to HBase (1.2.4)
* Spark (2.1.0 with included hadoop) job (executing on YARN) to query the data 
from HBase and write the results to HDFS

Looks good.

Marton


> On Mar 19, 2017, at 6:01 PM, Sunil Govind  wrote:
> 
> +1 (non-binding). Thanks Junping for the effort.
> 
> I have used release package and verified below cases
> - Ran MR sleep job and wordcount successfully in where nodes are configured
> with labels.
> - Verified application priority feature and I could see high priority apps
> are getting resource over lower priority apps when configured
> - Verified RM web UI pages and looks fine (priority could be seen)
> - Intra-queue preemption related to app priority also seems fine
> 
> Thanks
> Sunil
> 
> 
> On Fri, Mar 17, 2017 at 2:48 PM Junping Du  wrote:
> 
>> Hi all,
>> With fix of HDFS-11431 get in, I've created a new release candidate
>> (RC3) for Apache Hadoop 2.8.0.
>> 
>> This is the next minor release to follow up 2.7.0 which has been
>> released for more than 1 year. It comprises 2,900+ fixes, improvements, and
>> new features. Most of these commits are released for the first time in
>> branch-2.
>> 
>>  More information about the 2.8.0 release plan can be found here:
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>> 
>>  New RC is available at:
>> http://home.apache.org/~junping_du/hadoop-2.8.0-RC3
>> 
>>  The RC tag in git is: release-2.8.0-RC3, and the latest commit id
>> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>> 
>>  The maven artifacts are available via repository.apache.org at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1057
>> 
>>  Please try the release and vote; the vote will run for the usual 5
>> days, ending on 03/22/2017 PDT time.
>> 
>> Thanks,
>> 
>> Junping
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Created] (YARN-6364) How to set the resource queue when start spark job running on yarn

2017-03-20 Thread sydt (JIRA)
sydt created YARN-6364:
--

 Summary: How to set the resource queue  when start spark job 
running on yarn
 Key: YARN-6364
 URL: https://issues.apache.org/jira/browse/YARN-6364
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: sydt


As we all know, yarn takes charge of resource manage for hadoop. When zeppelin 
start a spark job with yarn-client mode, how to set the designated resource 
queue on yarn ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Resolved] (YARN-6364) How to set the resource queue when start spark job running on yarn

2017-03-20 Thread Jeff Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Zhang resolved YARN-6364.
--
Resolution: Invalid

> How to set the resource queue  when start spark job running on yarn
> ---
>
> Key: YARN-6364
> URL: https://issues.apache.org/jira/browse/YARN-6364
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: sydt
>
> As we all know, yarn takes charge of resource manage for hadoop. When 
> zeppelin start a spark job with yarn-client mode, how to set the designated 
> resource queue on yarn in order to make different spark applications belongs 
> to respective  user running different yarn resource queue?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



[jira] [Created] (YARN-6365) slsrun.sh creating random html directories

2017-03-20 Thread Allen Wittenauer (JIRA)
Allen Wittenauer created YARN-6365:
--

 Summary: slsrun.sh creating random html directories
 Key: YARN-6365
 URL: https://issues.apache.org/jira/browse/YARN-6365
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler-load-simulator
Affects Versions: 3.0.0-alpha3
Reporter: Allen Wittenauer
Priority: Blocker


YARN-6275 causes slsrun.sh to randomly create or override html directories 
wherever it is run.  

{code}
# copy 'html' directory to current directory to make sure web sever can access
cp -r "${bin}/../html" "$(pwd)"
{code}

Instead, the Java could should be changed to take a system property that slsrun 
can populate at run time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-03-20 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/351/

[Mar 19, 2017 6:34:23 PM] (naganarasimha_gr) MAPREDUCE-6865. Fix typo in 
javadoc for DistributedCache. Contributed by


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-20 Thread Junping Du
Thank you for reporting the issue, John! Does this issue only affect ADL (Azure 
Data Lake) which is a new feature for 2.8 rather than other existing FS? If so, 
I think we can leave the fix to 2.8.1 to fix given this is not a regression and 
just a new feature get broken.?


Thanks,


Junping


From: John Zhuge 
Sent: Monday, March 20, 2017 9:07 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No FileSystem 
for scheme: adl".

The issue were caused by backporting HADOOP-13037 to branch-2 and earlier. 
HADOOP-12666 should not be backported, but some changes are needed: property 
fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.

I am working on a patch.


John Zhuge
Software Engineer, Cloudera

On Fri, Mar 17, 2017 at 2:18 AM, Junping Du 
> wrote:
Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping



Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-20 Thread Wangda Tan
Thanks Junping for doing this.

*+1 (Binding),*

Built from source code, deployed a single node cluster. Enabled node labels
and tried to run sample jobs. Haven't seen any issue so far.

Thanks,
Wangda


On Mon, Mar 20, 2017 at 10:02 AM, Junping Du  wrote:

> Thank you for reporting the issue, John! Does this issue only affect ADL
> (Azure Data Lake) which is a new feature for 2.8 rather than other existing
> FS? If so, I think we can leave the fix to 2.8.1 to fix given this is not a
> regression and just a new feature get broken.?
>
>
> Thanks,
>
>
> Junping
>
> 
> From: John Zhuge 
> Sent: Monday, March 20, 2017 9:07 AM
> To: Junping Du
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>
> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
> FileSystem for scheme: adl".
>
> The issue were caused by backporting HADOOP-13037 to branch-2 and earlier.
> HADOOP-12666 should not be backported, but some changes are needed:
> property fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.
>
> I am working on a patch.
>
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Fri, Mar 17, 2017 at 2:18 AM, Junping Du  u...@hortonworks.com>> wrote:
> Hi all,
>  With fix of HDFS-11431 get in, I've created a new release candidate
> (RC3) for Apache Hadoop 2.8.0.
>
>  This is the next minor release to follow up 2.7.0 which has been
> released for more than 1 year. It comprises 2,900+ fixes, improvements, and
> new features. Most of these commits are released for the first time in
> branch-2.
>
>   More information about the 2.8.0 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at: http://home.apache.org/~
> junping_du/hadoop-2.8.0-RC3
>
>   The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>
>   The maven artifacts are available via repository.apache.org repository.apache.org> at: https://repository.apache.org/
> content/repositories/orgapachehadoop-1057
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 03/22/2017 PDT time.
>
> Thanks,
>
> Junping
>
>


Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-20 Thread Junping Du
?Thanks for update, John. Then we should be OK with fixing this issue in 2.8.1.

Mark the target version of HADOOP-14205 to 2.8.1 instead of 2.8.0 and bump up 
to blocker in case we could miss this in releasing 2.8.1. :)


Thanks,


Junping


From: John Zhuge 
Sent: Monday, March 20, 2017 10:31 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Yes, it only affects ADL. There is a workaround of adding these 2 properties to 
core-site.xml:

  
fs.adl.impl
org.apache.hadoop.fs.adl.AdlFileSystem
  

  
fs.AbstractFileSystem.adl.impl
org.apache.hadoop.fs.adl.Adl
  

I have the initial patch ready but hitting these live unit test failures:

Failed tests:
  
TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testListStatus:257
 expected:<1> but was:<10>

Tests in error:
  
TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:254
 » AccessControl
  
TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:190
 » AccessControl


Stay tuned...

John Zhuge
Software Engineer, Cloudera

On Mon, Mar 20, 2017 at 10:02 AM, Junping Du 
> wrote:

Thank you for reporting the issue, John! Does this issue only affect ADL (Azure 
Data Lake) which is a new feature for 2.8 rather than other existing FS? If so, 
I think we can leave the fix to 2.8.1 to fix given this is not a regression and 
just a new feature get broken.?


Thanks,


Junping


From: John Zhuge >
Sent: Monday, March 20, 2017 9:07 AM
To: Junping Du
Cc: common-...@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No FileSystem 
for scheme: adl".

The issue were caused by backporting HADOOP-13037 to branch-2 and earlier. 
HADOOP-12666 should not be backported, but some changes are needed: property 
fs.adl.impl in core-default.xml and hadoop-tools-dist/pom.xml.

I am working on a patch.


John Zhuge
Software Engineer, Cloudera

On Fri, Mar 17, 2017 at 2:18 AM, Junping Du 
> wrote:
Hi all,
 With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

 This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

  More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

  The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping