[jira] [Created] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor

2016-10-03 Thread Sidharta Seethana (JIRA) 
Sidharta Seethana created YARN-5704:
---

 Summary: Provide config knobs to control enabling/disabling 
new/work in progress features in container-executor
 Key: YARN-5704
 URL: https://issues.apache.org/jira/browse/YARN-5704
 Project: Hadoop YARN
  Issue Type: Task
  Components: yarn
Affects Versions: 3.0.0-alpha1, 2.7.3, 2.8.0, 3.0.0-alpha2
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana


Provide a mechanism to enable/disable Docker and TC (Traffic Control) 
functionality at the container-executor level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5703) ReservationAgents are not correctly configured

2016-10-03 Thread Sean Po (JIRA) 
Sean Po created YARN-5703:
-

 Summary: ReservationAgents are not correctly configured
 Key: YARN-5703
 URL: https://issues.apache.org/jira/browse/YARN-5703
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler, resourcemanager
Affects Versions: 3.0.0-alpha1
Reporter: Sean Po
Assignee: Sean Po


In AbstractReservationSystem, the method that instantiates a ReservationAgent 
does not properly initialize it with the appropriate configuration because it 
expects the ReservationAgent to implement Configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5702) Refactor TestPBImplRecords so that we can reuse for testing protocol records in other YARN modules

2016-10-03 Thread Subru Krishnan (JIRA) 
Subru Krishnan created YARN-5702:


 Summary: Refactor TestPBImplRecords so that we can reuse for 
testing protocol records in other YARN modules
 Key: YARN-5702
 URL: https://issues.apache.org/jira/browse/YARN-5702
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Subru Krishnan
Assignee: Subru Krishnan


The {{TestPBImplRecords}} has generic helper methods to validate YARN api 
records. This JIRA proposes to refactor the generic helper methods into a base 
class that can then be reused by other YARN modules for testing internal API 
protocol records like in yarn-server-common for Federation (YARN-2915). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2016-10-03 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/

[Oct 3, 2016 5:09:43 AM] (kasha) YARN-5678. Log demand as demand in FSLeafQueue 
and FSParentQueue. (Yufei
[Oct 3, 2016 6:02:26 AM] (naganarasimha_gr) YARN-4855. Should check if node 
exists when replace nodelabels.




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService 
   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.timelineservice.storage.common.TestRowKeys 
   hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters 
   hadoop.yarn.server.timelineservice.storage.common.TestSeparator 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   
hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl
 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-compile-root.txt
  [312K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-compile-root.txt
  [312K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-compile-root.txt
  [312K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [192K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
  [20K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
  [16K]

Re: Maven build: YARN timeline service downloading maven-metadata from personal repository?

2016-10-03 Thread Wangda Tan 
Filed: https://issues.apache.org/jira/browse/HBASE-16749

On Mon, Oct 3, 2016 at 10:18 AM, Wangda Tan  wrote:

> Thanks Sangjin/Ted/Steve for your comments/suggestions, I will file a
> HBase JIRA later.
>
> Regards,
> Wangda
>
> On Mon, Oct 3, 2016 at 4:43 AM, Steve Loughran 
> wrote:
>
>> HBase really ought to have a profile for D/Ling from somewhere like this,
>> and, perhaps, list the ASF snapsphot repo first
>>
>> On 1 Oct 2016, at 17:37, Wangda Tan  wrote:
>> >
>> > Hi YARN-dev,
>> >
>> > (cc common-dev),
>> >
>> > YARN timeline service currently sometimes downloads maven-metadata.xml
>> from
>> > a personal apache site, log looks like:
>> >
>> > [INFO] 
>> > 
>> > [INFO] Building Apache Hadoop YARN Timeline Service
>> 3.0.0-alpha2-SNAPSHOT
>> > [INFO] 
>> > 
>> > Downloading: http://conjars.org/repo/org/apache/hadoop/hadoop-client/3.
>> > 0.0-alpha2-SNAPSHOT/maven-metadata.xml
>> > ...
>> > Downloading: http://people.apache.org/~garyh/mvn/org/apache/hadoop/
>> > hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
>> > ...
>> > Downloading: http://people.apache.org/~garyh/mvn/org/apache/hadoop/
>> > hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
>> >
>> > I noticed this happens for a while, I'm not sure if it causes by my
>> local
>>
>> is it actually D/Lin'g the files? Or is it looking for them and then
>> going on to try the next repo?
>>
>
>

Re: Maven build: YARN timeline service downloading maven-metadata from personal repository?

2016-10-03 Thread Wangda Tan 
Thanks Sangjin/Ted/Steve for your comments/suggestions, I will file a HBase
JIRA later.

Regards,
Wangda

On Mon, Oct 3, 2016 at 4:43 AM, Steve Loughran 
wrote:

> HBase really ought to have a profile for D/Ling from somewhere like this,
> and, perhaps, list the ASF snapsphot repo first
>
> On 1 Oct 2016, at 17:37, Wangda Tan  wrote:
> >
> > Hi YARN-dev,
> >
> > (cc common-dev),
> >
> > YARN timeline service currently sometimes downloads maven-metadata.xml
> from
> > a personal apache site, log looks like:
> >
> > [INFO] 
> > 
> > [INFO] Building Apache Hadoop YARN Timeline Service 3.0.0-alpha2-SNAPSHOT
> > [INFO] 
> > 
> > Downloading: http://conjars.org/repo/org/apache/hadoop/hadoop-client/3.
> > 0.0-alpha2-SNAPSHOT/maven-metadata.xml
> > ...
> > Downloading: http://people.apache.org/~garyh/mvn/org/apache/hadoop/
> > hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> > ...
> > Downloading: http://people.apache.org/~garyh/mvn/org/apache/hadoop/
> > hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> >
> > I noticed this happens for a while, I'm not sure if it causes by my local
>
> is it actually D/Lin'g the files? Or is it looking for them and then going
> on to try the next repo?
>

[jira] [Created] (YARN-5701) Fix issues in yarn native services apps-of-apps

2016-10-03 Thread Billie Rinaldi (JIRA) 
Billie Rinaldi created YARN-5701:


 Summary: Fix issues in yarn native services apps-of-apps
 Key: YARN-5701
 URL: https://issues.apache.org/jira/browse/YARN-5701
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi


In testing out complex apps-of-apps with the docker provider, I found a few 
issues primarily related to exports, as well as a couple of issues in config 
generation and CredentialProvider handling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5700) testAMRestartNotLostContainerCompleteMsg times out intermittently in 2.8

2016-10-03 Thread Eric Badger (JIRA) 
Eric Badger created YARN-5700:
-

 Summary: testAMRestartNotLostContainerCompleteMsg times out 
intermittently in 2.8
 Key: YARN-5700
 URL: https://issues.apache.org/jira/browse/YARN-5700
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Eric Badger
Assignee: Eric Badger


{noformat}
java.lang.Exception: test timed out after 3 milliseconds
at java.lang.Thread.sleep(Native Method)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:301)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:286)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:281)
at 
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart.testAMRestartNotLostContainerCompleteMsg(TestAMRestart.java:774)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2016-10-03 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/

[Oct 3, 2016 5:09:43 AM] (kasha) YARN-5678. Log demand as demand in FSLeafQueue 
and FSParentQueue. (Yufei
[Oct 3, 2016 6:02:26 AM] (naganarasimha_gr) YARN-4855. Should check if node 
exists when replace nodelabels.




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.ha.TestZKFailoverController 
   hadoop.hdfs.TestSafeMode 
   hadoop.hdfs.TestFileCreationDelete 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.mapreduce.v2.app.TestKill 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-compile-javac-root.txt
  [168K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [120K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/183/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Maven build: YARN timeline service downloading maven-metadata from personal repository?

2016-10-03 Thread Steve Loughran 
HBase really ought to have a profile for D/Ling from somewhere like this, and, 
perhaps, list the ASF snapsphot repo first

On 1 Oct 2016, at 17:37, Wangda Tan  wrote:
> 
> Hi YARN-dev,
> 
> (cc common-dev),
> 
> YARN timeline service currently sometimes downloads maven-metadata.xml from
> a personal apache site, log looks like:
> 
> [INFO] 
> 
> [INFO] Building Apache Hadoop YARN Timeline Service 3.0.0-alpha2-SNAPSHOT
> [INFO] 
> 
> Downloading: http://conjars.org/repo/org/apache/hadoop/hadoop-client/3.
> 0.0-alpha2-SNAPSHOT/maven-metadata.xml
> ...
> Downloading: http://people.apache.org/~garyh/mvn/org/apache/hadoop/
> hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> ...
> Downloading: http://people.apache.org/~garyh/mvn/org/apache/hadoop/
> hadoop-client/3.0.0-alpha2-SNAPSHOT/maven-metadata.xml
> 
> I noticed this happens for a while, I'm not sure if it causes by my local

is it actually D/Lin'g the files? Or is it looking for them and then going on 
to try the next repo?

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5699) Retrospect container entity fields which are publishing in events info fields.

2016-10-03 Thread Rohith Sharma K S (JIRA) 
Rohith Sharma K S created YARN-5699:
---

 Summary: Retrospect container entity fields which are publishing 
in events info fields.
 Key: YARN-5699
 URL: https://issues.apache.org/jira/browse/YARN-5699
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S


Currently, all the container information are published at 2 places. Some of 
them are at entity info(top-level) and some are  at event info. 

For containers, some of the event info should be published at container info 
level. For example : container exist status, container state, createdTime, 
finished time. These are general information to container required for 
container-report. So it is better to publish at top level info field. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5698) [YARN-3368] Launch new YARN UI under hadoop web app port

2016-10-03 Thread Sunil G (JIRA) 
Sunil G created YARN-5698:
-

 Summary: [YARN-3368] Launch new YARN UI under hadoop web app port
 Key: YARN-5698
 URL: https://issues.apache.org/jira/browse/YARN-5698
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Sunil G
Assignee: Sunil G


As discussed in YARN-5145, it will be better to launch new web ui as a new 
webapp under same old port.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org