[jira] [Created] (YARN-5795) FairScheduler set AppMaster vcores didn't work

2016-10-27 Thread tangshangwen (JIRA) 
tangshangwen created YARN-5795:
--

 Summary: FairScheduler set AppMaster vcores didn't work
 Key: YARN-5795
 URL: https://issues.apache.org/jira/browse/YARN-5795
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.7.1
Reporter: tangshangwen
Assignee: tangshangwen


In our cluster, we use the linux container, I would like to increase the number 
of cpu to get more CPU time slice, but it did not take effect， i set 
yarn.app.mapreduce.am.resource.cpu-vcores = 2 ,but i found the resourcemanager 
log
{noformat}
[2016-10-27T16:36:37.280 08:00] [INFO] 
resourcemanager.scheduler.SchedulerNode.allocateContainer(SchedulerNode.java 
153) [ResourceManager Event Processor] : Assigned container 
container_1477059529836_336635_01_01 of capacity 
{noformat}

Because scheduler.getResourceCalculator() only computes memory
{code:title=RMAppManager.java|borderStyle=solid}
 private ResourceRequest validateAndCreateResourceRequest(
  ApplicationSubmissionContext submissionContext, boolean isRecovery)
  throws InvalidResourceRequestException {
...
  SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
  scheduler.getClusterResource(),
  scheduler.getMinimumResourceCapability(),
  scheduler.getMaximumResourceCapability(),
  scheduler.getMinimumResourceCapability());
...
}
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5794) Adding Apache License Headers to fix the asflicense warnings

2016-10-27 Thread Yiqun Lin (JIRA) 
Yiqun Lin created YARN-5794:
---

 Summary: Adding Apache License Headers to fix the asflicense 
warnings
 Key: YARN-5794
 URL: https://issues.apache.org/jira/browse/YARN-5794
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yiqun Lin
Assignee: Yiqun Lin


There are some asflicense warnings showed when building the jenkins of 
hadoop-common. The warning infos:
{code}
Lines that start with ? in the ASF License  report indicate files that do 
not have an Apache license header:
 !? 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js
 !? 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/js/jquery.dataTables.min.js
 !? 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery/jquery-ui-1.9.1.custom.min.js
 !? 
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery/jquery-1.8.2.min.js
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5790) Address collector life-cycle/persistency issues on the NM side.

2016-10-27 Thread Vrushali C (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C resolved YARN-5790.
--
Resolution: Duplicate


Thanks [~varun_saxena], marking this closed as a duplicate of YARN-3115

> Address collector life-cycle/persistency issues on the NM side.
> ---
>
> Key: YARN-5790
> URL: https://issues.apache.org/jira/browse/YARN-5790
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Vrushali C
>
> As per discussions in YARN-3359 with [~gtCarrera9], filing jira to think over 
> and address collector life-cycle/persistency issues on the NM side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5793) Trim configuration values in DockerLinuxContainerRuntime

2016-10-27 Thread Tianyin Xu (JIRA) 
Tianyin Xu created YARN-5793:


 Summary: Trim configuration values in DockerLinuxContainerRuntime
 Key: YARN-5793
 URL: https://issues.apache.org/jira/browse/YARN-5793
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.0.0-alpha1
Reporter: Tianyin Xu
Priority: Minor


The current implementation of {{DockerLinuxContainerRuntime}} does not follow 
the practice of trimming configuration values. This leads to errors if users 
set values containing space or newline.

see the following YARN commits as reference:
YARN-3395. FairScheduler: Trim whitespaces when using username for queuename.
YARN-2869. CapacityScheduler should trim sub queue names when parse 
configuration.
YARN-2843. Fixed NodeLabelsManager to trim inputs for hosts and labels so as to 
make them work correctly.

and many other Hadoop/HDFS commits (just list a few):
HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames
HDFS-2799. Trim fs.checkpoint.dir values.
HADOOP-6578. Configuration should trim whitespace around a lot of value types
HADOOP-6534. Trim whitespace from directory lists initializing

Patch is available against trunk
{code:title=DockerLinuxContainerRuntime.java|borderStyle=solid}
@@ -219,9 +219,9 @@ public void initialize(Configuration conf)
 dockerClient = new DockerClient(conf);
 allowedNetworks.clear();
 allowedNetworks.addAll(Arrays.asList(
-conf.getStrings(YarnConfiguration.NM_DOCKER_ALLOWED_CONTAINER_NETWORKS,
+
conf.getTrimmedStrings(YarnConfiguration.NM_DOCKER_ALLOWED_CONTAINER_NETWORKS,
 YarnConfiguration.DEFAULT_NM_DOCKER_ALLOWED_CONTAINER_NETWORKS)));
-defaultNetwork = conf.get(
+defaultNetwork = conf.getTrimmed(
 YarnConfiguration.NM_DOCKER_DEFAULT_CONTAINER_NETWORK,
 YarnConfiguration.DEFAULT_NM_DOCKER_DEFAULT_CONTAINER_NETWORK);
 
@@ -237,7 +237,7 @@ public void initialize(Configuration conf)
   throw new ContainerExecutionException(message);
 }
 
-privilegedContainersAcl = new AccessControlList(conf.get(
+privilegedContainersAcl = new AccessControlList(conf.getTrimmed(
 YarnConfiguration.NM_DOCKER_PRIVILEGED_CONTAINERS_ACL,
 YarnConfiguration.DEFAULT_NM_DOCKER_PRIVILEGED_CONTAINERS_ACL));
   }
@@ -439,7 +439,7 @@ public void launchContainer(ContainerRuntimeContext ctx)
 LOCALIZED_RESOURCES);
 @SuppressWarnings("unchecked")
 List userLocalDirs = ctx.getExecutionAttribute(USER_LOCAL_DIRS);
-Set capabilities = new HashSet<>(Arrays.asList(conf.getStrings(
+Set capabilities = new 
HashSet<>(Arrays.asList(conf.getTrimmedStrings(
 YarnConfiguration.NM_DOCKER_CONTAINER_CAPABILITIES,
 YarnConfiguration.DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES)));
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5792) adopt the id prefix for YARN, MR, and DS entities

2016-10-27 Thread Sangjin Lee (JIRA) 
Sangjin Lee created YARN-5792:
-

 Summary: adopt the id prefix for YARN, MR, and DS entities
 Key: YARN-5792
 URL: https://issues.apache.org/jira/browse/YARN-5792
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-5355
Reporter: Sangjin Lee


We introduced the entity id prefix to support flexible entity sorting 
(YARN-5715). We should adopt the id prefix for YARN entities, MR entities, and 
DS entities to take advantage of the id prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] what exactly are the stability guarantees of the YARN APIs

2016-10-27 Thread Sangjin Lee 
Resurrecting an old thread as part of the YARN bug bash (YARN-5130).

At minimum, I believe we agree to the following (do let me know if that is
not the case):
(1) If the class is declared Public/Stable, no changes to the class that
breaks the Stable contract should be made between non-major releases
**regardless
of the method visibility/stability**. For example, the following would
break the stability:
- adding a new abstract method, whether that method is stable, evolving, or
even private
- renaming a public method
Although it may be possible to have methods with weaker
stability/visibility, they still MUST not break the class contract.

(2) We need to address the existing violations to ContainerStatus and
NodeReport by adding a default implementation for **minor releases**.
- ContainerStatus: YARN-3866 (2.8)
- NodeReport: YARN-4293 (2.8)

There are subsequent changes to ContainerStatus by YARN-2882 and YARN-5430,
but they are marked 2.9.0. Is there going to be 2.9.0? If not, then these
might not matter as 3.0.0 permits backward incompatible changes.

Thoughts?

On Tue, May 31, 2016 at 10:48 AM, Sangjin Lee  wrote:

>
>
> On Tue, May 31, 2016 at 10:10 AM, Chris Nauroth 
> wrote:
>
>> I recommend that we update the compatibility guide with some text that
>> explicitly addresses subclassing/interface inheritance stability for
>> classes/interfaces annotated Stable.  This is for our own benefit too.  (I
>> often refer back to that doc when I'm coding a patch that might have a
>> chance of being backwards-incompatible.)
>>
>
> I agree that making this distinction helps not only users but also the
> hadoop contributors. In addition to updating the compatibility guide, how
> about adding a new audience annotation for interfaces & abstract classes
> that spells out whether a 3rd-party is expected to extend/implement it?
>
> For example, some interface can be Public/Stable for use but could be
> off-limits in terms of extending/implementing it, while another can be
> Public/Stable for use and allowed to be extended but with an Evolving
> stability. It requires a little design, but should helps us a great deal on
> both ends. My 2 cents.
>
> Sangjin
>
>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 5/31/16, 9:46 AM, "Karthik Kambatla"  wrote:
>>
>> >Argh! Totally my bad on YARN-2882. Kept missing the changes to
>> >ContainerStatus even after you pointed out.
>> >
>> >Filed YARN-5184 to fix this before we release it. Thanks for pointing it
>> >out, Steve.
>> >
>> >On Tue, May 31, 2016 at 6:00 AM, Steve Loughran 
>> >wrote:
>> >
>> >>
>> >> On 31 May 2016, at 05:44, Karthik Kambatla  >> ka...@cloudera.com>> wrote:
>> >>
>> >> Inline.
>> >>
>> >> On Sat, May 28, 2016 at 11:34 AM, Sangjin Lee > > >> sjl...@gmail.com>> wrote:
>> >> I think there is more to it. The InterfaceStability javadoc states:
>> >> Incompatible changes must not be made to classes marked as stable.
>> >>
>> >> And in practice, I don't think the class annotation can be considered a
>> >> simple sum of method annotations. There is a notion of class
>> >>compatibility
>> >> distinct from method stability. One key example is interfaces and
>> >>abstract
>> >> classes as in this case. The moment a new abstract method is added, the
>> >> class becomes incompatible as it would break all downstream subclasses
>> >>or
>> >> implementing classes. That's the case even if *all methods are declared
>> >> stable*. Thus, adding any abstract method (no matter what their
>> >> scope/stability is) should be considered in violation of the stable
>> >> contract of the class.
>> >>
>> >> Fair point. I was referring to them in the context of adding @Evolving
>> >> methods to @Stable classes. Our policy states that "Classes not
>> >>annotated
>> >> are implicitly ³Private². Class members not annotated inherit the
>> >> annotations of the enclosing class." So, the annotation on a method
>> >> overrides that of the enclosing class. This seems pretty reasonable to
>> >>me.
>> >>
>> >>
>> >>
>> >> My code wouldn't even compile because new abstract methods were added
>> >>to a
>> >> class tagged as stable.
>> >>
>> >> As far as I'm concerned, it doesn't meat the strict semantics of
>> >>"stable",
>> >> unless there is some nuance I'm missing.
>> >>
>> >> Therefore, I'm with Sangin: adding new abstract methods to an existing
>> >> @Stable class breaks compatibility. Adding new non-abstract methods
>> >>‹fine.
>> >> It would have been straightforward to add some new methods to, say
>> >> ContainerReport, which were no-ops/exception raising, but which at
>> least
>> >> didn't break compilation. (though they may have broken codepaths which
>> >> required the methods to act as getters/settes)
>> >>
>> >> Do you think there is reason to revisit this? If yes, we should update
>> >> this for Hadoop 3.
>> >>
>> >> I'm not sure about revisiting. I'm raising

[jira] [Resolved] (YARN-3755) Log the command of launching containers

2016-10-27 Thread Vinod Kumar Vavilapalli (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-3755.
---
Resolution: Duplicate

Actually I think this is a dup of YARN-4309 which now dumps a lot more useful 
information in addition to command line, env etc.

Closing this as duplicate for now, please reopen if you disagree.

> Log the command of launching containers
> ---
>
> Key: YARN-3755
> URL: https://issues.apache.org/jira/browse/YARN-3755
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>  Labels: oct16-easy
> Attachments: YARN-3755-1.patch, YARN-3755-2.patch
>
>
> In the resource manager log, yarn would log the command for launching AM, 
> this is very useful. But there's no such log in the NN log for launching 
> containers. It would be difficult to diagnose when containers fails to launch 
> due to some issue in the commands. Although user can look at the commands in 
> the container launch script file, this is an internal things of yarn, usually 
> user don't know that. In user's perspective, they only know what commands 
> they specify when building yarn application. 
> {code}
> 2015-06-01 16:06:42,245 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command 
> to launch container container_1433145984561_0001_01_01 : 
> $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true 
> -Dhadoop.metrics.log.level=WARN  -Xmx1024m  
> -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator 
> -Dlog4j.configuration=tez-container-log4j.properties 
> -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA 
> -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster 
> 1>/stdout 2>/stderr
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5791) [YARN Native Service] Build application specific UI on top of data posted to timeline service V.2

2016-10-27 Thread Gour Saha (JIRA) 
Gour Saha created YARN-5791:
---

 Summary: [YARN Native Service] Build application specific UI on 
top of data posted to timeline service V.2
 Key: YARN-5791
 URL: https://issues.apache.org/jira/browse/YARN-5791
 Project: Hadoop YARN
  Issue Type: New Feature
Reporter: Gour Saha


As per YARN-5780 we will start getting application-specific data in timeline 
service v2 for all YARN native services. Exposing a UI for these 
application-specific data would be very beneficial to application owners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5790) Address collector life-cycle/persistency issues on the NM side.

2016-10-27 Thread Vrushali C (JIRA) 
Vrushali C created YARN-5790:


 Summary: Address collector life-cycle/persistency issues on the NM 
side.
 Key: YARN-5790
 URL: https://issues.apache.org/jira/browse/YARN-5790
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Vrushali C



As per discussions in YARN-3359 with [~gtCarrera9], filing jira to think over 
and address collector life-cycle/persistency issues on the NM side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5789) Add a web-based visualizer for scheduled applications

2016-10-27 Thread Xiaolin Zang (JIRA) 
Xiaolin Zang created YARN-5789:
--

 Summary: Add a web-based visualizer for scheduled applications
 Key: YARN-5789
 URL: https://issues.apache.org/jira/browse/YARN-5789
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: resourcemanager, webapp
Reporter: Xiaolin Zang


During a research project, we at Parallel Data Lab of Carnegie Mellon 
University had made a web-based visualizer for Yarn's scheduled applications.  
The visualizer is scheduler-independent, can be accessed from Yarn's website, 
and provides both node-based and rack-based views of the applications.  Now 
we'd like to contribute the visualizer, which is called Atlas, to Hadoop.

Near the end of http://www.pdl.cmu.edu/TetriScope/index.shtml, 3 captured 
screencasts demonstrate the feaures of the visualizer (the UI has been further 
refined since).  Please note that we only propose to integrate Atlas, the 
visualization component of the project, into Hadoop.  We do not intend to 
contribute TetriSched, a novel scheduler developed at CMU to Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2016-10-27 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/137/

[Oct 26, 2016 1:07:53 PM] (kihwal) HDFS-11050. Change log level to 'warn' when 
ssl initialization fails and
[Oct 26, 2016 2:16:13 PM] (kihwal) HDFS-11053. Unnecessary superuser check in 
versionRequest(). Contributed
[Oct 26, 2016 3:27:26 PM] (cnauroth) HADOOP-13614. Purge some 
superfluous/obsolete S3 FS tests that are
[Oct 26, 2016 3:55:42 PM] (cnauroth) HADOOP-13502. Split 
fs.contract.is-blobstore flag into more descriptive
[Oct 26, 2016 5:32:35 PM] (lei) HDFS-10638. Modifications to remove the 
assumption that StorageLocation
[Oct 26, 2016 6:31:00 PM] (sjlee) YARN-5433. Audit dependencies for Category-X. 
Contributed by Sangjin
[Oct 26, 2016 9:08:54 PM] (weichiu) Addendum patch for HADOOP-13514 Upgrade 
maven surefire plugin to 2.19.1.
[Oct 26, 2016 9:11:38 PM] (liuml07) HDFS-10921. TestDiskspaceQuotaUpdate 
doesn't wait for NN to get out of
[Oct 26, 2016 9:25:03 PM] (wang) HADOOP-8299. ViewFileSystem link slash mount 
point crashes with
[Oct 27, 2016 2:39:02 AM] (aengineer) HDFS-11038. DiskBalancer: support running 
multiple commands in single
[Oct 27, 2016 6:04:07 AM] (rohithsharmaks) YARN-4555. 
TestDefaultContainerExecutor#testContainerLaunchError fails
[Oct 27, 2016 6:27:17 AM] (rohithsharmaks) YARN-4363. In TestFairScheduler, 
testcase should not create
[Oct 27, 2016 6:46:59 AM] (iwasakims) HADOOP-13017. Implementations of 
InputStream.read(buffer, offset, bytes)
[Oct 27, 2016 6:50:15 AM] (vinayakumarb) HDFS-9929. Duplicate keys in 
NAMENODE_SPECIFIC_KEYS (Contributed by
[Oct 27, 2016 7:30:57 AM] (aajisaka) HDFS-11049. The description of 
dfs.block.replicator.classname is not
[Oct 27, 2016 8:11:49 AM] (varunsaxena) YARN-5752. 
TestLocalResourcesTrackerImpl#testLocalResourceCache times
[Oct 27, 2016 8:25:17 AM] (varunsaxena) YARN-5710. Fix inconsistent naming in 
class ResourceRequest (Yufei Gu
[Oct 27, 2016 8:32:29 AM] (varunsaxena) YARN-5686. DefaultContainerExecutor 
random working dir algorigthm skews
[Oct 27, 2016 8:46:03 AM] (vinayakumarb) HDFS-10769. BlockIdManager.clear 
doesn't reset the counter for
[Oct 27, 2016 10:06:59 AM] (varunsaxena) MAPREDUCE-6798. Fix intermittent 
failure of
[Oct 27, 2016 11:14:00 AM] (vinayakumarb) HDFS-8492. DN should notify NN when 
client requests a missing block
[Oct 27, 2016 11:40:02 AM] (naganarasimha_gr) YARN-3848. 
TestNodeLabelContainerAllocation is timing out. Contributed
[Oct 27, 2016 11:51:01 AM] (varunsaxena) YARN-5757. RM REST API documentation 
is not up to date (Miklos Szegedi
[Oct 27, 2016 12:33:13 PM] (naganarasimha_gr) MAPREDUCE-6541. Exclude scheduled 
reducer memory when calculating
[Oct 27, 2016 12:52:07 PM] (naganarasimha_gr) YARN-5420. Delete




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService 
   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.timelineservice.storage.common.TestRowKeys 
   hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters 
   hadoop.yarn.server.timelineservice.storage.common.TestSeparator 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.resourcemanager.TestResourceTrackerService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   
hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl
 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity

[jira] [Resolved] (YARN-2981) DockerContainerExecutor must support a Cluster-wide default Docker image

2016-10-27 Thread Vinod Kumar Vavilapalli (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-2981.
---
Resolution: Won't Fix

Agreed. I am closing this as "WON'T FIX" for now given YARN-5388. Please reopen 
this if you disagree. Thanks.

> DockerContainerExecutor must support a Cluster-wide default Docker image
> 
>
> Key: YARN-2981
> URL: https://issues.apache.org/jira/browse/YARN-2981
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abin Shahab
>Assignee: Abin Shahab
>  Labels: oct16-easy
> Attachments: YARN-2981.patch, YARN-2981.patch, YARN-2981.patch, 
> YARN-2981.patch
>
>
> This allows the yarn administrator to add a cluster-wide default docker image 
> that will be used when there are no per-job override of docker images. With 
> this features, it would be convenient for newer applications like slider to 
> launch inside a cluster-default docker container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5788) AM limit resource in UI and REST not updated after -replaceNode

2016-10-27 Thread Bibin A Chundatt (JIRA) 
Bibin A Chundatt created YARN-5788:
--

 Summary: AM limit resource in UI and REST not updated after 
-replaceNode
 Key: YARN-5788
 URL: https://issues.apache.org/jira/browse/YARN-5788
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor


Steps to reproduce
==
# Enable node labels
# Configure capacity scheduler xml with label capacity configuration
# Add labelx to cluster
# Replace Node labels on node1
# Check scheduler and Rest API

*Actual*
Am limit in scheduler UI is still based on old resource
*Expected*
AM limit to be updated based new partition resource



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2016-10-27 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/

[Oct 26, 2016 6:59:39 AM] (rkanter) YARN-5753. fix NPE in 
AMRMClientImpl.getMatchingRequests() (haibochen
[Oct 26, 2016 11:39:09 AM] (weichiu) HADOOP-13659. Upgrade jaxb-api version. 
Contributed by Sean Mackrory.
[Oct 26, 2016 1:07:53 PM] (kihwal) HDFS-11050. Change log level to 'warn' when 
ssl initialization fails and
[Oct 26, 2016 2:16:13 PM] (kihwal) HDFS-11053. Unnecessary superuser check in 
versionRequest(). Contributed
[Oct 26, 2016 3:27:26 PM] (cnauroth) HADOOP-13614. Purge some 
superfluous/obsolete S3 FS tests that are
[Oct 26, 2016 3:55:42 PM] (cnauroth) HADOOP-13502. Split 
fs.contract.is-blobstore flag into more descriptive
[Oct 26, 2016 5:32:35 PM] (lei) HDFS-10638. Modifications to remove the 
assumption that StorageLocation
[Oct 26, 2016 6:31:00 PM] (sjlee) YARN-5433. Audit dependencies for Category-X. 
Contributed by Sangjin
[Oct 26, 2016 9:08:54 PM] (weichiu) Addendum patch for HADOOP-13514 Upgrade 
maven surefire plugin to 2.19.1.
[Oct 26, 2016 9:11:38 PM] (liuml07) HDFS-10921. TestDiskspaceQuotaUpdate 
doesn't wait for NN to get out of
[Oct 26, 2016 9:25:03 PM] (wang) HADOOP-8299. ViewFileSystem link slash mount 
point crashes with
[Oct 27, 2016 2:39:02 AM] (aengineer) HDFS-11038. DiskBalancer: support running 
multiple commands in single
[Oct 27, 2016 6:04:07 AM] (rohithsharmaks) YARN-4555. 
TestDefaultContainerExecutor#testContainerLaunchError fails
[Oct 27, 2016 6:27:17 AM] (rohithsharmaks) YARN-4363. In TestFairScheduler, 
testcase should not create




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.ha.TestZKFailoverController 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager 
   org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy 
   
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
   
org.apache.hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
 
   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS 
   org.apache.hadoop.tools.TestHadoopArchives 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-compile-javac-root.txt
  [168K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [120K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [120K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [276K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-unit-hadoop-tools_hadoop-archives.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [16K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/207/artifact/out/patch-asflicense-problems.txt
  [4.0K]

[jira] [Resolved] (YARN-1468) TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.

2016-10-27 Thread Jason Lowe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved YARN-1468.
--
Resolution: Duplicate

Closing this as a duplicate of YARN-5416 since that other JIRA has a proposed 
patch and recent discussion.

> TestRMRestart.testRMRestartWaitForPreviousAMToFinish get failed.
> 
>
> Key: YARN-1468
> URL: https://issues.apache.org/jira/browse/YARN-1468
> Project: Hadoop YARN
>  Issue Type: Test
>  Components: resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
>
> Log is as following:
> {code}
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 149.968 sec 
> <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> testRMRestartWaitForPreviousAMToFinish(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
>   Time elapsed: 44.197 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: AppAttempt state is not correct 
> (timedout) expected: but was:
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.MockAM.waitForState(MockAM.java:82)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:292)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.launchAM(TestRMRestart.java:826)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartWaitForPreviousAMToFinish(TestRMRestart.java:464)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] HADOOP-13603 - Remove package line length checkstyle rule

2016-10-27 Thread Shane Kumpf 
Thank you to everyone for the discussion.

To summarize, it appears there are no objections with moving forward
with HADOOP-13603,
which will remove the package line length checkstyle rule. Removing or
expanding the 80 character line length limit globally will not be changed
at this time.

Suppression of other checkstyle rules can be accomplished via annotations,
where appropriate, as of HADOOP-13411 (thank you for this work, it will be
quite useful!), so additional rule changes are not warranted at this time.

I will move forward on HADOOP-13603. Please continue to share feedback
and/or let me know if you disagree with the summary above.

Thank you!
-Shane Kumpf

On Fri, Oct 21, 2016 at 12:49 PM, Andrew Wang 
wrote:

> Thanks for the clarification Akira, I'm fine with removing it for the
> package line too (and imports if that's a problem), +1.
>
> On Fri, Oct 21, 2016 at 2:02 AM, Akira Ajisaka 
> wrote:
>
> > This discussion was split into two separate topics.
> >
> > 1) Remove line length checkstyle rule for package line
> > 2) Remove line length checkstyle rule for the entire source code
> >
> > 1) I'm +1 for removing the rule for package line. I can provide a trivial
> > patch shortly in HADOOP-13603.
> >
> > 2) As Andrew said, the discussion was done in 2015. If we really want to
> > change the rule, we need to discuss again.
> >
> > Regards,
> > Akira
> >
> >
> > On 10/21/16 07:12, Andrew Wang wrote:
> >
> >> I don't think anything has really changed since we had this discussion
> in
> >> 2015 [1]. Github and gerrit and IDEs existed then too, and we decided to
> >> leave it at 80 characters due to split screens and readability.
> >>
> >> I personally still like 80 chars for these same reasons.
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970f
> >> a0f534811cfe00795a8cd1100580f27dc@1430849118@%3Ccommon-dev.h
> >> adoop.apache.org%3E
> >>
> >> On Thu, Oct 20, 2016 at 7:46 AM, John Zhuge 
> wrote:
> >>
> >> With HADOOP-13411, it is possible to suppress any checkstyle warning
> with
> >>> an annotation.
> >>>
> >>> In this case, just add the following annotation before the class or
> >>> method:
> >>>
> >>> @SuppressWarnings("checkstyle:linelength")
> >>>
> >>> However this will not work if the warning is widespread in different
> >>> classes or methods.
> >>>
> >>> Thanks,
> >>> John Zhuge
> >>>
> >>> John Zhuge
> >>> Software Engineer, Cloudera
> >>>
> >>> On Thu, Oct 20, 2016 at 3:22 AM, Steve Loughran <
> ste...@hortonworks.com>
> >>> wrote:
> >>>
> >>>
>  On 19 Oct 2016, at 14:52, Shane Kumpf 
> >
>  wrote:
> 
> >
> > All,
> >
> > I would like to start a discussion on the possibility of removing the
> > package line length checkstyle rule (HADOOP-13603
> > ).
> >
> > While working on various aspects of YARN container runtimes, all of
> my
> > pre-commit jobs would fail as the package line length exceeded 80
> > characters. While I'm all for automated checks, I feel checks need to
> >
>  be
> >>>
>  enforceable and provide value. Fixing the package line length error
> >
>  does
> >>>
>  not improve readability or maintainability of the code, and IMO should
> >
>  be
> >>>
>  removed.
> >
> >
>  I kind of agree here
> 
>  working on other projects with wider line lenghts (100, 120) means
> that
>  you find going back to 80 chars so restrictive; and as we adopt java 8
> 
> >>> code
> >>>
>  with closures, your nesting gets even more complex. Trying to fit
> things
>  into 80 char width often adds lots of line breaks which can make the
>  code
>  messier than if it need be.
> 
>  the argument against wider lines has historically been "helped
>  side-by-side" patch reviews. But we have so much patch review software
>  these days: github, gerrit, IDEs. i don't think we need to stay in
>  punched-card width code limits just because it worked with a review
> 
> >>> process
> >>>
>  of 6+ years ago
> 
> 
>  While on this topic, are there other automated checks that are
> >
>  difficult
> >>>
>  to
> 
> > enforce or you feel are not providing value (perhaps the 150 line
> >
>  method
> >>>
>  length)?
> >
> >
>  I like that as a warning sign of complexity...it's not a hard veto
> after
>  all.
> 
>  -
>  To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>  For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
> 
> 
> >>>
> >>
> >
>

[jira] [Created] (YARN-5787) Deadlock cycle in FairScheduler

2016-10-27 Thread Rohith Sharma K S (JIRA) 
Rohith Sharma K S created YARN-5787:
---

 Summary: Deadlock cycle in FairScheduler 
 Key: YARN-5787
 URL: https://issues.apache.org/jira/browse/YARN-5787
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Reporter: Rohith Sharma K S


While testing YARN-5375 patch, found a dead lock cycle in fairscheduler. 
Basically it was cause by patch. cc:/[~sandflee]

Even though in reality this scenario does not happen, it would be better to 
handle it from fairscheduler. I am not sure whether it was noticed earlier, but 
just thought it should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5786) RM REST API documentation does not capture labels and scheduler activities related endpoints

2016-10-27 Thread Varun Saxena (JIRA) 
Varun Saxena created YARN-5786:
--

 Summary: RM  REST API documentation does not capture labels and 
scheduler activities related endpoints
 Key: YARN-5786
 URL: https://issues.apache.org/jira/browse/YARN-5786
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Saxena
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5785) [YARN-3368] Accessing applications and containers list from Node page is throwing few exceptions in console

2016-10-27 Thread Sunil G (JIRA) 
Sunil G created YARN-5785:
-

 Summary: [YARN-3368] Accessing applications and containers list 
from Node page is throwing few exceptions in console
 Key: YARN-5785
 URL: https://issues.apache.org/jira/browse/YARN-5785
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn-ui-v2
Reporter: Sunil G
Assignee: Akhil PB


In node page, "List of Applications" and "List of Containers" links are causing 
few error logs in console.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5784) Ability to Auto Refresh Cluster Overview page, Application page.

2016-10-27 Thread Rohith Sharma K S (JIRA) 
Rohith Sharma K S created YARN-5784:
---

 Summary: Ability to Auto Refresh Cluster Overview page, 
Application page. 
 Key: YARN-5784
 URL: https://issues.apache.org/jira/browse/YARN-5784
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Rohith Sharma K S


It would be nice if we can provide ability to auto refresh functionality in 
YARN Web UI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org