date:20160928


[ 
https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531876#comment-15531876
 ] 

Hadoop QA commented on YARN-3139:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s 
{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 6 new + 242 unchanged - 54 fixed = 248 total (was 296) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 26s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 0s {color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830850/YARN-3139.6.patch |
| JIRA Issue | YARN-3139 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 49cd0d18bfa6 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 47f8092 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/13247/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/13247/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-YARN-Build/13247/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results |

[jira] [Updated] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters

2016-09-28 Thread Weiwei Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-4327:
--
Component/s: security

> RM can not renew  TIMELINE_DELEGATION_TOKEN in secure clusters
> --
>
> Key: YARN-4327
> URL: https://issues.apache.org/jira/browse/YARN-4327
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager, security, timelineserver
>Affects Versions: 2.7.1
> Environment: hadoop 2.7.1hdfs,yarn, mrhistoryserver, ATS all use 
> kerberos security.
> conf like this:
> 
>   hadoop.security.authorization
>   true
>   Is service-level authorization enabled?
> 
> 
>   hadoop.security.authentication
>   kerberos
>   Possible values are simple (no authentication), and kerberos
>   
> 
>Reporter: zhangshilong
>
> bin hadoop 2.7.1
> ATS conf like this: 
> 
> yarn.timeline-service.http-authentication.type
> simple
> 
> 
> yarn.timeline-service.http-authentication.kerberos.principal
> HTTP/_h...@xxx.com
> 
> 
> yarn.timeline-service.http-authentication.kerberos.keytab
> /etc/hadoop/keytabs/xxx.keytab
> 
> 
> yarn.timeline-service.principal
> xxx/_h...@xxx.com
> 
> 
> yarn.timeline-service.keytab
> /etc/hadoop/keytabs/xxx.keytab
> 
> 
> yarn.timeline-service.best-effort
> true
> 
> 
> yarn.timeline-service.enabled
> true
>   
>  
> I'd like to allow everyone to access ATS from HTTP as RM,HDFS.
> client can submit job to RM and  add TIMELINE_DELEGATION_TOKEN  to AM 
> Context, but RM can not renew  TIMELINE_DELEGATION_TOKEN and make application 
> to failure.
> RM logs:
> 2015-11-03 11:58:38,191 WARN 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
>  Unable to add the application to the delegation token renewer.
> java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, 
> Service: 10.12.38.4:8188, Ident: (owner=yarn-test, renewer=yarn-test, 
> realUser=, issueDate=1446523118046, maxDate=1447127918046, sequenceNumber=9, 
> masterKeyId=2)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:439)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:78)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:847)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:828)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: HTTP status [500], message [Null user]
> at 
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:287)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:212)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414)
> at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:396)
> at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:378)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:451)
> at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:183)
> at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:466)
> at 
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:400)
> at 
> org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:81)
> at org.apache.hadoop.security.token.Token.renew(Token.java:377)
> at 
>

[jira] [Updated] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler

2016-09-28 Thread Wangda Tan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-3139:
-
Attachment: YARN-3139.6.patch

[~jianhe], added the null check, somehow I forgot updating this. Fixed this in 
ver.6 patch, mind to review again?

> Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
> --
>
> Key: YARN-3139
> URL: https://issues.apache.org/jira/browse/YARN-3139
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager, scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3139.0.patch, YARN-3139.1.patch, YARN-3139.2.patch, 
> YARN-3139.3.patch, YARN-3139.4.patch, YARN-3139.5.patch, YARN-3139.6.patch
>
>
> Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as 
> mentioned in YARN-3091, a possible solution is using read/write lock. Other 
> fine-graind locks for specific purposes / bugs should be addressed in 
> separated tickets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs


[ 
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531569#comment-15531569
 ] 

Tao Yang commented on YARN-5683:


Thanks for comments, [~grey]. 
We think [YARN-3409|https://issues.apache.org/jira/browse/YARN-3409] is helpful 
for applications to be allocated on the required nodes and will meet the 
demands of some other requirements except this.  
We also pay attention to  
[YARN-2139|https://issues.apache.org/jira/browse/YARN-2139]. SSD is tight on 
heterogeneous clusters and might be over-allocated without constraints. Take it 
as a resource may be a good solution and we will discuss the support of storage 
type in this feature later.

> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png
>
>
> h3.  Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc) 
> using local storage (checkpoint, shuffle etc) might require high IO 
> performance. It's useful to allocate local directories to high performance 
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications 
> cannot selectively use storage media with different performance 
> characteristics. Adding awareness of storage media can allow YARN to make 
> better decisions about the placement of local directories.
> h3.  Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration 
> should allow the cluster administrator to optionally specify the storage type 
> for each local directories. Example: 
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to 
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default 
> storage type. 
> ** StorageLocation separates storage type and directory path, used by 
> LocalDirAllocator to aware the types of local dirs, the default storage type 
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the 
> local directory of the specified storage type, and will fallback to not care 
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch. 
> All application frameworks can set the environment variables 
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage 
> type of local/log directories.
> * Allow specified storage type for various frameworks (Take MapReduce as an 
> example)
> ** Add new configurations should allow application administrator to 
> optionally specify the storage type of local/log directories. (MapReduce add 
> configurations: mapreduce.job.local-storage-type and 
> mapreduce.job.log-storage-type)
> ** Support for container work directories. Set the environment variables 
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations 
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce 
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local 
> directories of frameworks (such as shuffle directories for MapReduce). 
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to 
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce.png!
> h3.  Further Discussion
> * The requirement of storage type for local/log directories may not be 
> satisfied on heterogeneous clusters. To achieve global optimum, scheduler 
> should aware and manage disk resources. 
> [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that 
> but seems not support multiple storage types, maybe we should do even more to 
> aware the storage type of disk resource?
> * Node labels or node constraints 
> ([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409]) can also make a 
> higher chance to satisfy the requirement of specified storage type.
> * Fallback strategy still needs to be concerned. Certain applications might 
> not work well when the requirement of storage type is not satisfied. When 
> none of desired storage type disk are available, should container launching 
> be failed? let AM handle?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs

[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tao Yang updated YARN-5683:
---
Description:
h3. Introduction
* Some applications of various frameworks (Flink, Spark and MapReduce etc)
using local storage (checkpoint, shuffle etc) might require high IO
performance. It's useful to allocate local directories to high performance
storage media for these applications on heterogeneous clusters.
* YARN does not distinguish different storage types and hence applications
cannot selectively use storage media with different performance
characteristics. Adding awareness of storage media can allow YARN to make
better decisions about the placement of local directories.

h3. Approach
* NodeManager will distinguish storage types for local directories.
** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
should allow the cluster administrator to optionally specify the storage type
for each local directories. Example:
[SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
[SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
** StorageType defines DISK/SSD storage types and takes DISK as the default
storage type.
** StorageLocation separates storage type and directory path, used by
LocalDirAllocator to aware the types of local dirs, the default storage type is
DISK.
** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
local directory of the specified storage type, and will fallback to not care
storage type if the requirement can not be satisfied.
** Support for container related local/log directories by ContainerLaunch. All
application frameworks can set the environment variables (LOCAL_STORAGE_TYPE
and LOG_STORAGE_TYPE) to specified the desired storage type of local/log
directories.
* Allow specified storage type for various frameworks (Take MapReduce as an
example)
** Add new configurations should allow application administrator to optionally
specify the storage type of local/log directories. (MapReduce add
configurations: mapreduce.job.local-storage-type and
mapreduce.job.log-storage-type)
** Support for container work directories. Set the environment variables
includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
should update YARNRunner and TaskAttemptImpl)
** Add storage type prefix for request path to support for other local
directories of frameworks (such as shuffle directories for MapReduce).
(MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
support for output/work directories)
** Flow diagram for MapReduce framework
!flow_diagram_for_MapReduce.png!

h3. Further Discussion
* The requirement of storage type for local/log directories may not be
satisfied on heterogeneous clusters. To achieve global optimum, scheduler
should aware and manage disk resources.
[YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that
but seems not support multiple storage types, maybe we should do even more to
aware the storage type of disk resource?
* Node labels or node constraints
([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409]) can also make a
higher chance to satisfy the requirement of specified storage type.
* Fallback strategy still needs to be concerned. Certain applications might not
work well when the requirement of storage type is not satisfied. When none of
desired storage type disk are available, should container launching be failed?
let AM handle?

was:
h3. Introduction
* Some applications of various frameworks (Flink, Spark and MapReduce etc)
using local storage (checkpoint, shuffle etc) might require high IO
performance. It's useful to allocate local directories to high performance
storage media for these applications on heterogeneous clusters.
* YARN does not distinguish different storage types and hence applications
cannot selectively use storage media with different performance
characteristics. Adding awareness of storage media can allow YARN to make
better decisions about the placement of local directories.

[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs


[ 
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531542#comment-15531542
 ] 

Hadoop QA commented on YARN-5384:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 59s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
50s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s 
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 50s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 6s 
{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
20s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 34s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830828/YARN-5384.v6.patch |
| JIRA Issue | YARN-5384 |
| Optional Tests |  asflicense

[jira] [Commented] (YARN-5486) Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests


[ 
https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531533#comment-15531533
 ] 

Hadoop QA commented on YARN-5486:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 7s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 21s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 18s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 
new + 141 unchanged - 0 fixed = 142 total (was 141) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s 
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 11s 
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 46s 
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 9s 
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 100m 21s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830811/YARN-5486.004.patch |
| JIRA Issue | YARN-5486 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 76bdc93c098c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 47f8092 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/13243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
 |
|  Test

[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs


[ 
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531522#comment-15531522
 ] 

Hadoop QA commented on YARN-5384:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 14s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 13s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
7s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 54s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 
new + 31 unchanged - 0 fixed = 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 43s 
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 30s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s 
{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 45s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
|   | 
hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
|

[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period


[ 
https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531486#comment-15531486
 ] 

Hadoop QA commented on YARN-5677:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 50s 
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 41s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830814/YARN-5677.001.patch |
| JIRA Issue | YARN-5677 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 378e0f79203f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 47f8092 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13245/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/13245/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> RM can be in active-active state for an extended period
> ---
>
> Key: YARN-5677
> URL: https://issues.apache.org/jira/browse/YARN-5677
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel

[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs

2016-09-28 Thread Sean Po (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Po updated YARN-5384:
--
Attachment: YARN-5384.v6.patch

YARN-5384.v6.patch fixes the checkstyle issues.

> Expose priority in ReservationSystem submission APIs
> 
>
> Key: YARN-5384
> URL: https://issues.apache.org/jira/browse/YARN-5384
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, 
> YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch, YARN-5384.v6.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations 
> in the YARN ReservationSystem. This JIRA is a sub-task to track the changes 
> needed in ApplicationClientProtocol to accomplish it. Please refer to the 
> design doc in the parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs


[ 
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531411#comment-15531411
 ] 

Hadoop QA commented on YARN-5384:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 15s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
5s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 
new + 132 unchanged - 0 fixed = 133 total (was 132) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s 
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 1s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 0s {color} 
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 7s 
{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 28s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent
 |
|   |

[jira] [Updated] (YARN-5677) RM can be in active-active state for an extended period


 [ 
https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-5677:
---
Attachment: YARN-5677.001.patch

This patch fixes the issue in trunk.  I opted to be conservative and wait out 
the ZK session timeout rather than failing over immediately.  The delay extends 
the period of time that the cluster is in active-active, but it hopefully 
reduces jitter in the face of minor network disturbances.

I'll need to post an entirely different fix for branch-2.7.  Should I open a 
second JIRA for that?

> RM can be in active-active state for an extended period
> ---
>
> Key: YARN-5677
> URL: https://issues.apache.org/jira/browse/YARN-5677
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.3, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-5677.001.patch
>
>
> Both branch-2.8/trunk and branch-2.7 have issues when the active RM loses 
> contact with the ZK node(s).
> In branch-2.7, the RM will retry the connection 1000 times by default.  
> Attempting to contact a node which cannot be reached is slow, which means the 
> active can take over an hour to realize it is no longer active.  I clocked it 
> at about an hour and a half in my tests.  The solution appears to be to add 
> some time awareness into the retry loop.
> In branch-2.8/trunk, there is no maximum number of retries that I see.  It 
> appears the connection will be retried forever, with the active never 
> figuring out it's no longer active.  In my testing, the active-active state 
> lasted almost 2 hours with no sign of stopping before I killed it.  The 
> solution appears to be to cap the number of retries or amount of time spent 
> retrying.
> This issue is significant because of the asynchronous nature of job 
> submission.  If the active doesn't know it's not active, it will buffer up 
> job submissions until it finally realizes it has become the standby. Then it 
> will fail all the job submissions in bulk. In high-volume workflows, that 
> behavior can create huge mass job failures.
> This issue is also important because the node managers will not fail over to 
> the new active until the old active realizes it's the standby.  Workloads 
> submitted after the old active loses contact with ZK will therefore fail to 
> be executed regardless of which RM the clients contact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5688) Make allocation of opportunistic containers asynchronous

Konstantinos Karanasos created YARN-5688:


 Summary: Make allocation of opportunistic containers asynchronous
 Key: YARN-5688
 URL: https://issues.apache.org/jira/browse/YARN-5688
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Konstantinos Karanasos


In the current implementation of the 
{{OpportunisticContainerAllocatorAMService}}, we synchronously perform the 
allocation of opportunistic containers. This results in "blocking" the service 
at the RM when scheduling the opportunistic containers.
The {{OpportunisticContainerAllocator}} should instead asynchronously run as a 
separate thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5687) Refactor TestOpportunisticContainerAllocation to extend TestAMRMClient

Konstantinos Karanasos created YARN-5687:


 Summary: Refactor TestOpportunisticContainerAllocation to extend 
TestAMRMClient
 Key: YARN-5687
 URL: https://issues.apache.org/jira/browse/YARN-5687
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Konstantinos Karanasos


Since {{TestOpportunisticContainerAllocation}} shares a lot of code with the 
{{TestAMRMClient}}, we should refactor the former, making it a subclass of the 
latter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5486) Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests


 [ 
https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5486:
-
Attachment: YARN-5486.004.patch

Adding new patch.
Fixed remaining checkstyle issues and [~asuresh]'s comments.
The testcase is not failing locally for me and does not seem related.

[~asuresh], I will create JIRAs to track the two issues you mentioned.
Good point about the opportunistic container allocation. We should make it 
asynchronous.

> Update OpportunisticContainerAllocatorAMService::allocate method to handle 
> OPPORTUNISTIC container requests
> ---
>
> Key: YARN-5486
> URL: https://issues.apache.org/jira/browse/YARN-5486
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5486.001.patch, YARN-5486.002.patch, 
> YARN-5486.003.patch, YARN-5486.004.patch
>
>
> YARN-5457 refactors the Distributed Scheduling framework to move the 
> container allocator to yarn-server-common.
> This JIRA proposes to update the allocate method in the new AM service to use 
> the OpportunisticContainerAllocator to allocate opportunistic containers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs

2016-09-28 Thread Sean Po (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Po updated YARN-5384:
--
Attachment: YARN-5384.v5.patch

Thanks [~subru] for your comments! YARN-5384.v5.patch addresses all your 
comments, and removes some unrelated changes from YARN-5384.v4.patch,

> Expose priority in ReservationSystem submission APIs
> 
>
> Key: YARN-5384
> URL: https://issues.apache.org/jira/browse/YARN-5384
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, 
> YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations 
> in the YARN ReservationSystem. This JIRA is a sub-task to track the changes 
> needed in ApplicationClientProtocol to accomplish it. Please refer to the 
> design doc in the parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5686) DefaultContainerExecutor random working dir algorigthm skews results

2016-09-28 Thread Miklos Szegedi (JIRA)

Miklos Szegedi created YARN-5686:


 Summary: DefaultContainerExecutor random working dir algorigthm 
skews results
 Key: YARN-5686
 URL: https://issues.apache.org/jira/browse/YARN-5686
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Miklos Szegedi
Priority: Minor


{code}
long randomPosition = RandomUtils.nextLong() % totalAvailable;
...
while (randomPosition > availableOnDisk[dir]) {
  randomPosition -= availableOnDisk[dir++];
}
{code}

The code above selects a disk based on the random number weighted by the free 
space on each disk respectively. For example, if I have two disks with 100 
bytes each, totalAvailable is 200. The value of randomPosition will be 0..199. 
0..99 should select the first disk, 100..199 should select the second disk 
inclusively. Random number 100 should select the second disk to be fair but 
this is not the case right now.

We need to use 
{code}
while (randomPosition >= availableOnDisk[dir])
{code}
instead of
{code}
while (randomPosition > availableOnDisk[dir])
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore

2016-09-28 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531223#comment-15531223
 ] 

Arpit Agarwal commented on YARN-5400:
-

No problem and thank you for the quick fix Robert. I verified that your latest 
commit fixed branch-2.

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch, YARN-5400.addendum.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-2995) Enhance UI to show cluster resource utilization of various container types


 [ 
https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-2995:
-
Attachment: YARN-2995.001.patch

Adding first version of the patch.

> Enhance UI to show cluster resource utilization of various container types
> --
>
> Key: YARN-2995
> URL: https://issues.apache.org/jira/browse/YARN-2995
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Sriram Rao
>Assignee: Konstantinos Karanasos
> Attachments: YARN-2995.001.patch
>
>
> This JIRA proposes to extend the Resource manager UI to show how cluster 
> resources are being used to run *guaranteed start* and *queueable* 
> containers.  For example, a graph that shows over time, the fraction of  
> running containers that are *guaranteed start* and the fraction of running 
> containers that are *queueable*. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore

2016-09-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531187#comment-15531187
 ] 

Hudson commented on YARN-5400:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10509 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10509/])
YARN-5400. Light cleanup in ZKRMStateStore (templedf via rkanter) (rkanter: rev 
bcb2528a51c33e4caff8d744c5e14c1accfc47d0)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMZKUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java


> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch, YARN-5400.addendum.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs

2016-09-28 Thread Sean Po (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Po updated YARN-5384:
--
Attachment: YARN-5384.v4.patch

> Expose priority in ReservationSystem submission APIs
> 
>
> Key: YARN-5384
> URL: https://issues.apache.org/jira/browse/YARN-5384
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, 
> YARN-5384.v3.patch, YARN-5384.v4.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations 
> in the YARN ReservationSystem. This JIRA is a sub-task to track the changes 
> needed in ApplicationClientProtocol to accomplish it. Please refer to the 
> design doc in the parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5400) Light cleanup in ZKRMStateStore


 [ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated YARN-5400:

Attachment: YARN-5400.addendum.patch

Ok, it should be fixed now.  Sorry about that.  

I've attached the addendum patch for reference.

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch, YARN-5400.addendum.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5400) Light cleanup in ZKRMStateStore


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531129#comment-15531129
 ] 

Robert Kanter edited comment on YARN-5400 at 9/28/16 10:57 PM:
---

Yup.  I was just talking to [~templedf] about that.  It's a problem with the 
JDK 7 compiler not being able to infer the diamond operator types as well as 
the JDK 8 compiler does.  I'll push an addendum patch with the explicit types 
shortly.


was (Author: rkanter):
Yup.  I was just talking to [~templedf] about that.  It's a problem with the 
JDK 7 compiler not being able to infer the diamond operator types as well as 
the JDK 8 compiler.  I'll push an addendum patch with the explicit types 
shortly.

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531129#comment-15531129
 ] 

Robert Kanter commented on YARN-5400:
-

Yup.  I was just talking to [~templedf] about that.  It's a problem with the 
JDK 7 compiler not being able to infer the diamond operator types as well as 
the JDK 8 compiler.  I'll push an addendum patch with the explicit types 
shortly.

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5466) DefaultContainerExecutor needs JavaDocs

2016-09-28 Thread Miklos Szegedi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531132#comment-15531132
 ] 

Miklos Szegedi commented on YARN-5466:
--

looks good to me +1(non-binding)

> DefaultContainerExecutor needs JavaDocs
> ---
>
> Key: YARN-5466
> URL: https://issues.apache.org/jira/browse/YARN-5466
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Minor
> Attachments: YARN-5466.001.patch
>
>
> Following on YARN-5455, let's document the DefaultContainerExecutor as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore

2016-09-28 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531122#comment-15531122
 ] 

Arpit Agarwal commented on YARN-5400:
-

Looks like this commit broke the build for branch-2. [~rkanter] can you please 
take a look?

{code}
[ERROR] COMPILATION ERROR :
[ERROR] 
/Users/aagarwal/src/commit/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java:[415,40]
 method put in interface java.util.Map cannot be applied to given types;
  required: 
java.lang.String,java.util.Map
  found: java.lang.String,java.util.HashMap
  reason: actual argument java.util.HashMap 
cannot be converted to 
java.util.Map
 by method invocation conversion
{code}

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken


[ 
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530963#comment-15530963
 ] 

Daniel Templeton commented on YARN-5685:


BTW, the bug introduced by YARN-4559 (that I had in the original description) 
would break non-embedded failover if it worked, but since it doesn't, it's 
kinda harmless.  Still wrong, though.

> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> If HA is enabled with automatic failover enabled and embedded failover 
> disabled, all RMs all come up in standby state.  To make one of them active, 
> the {{--forcemanual}} flag must be used when manually triggering the state 
> change.  Should the active go down, the standby will not become active and 
> must be manually transitioned with the {{--forcemanual}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5685) Non-embedded HA failover is broken


 [ 
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-5685:
---
Description: If HA is enabled with automatic failover enabled and embedded 
failover disabled, all RMs all come up in standby state.  To make one of them 
active, the {{--forcemanual}} flag must be used when manually triggering the 
state change.  Should the active go down, the standby will not become active 
and must be manually transitioned with the {{--forcemanual}} flag.  (was: 
YARN-4559 broke RM HA when embedded automatic failover is disabled.  The 
{{ZKRMStateStore}} will now only start its monitoring thread when automatic 
failover not enabled (which is patently useless).  I presume the intended 
change was to have the monitoring thread started when automatic failover is not 
*embedded*.

If HA is enabled with automatic failover enabled and embedded failover 
disabled, all RMs all come up in standby state.  To make one of them active, 
the {{--forcemanual}} flag must be used when manually triggering the state 
change.  Should the active go down, the standby will not become active and must 
be manually transitioned with the {{--forcemanual}} flag.)

> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> If HA is enabled with automatic failover enabled and embedded failover 
> disabled, all RMs all come up in standby state.  To make one of them active, 
> the {{--forcemanual}} flag must be used when manually triggering the state 
> change.  Should the active go down, the standby will not become active and 
> must be manually transitioned with the {{--forcemanual}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken


[ 
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530951#comment-15530951
 ] 

Daniel Templeton commented on YARN-5685:


I think I finally get it.  The net is that HA now requires either Curator or 
embedded to be enabled.  Letting the {{ZKRMStateStore}} handle the RM state 
transitions is no longer an option and hasn't been for a while.  There are only 
three ways {{transitionToActive()}} gets called:

# {{EmbeddedElectorService.becomeActive()}} -- embedded failover
# {{LeaderElectorService.isLeader()}} -- curator failover
# {{AdminService.transitionToActive()}} -- CLI

The question is what to do about non-embedded failover.  I don't think it's 
worth making non-embedded failover work again.  I think we're better off 
declaring non-embedded failover to be dead and removing the option altogether.

Given that this has been broken for a while and no one has noticed, I'm going 
to bump the priority down a bit.

> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> YARN-4559 broke RM HA when embedded automatic failover is disabled.  The 
> {{ZKRMStateStore}} will now only start its monitoring thread when automatic 
> failover not enabled (which is patently useless).  I presume the intended 
> change was to have the monitoring thread started when automatic failover is 
> not *embedded*.
> If HA is enabled with automatic failover enabled and embedded failover 
> disabled, all RMs all come up in standby state.  To make one of them active, 
> the {{--forcemanual}} flag must be used when manually triggering the state 
> change.  Should the active go down, the standby will not become active and 
> must be manually transitioned with the {{--forcemanual}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5685) Non-embedded HA failover is broken


[ 
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530951#comment-15530951
 ] 

Daniel Templeton edited comment on YARN-5685 at 9/28/16 9:37 PM:
-

I think I finally get it.  The net is that HA now requires either Curator or 
embedded to be enabled.  Letting the {{ZKRMStateStore}} handle the RM state 
transitions is no longer an option and hasn't been for a while.  There are only 
three ways {{transitionToActive()}} gets called:

# {{EmbeddedElectorService.becomeActive()}} -- embedded failover
# {{LeaderElectorService.isLeader()}} -- curator failover
# {{AdminService.transitionToActive()}} -- CLI

The question is what to do about non-embedded failover.  I don't think it's 
worth making non-embedded failover work again.  I think we're better off 
declaring non-embedded failover to be dead and removing the option altogether.


was (Author: templedf):
I think I finally get it.  The net is that HA now requires either Curator or 
embedded to be enabled.  Letting the {{ZKRMStateStore}} handle the RM state 
transitions is no longer an option and hasn't been for a while.  There are only 
three ways {{transitionToActive()}} gets called:

# {{EmbeddedElectorService.becomeActive()}} -- embedded failover
# {{LeaderElectorService.isLeader()}} -- curator failover
# {{AdminService.transitionToActive()}} -- CLI

The question is what to do about non-embedded failover.  I don't think it's 
worth making non-embedded failover work again.  I think we're better off 
declaring non-embedded failover to be dead and removing the option altogether.

Given that this has been broken for a while and no one has noticed, I'm going 
to bump the priority down a bit.

> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> YARN-4559 broke RM HA when embedded automatic failover is disabled.  The 
> {{ZKRMStateStore}} will now only start its monitoring thread when automatic 
> failover not enabled (which is patently useless).  I presume the intended 
> change was to have the monitoring thread started when automatic failover is 
> not *embedded*.
> If HA is enabled with automatic failover enabled and embedded failover 
> disabled, all RMs all come up in standby state.  To make one of them active, 
> the {{--forcemanual}} flag must be used when manually triggering the state 
> change.  Should the active go down, the standby will not become active and 
> must be manually transitioned with the {{--forcemanual}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530923#comment-15530923
 ] 

Robert Kanter commented on YARN-5400:
-

+1

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530861#comment-15530861
 ] 

Hadoop QA commented on YARN-5400:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 52 unchanged - 10 fixed = 52 total (was 62) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 0 new + 935 unchanged - 6 fixed = 935 total (was 941) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 42s 
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 23s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830776/YARN-5400.003.patch |
| JIRA Issue | YARN-5400 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2fc163e7875a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / c3b235e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13241/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/13241/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
>

[jira] [Updated] (YARN-5678) Misleading log message in FSLeafQueue and FSParentQueue


 [ 
https://issues.apache.org/jira/browse/YARN-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5678:
---
Summary: Misleading log message in FSLeafQueue and FSParentQueue  (was: 
Misleading log message in FSLeafQueue)

> Misleading log message in FSLeafQueue and FSParentQueue
> ---
>
> Key: YARN-5678
> URL: https://issues.apache.org/jira/browse/YARN-5678
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> {code}  
> private void updateDemandForApp(FSAppAttempt sched, Resource maxRes) {
> sched.updateDemand();
> Resource toAdd = sched.getDemand();
> if (LOG.isDebugEnabled()) {
>   LOG.debug("Counting resource from " + sched.getName() + " " + toAdd
>   + "; Total resource consumption for " + getName() + " now "
>   + demand);
> }
> demand = Resources.add(demand, toAdd);
> demand = Resources.componentwiseMin(demand, maxRes);
>   }
> {code}
> Change the "resource consumption" to "resource demand".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5400) Light cleanup in ZKRMStateStore


 [ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-5400:
---
Attachment: YARN-5400.003.patch

Here's a rebased version.

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch, 
> YARN-5400.003.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530688#comment-15530688
 ] 

Robert Kanter commented on YARN-5400:
-

Hmm, it doesn't apply cleanly now:
{noformat}
>> [27] 12:51 : hadoop-git (trunk) :: patch -p0 < 
>> ~/Downloads/YARN-5400.002.patch
patching file 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMZKUtils.java
patching file 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
patching file 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
Hunk #34 FAILED at 764.
Hunk #35 succeeded at 801 (offset -3 lines).
Hunk #36 succeeded at 827 (offset -3 lines).
Hunk #37 succeeded at 860 (offset -3 lines).
Hunk #38 succeeded at 881 (offset -3 lines).
Hunk #39 succeeded at 910 (offset -3 lines).
Hunk #40 succeeded at 987 (offset -3 lines).
Hunk #41 succeeded at 1024 (offset -3 lines).
1 out of 41 hunks FAILED -- saving rejects to file 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java.rej
{noformat}

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5400) Light cleanup in ZKRMStateStore


[ 
https://issues.apache.org/jira/browse/YARN-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530671#comment-15530671
 ] 

Robert Kanter commented on YARN-5400:
-

Will do momentarily.

> Light cleanup in ZKRMStateStore
> ---
>
> Key: YARN-5400
> URL: https://issues.apache.org/jira/browse/YARN-5400
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-5400.001.patch, YARN-5400.002.patch
>
>
> of {{ZKRMStateStore}} contains a plethora whitespace issues as well as some 
> icky bits, like unused variables.  This JIRA is to clean that up.  It should 
> have no functional impact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5599) Publish AM launch command to ATS


[ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530666#comment-15530666
 ] 

Hadoop QA commented on YARN-5599:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 36s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 30s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
19s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s 
{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s 
{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 3s 
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
57s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
16s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s 
{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s 
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 13s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1 
new + 232 unchanged - 1 fixed = 233 total (was 233) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s 
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s 
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK 
v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s 
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK 
v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s 
{color} | {color:green} hadoop-yarn-server-common in the patch passed with JDK 
v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 9s 
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed 
with JDK v1.8.0_101. {color} |
|

[jira] [Commented] (YARN-5685) Non-embedded HA failover is broken


[ 
https://issues.apache.org/jira/browse/YARN-5685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530642#comment-15530642
 ] 

Daniel Templeton commented on YARN-5685:


The issue is more than that change from YARN-4559.  I'm still digging, but even 
with that issue resolved the RMs are still all stuck in standby because the 
state store isn't started until the RM transitions to active, but it doesn't 
transition to active unless the state store is started.

> Non-embedded HA failover is broken
> --
>
> Key: YARN-5685
> URL: https://issues.apache.org/jira/browse/YARN-5685
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> YARN-4559 broke RM HA when embedded automatic failover is disabled.  The 
> {{ZKRMStateStore}} will now only start its monitoring thread when automatic 
> failover not enabled (which is patently useless).  I presume the intended 
> change was to have the monitoring thread started when automatic failover is 
> not *embedded*.
> If HA is enabled with automatic failover enabled and embedded failover 
> disabled, all RMs all come up in standby state.  To make one of them active, 
> the {{--forcemanual}} flag must be used when manually triggering the state 
> change.  Should the active go down, the standby will not become active and 
> must be manually transitioned with the {{--forcemanual}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler


[ 
https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530633#comment-15530633
 ] 

Hadoop QA commented on YARN-4329:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 0 new + 938 unchanged - 3 fixed = 938 total (was 941) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 18s 
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 36s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830760/YARN-4329.002.patch |
| JIRA Issue | YARN-4329 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 39b5cdbe6643 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / e19b37e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13239/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/13239/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow fetching exact reason as to why a submitted app is in ACCEPTED state in 
> Fair Scheduler
> 
>
> Key: YARN-4329
> URL: https://issues.apache.org/jira/browse/YARN-4329
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components:

[jira] [Commented] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler


[ 
https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530632#comment-15530632
 ] 

Daniel Templeton commented on YARN-4329:


Thanks, [~yufei].  Maybe I'm an old codger, but I'd really rather you have a 
single return point from {{FSAppAttempt.hasContainerForNode()}} and 
{{MaxRunningAppsEnforcer.canAppBeRunnable()}}.  There's no advantage to having 
multiple exit points in this case, and it makes maintenance a tiny bit harder.

> Allow fetching exact reason as to why a submitted app is in ACCEPTED state in 
> Fair Scheduler
> 
>
> Key: YARN-4329
> URL: https://issues.apache.org/jira/browse/YARN-4329
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Yufei Gu
> Attachments: YARN-4329.001.patch, YARN-4329.002.patch
>
>
> Similar to YARN-3946, it would be useful to capture possible reason why the 
> Application is in accepted state in FairScheduler



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4561) Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting


[ 
https://issues.apache.org/jira/browse/YARN-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530597#comment-15530597
 ] 

Joep Rottinghuis commented on YARN-4561:


All data that comes in late would be marked as suspicious. Only if accompanied 
with non-suspicious data will we keep this. If there is no other normal of 
final-value record we can discard this suspicious data.

> Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting
> ---
>
> Key: YARN-4561
> URL: https://issues.apache.org/jira/browse/YARN-4561
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> YARN-4062 deals with the flush and compaction related coprocessor basic 
> functionality. We also need to ensure we can turn compaction on/off as a 
> whole (in case of dealing with production issues) as well as provide a way to 
> allow for blacklisting and whitelisting of processing compaction for certain 
> records.
> For instance, we may want to compact only those records which belong to 
> applications in that datacenter. This way we donot interfere with hbase 
> replication causing coprocessors to process the same record in more than one 
> dc at the same time.
> Also, we might want to not compact/process certain records, perhaps whose 
> rowkey matches a certain criteria.
> Filing jira to track these enhancements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR


 [ 
https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5145:
--
Attachment: newUIInOldRMWebServer.png

> [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
> -
>
> Key: YARN-5145
> URL: https://issues.apache.org/jira/browse/YARN-5145
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Kai Sasaki
> Attachments: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch, 
> YARN-5145-YARN-3368.01.patch, newUIInOldRMWebServer.png
>
>
> Existing YARN UI configuration is under Hadoop package's directory: 
> $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to 
> $HADOOP_CONF_DIR like other configurations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR


 [ 
https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-5145:
--
Attachment: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch

Attaching a POC patch to host new web UI under same old webapp port.
{noformat}
http:///newUI/
{noformat}

Also attaching a screen shot.. I think we can choose this model..

Thoughts?

> [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
> -
>
> Key: YARN-5145
> URL: https://issues.apache.org/jira/browse/YARN-5145
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Kai Sasaki
> Attachments: 0001-YARN-5145-Run-NewUI-WithOldPort-POC.patch, 
> YARN-5145-YARN-3368.01.patch
>
>
> Existing YARN UI configuration is under Hadoop package's directory: 
> $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to 
> $HADOOP_CONF_DIR like other configurations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4561) Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting


[ 
https://issues.apache.org/jira/browse/YARN-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530564#comment-15530564
 ] 

Joep Rottinghuis commented on YARN-4561:


For late arriving data that was spooled using whatever we come up with in 
YARN-4061, we can still recognize records that came in after we got rid of the 
finalized application's last write.
If we can have the _first_ write marked as a special value, just like the final 
value is marked, then we can later on distinguish the case where we have a 
later-arriving update.

Aside from the very first record (marked special) we will always have an 
existing record that will slide in front of or behind exising values. In other 
words, we see either a new value and it is the first, or we see multiple 
values. We could use that to see if late values come in. The trick will be to 
see if we can do this after the fact ( in the read or flush compaction) rather 
than during writes. This may have to be a write-time copro that processes data 
only if it is later than the normal timewindow (more than a day old). For those 
cases we might have to do reads during writes, or at least mark records as 
suspicious for later analysis.

> Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting
> ---
>
> Key: YARN-4561
> URL: https://issues.apache.org/jira/browse/YARN-4561
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>  Labels: YARN-5355
>
> YARN-4062 deals with the flush and compaction related coprocessor basic 
> functionality. We also need to ensure we can turn compaction on/off as a 
> whole (in case of dealing with production issues) as well as provide a way to 
> allow for blacklisting and whitelisting of processing compaction for certain 
> records.
> For instance, we may want to compact only those records which belong to 
> applications in that datacenter. This way we donot interfere with hbase 
> replication causing coprocessors to process the same record in more than one 
> dc at the same time.
> Also, we might want to not compact/process certain records, perhaps whose 
> rowkey matches a certain criteria.
> Filing jira to track these enhancements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler


[ 
https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530473#comment-15530473
 ] 

Yufei Gu edited comment on YARN-4329 at 9/28/16 6:33 PM:
-

Hi [~templedf], thanks for the review. I uploaded the new patch for your 
comments. I didn't change 
{code}
diagnosticMessageBldr.append(" exceeds current queue or its parents"
+ " maximum resource allowed).");
{code}  
to two appends since concatenation of string constants is done at compile time. 


was (Author: yufeigu):
Hi [~templedf], thanks for the review. I uploaded the new patch for your 
comments. I didn't change 
{code}
diagnosticMessageBldr.append(" exceeds current queue or its parents"
+ " maximum resource allowed).");
{code}  
to two append clause since concatenation of string constants is done at compile 
time. 

> Allow fetching exact reason as to why a submitted app is in ACCEPTED state in 
> Fair Scheduler
> 
>
> Key: YARN-4329
> URL: https://issues.apache.org/jira/browse/YARN-4329
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Yufei Gu
> Attachments: YARN-4329.001.patch, YARN-4329.002.patch
>
>
> Similar to YARN-3946, it would be useful to capture possible reason why the 
> Application is in accepted state in FairScheduler



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4329) Allow fetching exact reason as to why a submitted app is in ACCEPTED state in Fair Scheduler


 [ 
https://issues.apache.org/jira/browse/YARN-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-4329:
---
Attachment: YARN-4329.002.patch

Hi [~templedf], thanks for the review. I uploaded the new patch for your 
comments. I didn't change 
{code}
diagnosticMessageBldr.append(" exceeds current queue or its parents"
+ " maximum resource allowed).");
{code}  
to two append clause since concatenation of string constants is done at compile 
time. 

> Allow fetching exact reason as to why a submitted app is in ACCEPTED state in 
> Fair Scheduler
> 
>
> Key: YARN-4329
> URL: https://issues.apache.org/jira/browse/YARN-4329
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Yufei Gu
> Attachments: YARN-4329.001.patch, YARN-4329.002.patch
>
>
> Similar to YARN-3946, it would be useful to capture possible reason why the 
> Application is in accepted state in FairScheduler



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5659) getPathFromYarnURL should use standard methods

2016-09-28 Thread Hitesh Shah (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530407#comment-15530407
 ] 

Hitesh Shah commented on YARN-5659:
---

[~sershe] Just annotate the functions that are only required by unit tests with 
@Private and @VisibleForTesting 

> getPathFromYarnURL should use standard methods
> --
>
> Key: YARN-5659
> URL: https://issues.apache.org/jira/browse/YARN-5659
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: YARN-5659.01.patch, YARN-5659.02.patch, 
> YARN-5659.03.patch, YARN-5659.04.patch, YARN-5659.04.patch, YARN-5659.patch
>
>
> getPathFromYarnURL does some string shenanigans where  standard ctors should 
> suffice.
> There are also bugs in it e.g. passing an empty scheme to the URI ctor is 
> invalid, null should be used. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5659) getPathFromYarnURL should use standard methods

2016-09-28 Thread Hitesh Shah (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530407#comment-15530407
 ] 

Hitesh Shah edited comment on YARN-5659 at 9/28/16 6:06 PM:


[~sershe] Just annotate the functions that are only required by unit tests with 
@Private and @VisibleForTesting . In this case, the simplest approach would to 
use the above annotations for all the new functions that are added as part of 
this patch. 


was (Author: hitesh):
[~sershe] Just annotate the functions that are only required by unit tests with 
@Private and @VisibleForTesting 

> getPathFromYarnURL should use standard methods
> --
>
> Key: YARN-5659
> URL: https://issues.apache.org/jira/browse/YARN-5659
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: YARN-5659.01.patch, YARN-5659.02.patch, 
> YARN-5659.03.patch, YARN-5659.04.patch, YARN-5659.04.patch, YARN-5659.patch
>
>
> getPathFromYarnURL does some string shenanigans where  standard ctors should 
> suffice.
> There are also bugs in it e.g. passing an empty scheme to the URI ctor is 
> invalid, null should be used. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5659) getPathFromYarnURL should use standard methods

2016-09-28 Thread Sergey Shelukhin (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530399#comment-15530399
 ] 

Sergey Shelukhin commented on YARN-5659:


Hmm.. which one should I add? I am not very familiar with approach in YARN. Can 
someone add those on commit?

> getPathFromYarnURL should use standard methods
> --
>
> Key: YARN-5659
> URL: https://issues.apache.org/jira/browse/YARN-5659
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: YARN-5659.01.patch, YARN-5659.02.patch, 
> YARN-5659.03.patch, YARN-5659.04.patch, YARN-5659.04.patch, YARN-5659.patch
>
>
> getPathFromYarnURL does some string shenanigans where  standard ctors should 
> suffice.
> There are also bugs in it e.g. passing an empty scheme to the URI ctor is 
> invalid, null should be used. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5685) Non-embedded HA failover is broken

Daniel Templeton created YARN-5685:
--

 Summary: Non-embedded HA failover is broken
 Key: YARN-5685
 URL: https://issues.apache.org/jira/browse/YARN-5685
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 3.0.0-alpha1, 2.9.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton
Priority: Critical


YARN-4559 broke RM HA when embedded automatic failover is disabled.  The 
{{ZKRMStateStore}} will now only start its monitoring thread when automatic 
failover not enabled (which is patently useless).  I presume the intended 
change was to have the monitoring thread started when automatic failover is not 
*embedded*.

If HA is enabled with automatic failover enabled and embedded failover 
disabled, all RMs all come up in standby state.  To make one of them active, 
the {{--forcemanual}} flag must be used when manually triggering the state 
change.  Should the active go down, the standby will not become active and must 
be manually transitioned with the {{--forcemanual}} flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2


[ 
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530324#comment-15530324
 ] 

Joep Rottinghuis commented on YARN-4061:


HBASE-12728 introduced the BufferedMutator.

> [Fault tolerance] Fault tolerant writer for timeline v2
> ---
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Li Lu
>  Labels: YARN-5355
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5599) Publish AM launch command to ATS


[ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530315#comment-15530315
 ] 

Rohith Sharma K S commented on YARN-5599:
-

updated branch-2 patch

> Publish AM launch command to ATS
> 
>
> Key: YARN-5599
> URL: https://issues.apache.org/jira/browse/YARN-5599
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, 
> 0003-YARN-5599.patch, YARN-5599-branch-2.patch
>
>
> To aid in debugging launch failures, it would be valuable to have an 
> application's launch script and logs posted to ATS.  Because the 
> application's command line may contain private credentials or other secure 
> information, access to the data in ATS should be restricted to the job owner, 
> including the at-rest data.
> Along with making the data available through ATS, the configuration parameter 
> introduced in YARN-5549 and the log line that it guards should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5599) Publish AM launch command to ATS


 [ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5599:

Attachment: YARN-5599-branch-2.patch

> Publish AM launch command to ATS
> 
>
> Key: YARN-5599
> URL: https://issues.apache.org/jira/browse/YARN-5599
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, 
> 0003-YARN-5599.patch, YARN-5599-branch-2.patch
>
>
> To aid in debugging launch failures, it would be valuable to have an 
> application's launch script and logs posted to ATS.  Because the 
> application's command line may contain private credentials or other secure 
> information, access to the data in ATS should be restricted to the job owner, 
> including the at-rest data.
> Along with making the data available through ATS, the configuration parameter 
> introduced in YARN-5549 and the log line that it guards should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out


[ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530187#comment-15530187
 ] 

Hadoop QA commented on YARN-4205:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 3 
new + 569 unchanged - 3 fixed = 572 total (was 572) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s 
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 52s 
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 6s {color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830718/0007-YARN-4205.2.patch
 |
| JIRA Issue | YARN-4205 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  xml  |
| uname | Linux 403272cbfc8b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / e19b37e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |

[jira] [Commented] (YARN-5676) Add a HashBasedRouterPolicy, that routes jobs based on queue name hash.

2016-09-28 Thread Carlo Curino (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530223#comment-15530223
 ] 

Carlo Curino commented on YARN-5676:


For very large queue structures, it is convenient to specify a policy for a 
subset of the queues (e.g., large/prod queues),  and use something like the 
{{HashBasedRouterPolicy}} to automatically map all the small to subclusters. 
This is better than any of the randomized policies because the mapping between 
queue and sub-cluster tends to be more stable, hence there is some better 
chance for locality.

I will post the patch soon (testing now).

> Add a HashBasedRouterPolicy, that routes jobs based on queue name hash.
> ---
>
> Key: YARN-5676
> URL: https://issues.apache.org/jira/browse/YARN-5676
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Affects Versions: YARN-2915
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis


 [ 
https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-5600:
---
Assignee: Miklos Szegedi  (was: Daniel Templeton)

> Add a parameter to ContainerLaunchContext to emulate 
> yarn.nodemanager.delete.debug-delay-sec on a per-application basis
> ---
>
> Key: YARN-5600
> URL: https://issues.apache.org/jira/browse/YARN-5600
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>
> To make debugging application launch failures simpler, I'd like to add a 
> parameter to the CLC to allow an application owner to request delayed 
> deletion of the application's launch artifacts.
> This JIRA solves largely the same problem as YARN-5599, but for cases where 
> ATS is not in use, e.g. branch-2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS

2016-09-28 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530162#comment-15530162
 ] 

Brahma Reddy Battula commented on YARN-3432:


Oh sorry, missed this..didn't get time to work.. will try to update the unit 
testcase on next week.

> Cluster metrics have wrong Total Memory when there is reserved memory on CS
> ---
>
> Key: YARN-3432
> URL: https://issues.apache.org/jira/browse/YARN-3432
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacityscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: Thomas Graves
>Assignee: Brahma Reddy Battula
> Attachments: YARN-3432-002.patch, YARN-3432-003.patch, YARN-3432.patch
>
>
> I noticed that when reservations happen when using the Capacity Scheduler, 
> the UI and web services report the wrong total memory.
> For example.  I have a 300GB of total memory in my cluster.  I allocate 50 
> and I reserve 10.  The cluster metrics for total memory get reported as 290GB.
> This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps 
> there is a difference between fair scheduler and capacity scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period


[ 
https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530124#comment-15530124
 ] 

Daniel Templeton commented on YARN-5677:


[~jianhe], is leader election on by default in Hadoop 3?  I'd recommend it.  
Shall I roll that into the patch for this JIRA?

> RM can be in active-active state for an extended period
> ---
>
> Key: YARN-5677
> URL: https://issues.apache.org/jira/browse/YARN-5677
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.3, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> Both branch-2.8/trunk and branch-2.7 have issues when the active RM loses 
> contact with the ZK node(s).
> In branch-2.7, the RM will retry the connection 1000 times by default.  
> Attempting to contact a node which cannot be reached is slow, which means the 
> active can take over an hour to realize it is no longer active.  I clocked it 
> at about an hour and a half in my tests.  The solution appears to be to add 
> some time awareness into the retry loop.
> In branch-2.8/trunk, there is no maximum number of retries that I see.  It 
> appears the connection will be retried forever, with the active never 
> figuring out it's no longer active.  In my testing, the active-active state 
> lasted almost 2 hours with no sign of stopping before I killed it.  The 
> solution appears to be to cap the number of retries or amount of time spent 
> retrying.
> This issue is significant because of the asynchronous nature of job 
> submission.  If the active doesn't know it's not active, it will buffer up 
> job submissions until it finally realizes it has become the standby. Then it 
> will fail all the job submissions in bulk. In high-volume workflows, that 
> behavior can create huge mass job failures.
> This issue is also important because the node managers will not fail over to 
> the new active until the old active realizes it's the standby.  Workloads 
> submitted after the old active loses contact with ZK will therefore fail to 
> be executed regardless of which RM the clients contact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period


[ 
https://issues.apache.org/jira/browse/YARN-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530118#comment-15530118
 ] 

Daniel Templeton commented on YARN-5677:


Just tested the leader election (with the right property enabled), and it works 
as advertised.  The active becomes standby by the time the standby becomes 
active.

> RM can be in active-active state for an extended period
> ---
>
> Key: YARN-5677
> URL: https://issues.apache.org/jira/browse/YARN-5677
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.3, 3.0.0-alpha1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> Both branch-2.8/trunk and branch-2.7 have issues when the active RM loses 
> contact with the ZK node(s).
> In branch-2.7, the RM will retry the connection 1000 times by default.  
> Attempting to contact a node which cannot be reached is slow, which means the 
> active can take over an hour to realize it is no longer active.  I clocked it 
> at about an hour and a half in my tests.  The solution appears to be to add 
> some time awareness into the retry loop.
> In branch-2.8/trunk, there is no maximum number of retries that I see.  It 
> appears the connection will be retried forever, with the active never 
> figuring out it's no longer active.  In my testing, the active-active state 
> lasted almost 2 hours with no sign of stopping before I killed it.  The 
> solution appears to be to cap the number of retries or amount of time spent 
> retrying.
> This issue is significant because of the asynchronous nature of job 
> submission.  If the active doesn't know it's not active, it will buffer up 
> job submissions until it finally realizes it has become the standby. Then it 
> will fail all the job submissions in bulk. In high-volume workflows, that 
> behavior can create huge mass job failures.
> This issue is also important because the node managers will not fail over to 
> the new active until the old active realizes it's the standby.  Workloads 
> submitted after the old active loses contact with ZK will therefore fail to 
> be executed regardless of which RM the clients contact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5145) [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR


[ 
https://issues.apache.org/jira/browse/YARN-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530061#comment-15530061
 ] 

Sunil G commented on YARN-5145:
---

Currently {{/logs}} has added as a new context to existing webserver in RM. 
Similarly we could try adding a new Context for new web UI. I am making some 
prototype for same, and will update status here.

> [YARN-3368] Move new YARN UI configuration to HADOOP_CONF_DIR
> -
>
> Key: YARN-5145
> URL: https://issues.apache.org/jira/browse/YARN-5145
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Kai Sasaki
> Attachments: YARN-5145-YARN-3368.01.patch
>
>
> Existing YARN UI configuration is under Hadoop package's directory: 
> $HADOOP_PREFIX/share/hadoop/yarn/webapps/, we should move it to 
> $HADOOP_CONF_DIR like other configurations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2


[ 
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529989#comment-15529989
 ] 

Joep Rottinghuis commented on YARN-4061:


I'm going to write up some requirements for what we'd want from a spooling 
BufferedMutator and open an HBase jira for the same. We get a type safe 
BufferedMutator from a shared connection in 
o.a.h.yarn.server.timelineservice.storage.common.BaseTable#getBufferedMutator 

In HBase the BufferedMutatorImpl is created up in 
HConnectionImplementation#getBufferedMutator, a static class inside 
ConnectionManager. It implements ClusterConnection which extends (the 
deprecated) HConnection, which in turn extends the Connection interface. While 
we can pass BufferedMutatorParams, but have no other way to inject our own 
implementation.
This means we'll have to either wrap the return implementation and override and 
delegate to the methods, or have some modifications in HBase in order to be 
able to create a different implementation.

BufferedMutatorParams has a (setter) method #listener which allows us to pass a 
BufferedMutator.ExceptionListener, this gives a glimmer of hope to be able to 
capture exceptions and possibly direct the BufferedMutator wrapper class to 
(temporarily) spool mutations to a FileSystem implementation. This could be 
HDFS, local filesystem, s3, gcs, or whatever the user wants to configure as a 
perfix path.



> [Fault tolerance] Fault tolerant writer for timeline v2
> ---
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Li Lu
>Assignee: Li Lu
>  Labels: YARN-5355
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out


 [ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4205:

Attachment: 0007-YARN-4205.2.patch

Fixing TestPBImplRecords failure and some of the checkstyles 

> Add a service for monitoring application life time out
> --
>
> Key: YARN-4205
> URL: https://issues.apache.org/jira/browse/YARN-4205
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, 
> 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, 
> 0006-YARN-4205.patch, 0007-YARN-4205.1.patch, 0007-YARN-4205.2.patch, 
> 0007-YARN-4205.patch, YARN-4205_01.patch, YARN-4205_02.patch, 
> YARN-4205_03.patch
>
>
> This JIRA intend to provide a lifetime monitor service. 
> The service will monitor the applications where the life time is configured. 
> If the application is running beyond the lifetime, it will be killed. 
> The lifetime will be considered from the submit time.
> The thread monitoring interval is configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5684) testDecreaseAfterIncreaseWithAllocationExpiration fails intermittently

Rohith Sharma K S created YARN-5684:
---

 Summary: testDecreaseAfterIncreaseWithAllocationExpiration fails 
intermittently 
 Key: YARN-5684
 URL: https://issues.apache.org/jira/browse/YARN-5684
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Rohith Sharma K S


Saw the following in a precommit:
{code}
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
testDecreaseAfterIncreaseWithAllocationExpiration(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer)
  Time elapsed: 10.726 sec  <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer.testDecreaseAfterIncreaseWithAllocationExpiration(TestIncreaseAllocationExpirer.java:367)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out


[ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529649#comment-15529649
 ] 

Hadoop QA commented on YARN-4205:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 7s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 20 
new + 539 unchanged - 3 fixed = 559 total (was 542) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 35s {color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 48s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 |
|   | hadoop.yarn.server.resourcemanager.TestRMAdminService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830685/0007-YARN-4205.1.patch
 |
| JIRA Issue | YARN-4205 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  xml  |
| uname | Linux 9550dff67904 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build

[jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs

2016-09-28 Thread Lei Guo (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529614#comment-15529614
 ] 

Lei Guo commented on YARN-5683:
---

This is related the constraint label YARN-3409, but different at detail level. 
This will rely on the constraint label for initial scheduling to find server 
with proper resource, and then node manager to help to set up proper 
environment for the workload. It's not just storage, we will also face similar 
issue on GPU or other hardware accelerator.

> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png
>
>
> h3.  Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc) 
> using local storage (checkpoint, shuffle etc) might require high IO 
> performance. It's useful to allocate local directories to high performance 
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications 
> cannot selectively use storage media with different performance 
> characteristics. Adding awareness of storage media can allow YARN to make 
> better decisions about the placement of local directories.
> h3.  Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration 
> should allow the cluster administrator to optionally specify the storage type 
> for each local directories. Example: 
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to 
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default 
> storage type. 
> ** StorageLocation separates storage type and directory path, used by 
> LocalDirAllocator to aware the types of local dirs, the default storage type 
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the 
> local directory of the specified storage type, and will fallback to not care 
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch. 
> All application frameworks can set the environment variables 
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage 
> type of local/log directories.
> * Allow specified storage type for various frameworks (Take MapReduce as an 
> example)
> ** Add new configurations should allow application administrator to 
> optionally specify the storage type of local/log directories. (MapReduce add 
> configurations: mapreduce.job.local-storage-type and 
> mapreduce.job.log-storage-type)
> ** Support for container work directories. Set the environment variables 
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations 
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce 
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local 
> directories of frameworks (such as shuffle directories for MapReduce). 
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to 
> support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce.png!
> h3.  Further Discussion
> * The requirement of storage type for local/log directories may not be 
> satisfied on heterogeneous clusters. To achieve global optimum, scheduler 
> should aware and management disk resources to. 
> [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that 
> but seems not support multiple storage types, maybe we should do even more to 
> aware the storage type of disk resource?
> * Node labels or node constraints can also make a higher chance to satisfy 
> the requirement of specified storage type.
> * Fallback strategy still needs to be concerned. Certain applications might 
> not work well when the requirement of storage type is not satisfied. When 
> none of desired storage type disk are available, should container launching 
> be failed? let AM handle?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5599) Publish AM launch command to ATS

2016-09-28 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529549#comment-15529549
 ] 

Varun Saxena commented on YARN-5599:


[~rohithsharma] can you provide a branch-2 patch ?
This would basically mean dropping ATSv2 changes.

> Publish AM launch command to ATS
> 
>
> Key: YARN-5599
> URL: https://issues.apache.org/jira/browse/YARN-5599
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, 
> 0003-YARN-5599.patch
>
>
> To aid in debugging launch failures, it would be valuable to have an 
> application's launch script and logs posted to ATS.  Because the 
> application's command line may contain private credentials or other secure 
> information, access to the data in ATS should be restricted to the job owner, 
> including the at-rest data.
> Along with making the data available through ATS, the configuration parameter 
> introduced in YARN-5549 and the log line that it guards should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out


[ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529513#comment-15529513
 ] 

Hadoop QA commented on YARN-4205:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 57s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 17s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 18 
new + 540 unchanged - 3 fixed = 558 total (was 543) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 36s {color} 
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 37s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
20s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 44s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
|   | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.TestRMAdminService |
|   | hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830678/0007-YARN-4205.patch |
| JIRA Issue | YARN-4205 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  xml  |
| uname | Linux f39ec40bf432 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64

[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out


 [ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4205:

Attachment: 0007-YARN-4205.1.patch

I forgot to include test file. Updating new patch with Test file included.

> Add a service for monitoring application life time out
> --
>
> Key: YARN-4205
> URL: https://issues.apache.org/jira/browse/YARN-4205
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, 
> 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, 
> 0006-YARN-4205.patch, 0007-YARN-4205.1.patch, 0007-YARN-4205.patch, 
> YARN-4205_01.patch, YARN-4205_02.patch, YARN-4205_03.patch
>
>
> This JIRA intend to provide a lifetime monitor service. 
> The service will monitor the applications where the life time is configured. 
> If the application is running beyond the lifetime, it will be killed. 
> The lifetime will be considered from the submit time.
> The thread monitoring interval is configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler


[ 
https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529409#comment-15529409
 ] 

Jian He edited comment on YARN-3139 at 9/28/16 12:10 PM:
-

Patch looks good to me. 

bq. Instead of taking readLock, I think we can check if 
getNode(container.getNodeId()) == null, and stop if it is.
In the latest patch, do you intend to not check "getNode(container.getNodeId()) 
== null" ? I think this is possible if a node is removed while the container on 
that is being released by AM.






was (Author: jianhe):
Patch looks good to me. 

bq. Instead of taking readLock, I think we can check if 
getNode(container.getNodeId()) == null, and stop if it is.
In the latest patch, do you intend to not check "getNode(container.getNodeId()) 
== null" ?





> Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
> --
>
> Key: YARN-3139
> URL: https://issues.apache.org/jira/browse/YARN-3139
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager, scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3139.0.patch, YARN-3139.1.patch, YARN-3139.2.patch, 
> YARN-3139.3.patch, YARN-3139.4.patch, YARN-3139.5.patch
>
>
> Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as 
> mentioned in YARN-3091, a possible solution is using read/write lock. Other 
> fine-graind locks for specific purposes / bugs should be addressed in 
> separated tickets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3139) Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler


[ 
https://issues.apache.org/jira/browse/YARN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529409#comment-15529409
 ] 

Jian He commented on YARN-3139:
---

Patch looks good to me. 

bq. Instead of taking readLock, I think we can check if 
getNode(container.getNodeId()) == null, and stop if it is.
In the latest patch, do you intend to not check "getNode(container.getNodeId()) 
== null" ?





> Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler
> --
>
> Key: YARN-3139
> URL: https://issues.apache.org/jira/browse/YARN-3139
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager, scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-3139.0.patch, YARN-3139.1.patch, YARN-3139.2.patch, 
> YARN-3139.3.patch, YARN-3139.4.patch, YARN-3139.5.patch
>
>
> Enhance locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler, as 
> mentioned in YARN-3091, a possible solution is using read/write lock. Other 
> fine-graind locks for specific purposes / bugs should be addressed in 
> separated tickets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out


[ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529380#comment-15529380
 ] 

Jian He commented on YARN-4205:
---

latest patch looks good to me

> Add a service for monitoring application life time out
> --
>
> Key: YARN-4205
> URL: https://issues.apache.org/jira/browse/YARN-4205
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, 
> 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, 
> 0006-YARN-4205.patch, 0007-YARN-4205.patch, YARN-4205_01.patch, 
> YARN-4205_02.patch, YARN-4205_03.patch
>
>
> This JIRA intend to provide a lifetime monitor service. 
> The service will monitor the applications where the life time is configured. 
> If the application is running beyond the lifetime, it will be killed. 
> The lifetime will be considered from the submit time.
> The thread monitoring interval is configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out


 [ 
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4205:

Attachment: 0007-YARN-4205.patch

Updated the patch with following major changes from previous one with more 
general way.

# Moved RMAppLifetimeMonitor to separate package i.e 
*org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor*
# Created RMAppToMonitor custom class to hold monitoring key. This consists of 
ApplicationId+ApplicationTimeoutType. This is essential because same 
application can have multiple timeouts.
# Removed ApplicationTimeouts class.
# In ApplicationSubmissionContext#setApplicationTimeouts is changed to 
Map. In future, if we want to support more 
timeouts like in Gour use case, we just need to add enum and provide 
corresponding implementation in RMAppImpl. 

> Add a service for monitoring application life time out
> --
>
> Key: YARN-4205
> URL: https://issues.apache.org/jira/browse/YARN-4205
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-4205.patch, 0002-YARN-4205.patch, 
> 0003-YARN-4205.patch, 0004-YARN-4205.patch, 0005-YARN-4205.patch, 
> 0006-YARN-4205.patch, 0007-YARN-4205.patch, YARN-4205_01.patch, 
> YARN-4205_02.patch, YARN-4205_03.patch
>
>
> This JIRA intend to provide a lifetime monitor service. 
> The service will monitor the applications where the life time is configured. 
> If the application is running beyond the lifetime, it will be killed. 
> The lifetime will be considered from the submit time.
> The thread monitoring interval is configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs

[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

h3. Further Discussion
* The requirement of storage type for local/log directories may not be
satisfied on heterogeneous clusters. To achieve global optimum, scheduler
should aware and management disk resources to.
[YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that
but seems not support multiple storage types, maybe we should do even more to
aware the storage type of disk resource?
* Node labels or node constraints can also make a higher chance to satisfy the
requirement of specified storage type.
* Fallback strategy still needs to be concerned. Certain applications might not
work well when the requirement of storage type is not satisfied. When none of
desired storage type disk are available, should container launching be failed?
let AM handle?

[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs

[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

h3. Further Discussion
* The requirement of storage type for local/log directories may not be
satisfied on heterogeneous clusters. To achieve global optimum, scheduler
should aware and management disk resources to.
[YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that
but seems not support multiple storage types, maybe we should do even more to
aware the storage type of disk resource?
* Node labels or node constraints can also make a higher chance to satisfy the
requirement of specified storage type.
* Fallback strategy still needs to be concerned. Certain applications might not
work well when the requirement of storage type is not satisfied. When none of
desired storage type disk are available, should container launching be failed?
let AM handle?

was:
h1. Introduction
* Some applications of various frameworks (Flink, Spark and MapReduce etc)
using local storage (checkpoint, shuffle etc) might require high IO
performance. It's useful to allocate local directories to high performance
storage media for these applications on heterogeneous clusters.
* YARN does not distinguish different storage types and hence applications
cannot selectively use storage media with different performance
characteristics. Adding awareness of storage media can allow YARN to make
better decisions about the placement of local/log directories with input from
applications. An application can choose the desired storage media by
configuration based on its performance and requirements.

[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs


 [ 
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Yang updated YARN-5683:
---
Attachment: flow_diagram_for_MapReduce.png

> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5683-1.patch, flow_diagram_for_MapReduce.png
>
>
> h1.  Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc) 
> using local storage (checkpoint, shuffle etc) might require high IO 
> performance. It's useful to allocate local directories to high performance 
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications 
> cannot selectively use storage media with different performance 
> characteristics. Adding awareness of storage media can allow YARN to make 
> better decisions about the placement of local/log directories with input from 
> applications. An application can choose the desired storage media by 
> configuration based on its performance and requirements.
> h1.  Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration 
> should allow the cluster administrator to optionally specify the storage type 
> for each local directories. Example: 
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to 
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default 
> storage type. 
> ** StorageLocation separates storage type and directory path, used by 
> LocalDirAllocator to aware the types of local dirs, the default storage type 
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the 
> local directory of the specified storage type, and will fallback to not care 
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch. 
> All application frameworks can set the environment variables 
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage 
> type of local/log directories.
> * Allow specified storage type for various frameworks (Take MapReduce as an 
> example)
> ** Add new configurations should allow application administrator to 
> optionally specify the storage type of local/log directories. (MapReduce add 
> configurations: mapreduce.job.local-storage-type and 
> mapreduce.job.log-storage-type)
> ** Support for container work directories. Set the environment variables 
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations 
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce 
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local 
> directories of frameworks (such as shuffle directories for MapReduce). 
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to 
> support for output/work directories)
> h1.  Further Discussion
> * The requirement of storage type for local/log directories may not be 
> satisfied on heterogeneous clusters. To achieve global optimum, scheduler 
> should aware and management disk resources to. 
> [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that 
> but seems not support multiple storage types, maybe we should do even more to 
> aware the storage type of disk resource?
> * Node labels or node constraints can also make a higher chance to satisfy 
> the requirement of specified storage type.
> * Fallback strategy still needs to be concerned. Certain applications might 
> not work well when the requirement of storage type is not satisfied. When 
> none of desired storage type disk are available, should container launching 
> be failed? let AM handle?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs


 [ 
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Yang updated YARN-5683:
---
Attachment: YARN-5683-1.patch

> Support specifying storage type for per-application local dirs
> --
>
> Key: YARN-5683
> URL: https://issues.apache.org/jira/browse/YARN-5683
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Tao Yang
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-5683-1.patch
>
>
> h1.  Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc) 
> using local storage (checkpoint, shuffle etc) might require high IO 
> performance. It's useful to allocate local directories to high performance 
> storage media for these applications on heterogeneous clusters.
> * YARN does not distinguish different storage types and hence applications 
> cannot selectively use storage media with different performance 
> characteristics. Adding awareness of storage media can allow YARN to make 
> better decisions about the placement of local/log directories with input from 
> applications. An application can choose the desired storage media by 
> configuration based on its performance and requirements.
> h1.  Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration 
> should allow the cluster administrator to optionally specify the storage type 
> for each local directories. Example: 
> [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to 
> [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default 
> storage type. 
> ** StorageLocation separates storage type and directory path, used by 
> LocalDirAllocator to aware the types of local dirs, the default storage type 
> is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the 
> local directory of the specified storage type, and will fallback to not care 
> storage type if the requirement can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch. 
> All application frameworks can set the environment variables 
> (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to specified the desired storage 
> type of local/log directories.
> * Allow specified storage type for various frameworks (Take MapReduce as an 
> example)
> ** Add new configurations should allow application administrator to 
> optionally specify the storage type of local/log directories. (MapReduce add 
> configurations: mapreduce.job.local-storage-type and 
> mapreduce.job.log-storage-type)
> ** Support for container work directories. Set the environment variables 
> includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations 
> above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce 
> should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local 
> directories of frameworks (such as shuffle directories for MapReduce). 
> (MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to 
> support for output/work directories)
> h1.  Further Discussion
> * The requirement of storage type for local/log directories may not be 
> satisfied on heterogeneous clusters. To achieve global optimum, scheduler 
> should aware and management disk resources to. 
> [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that 
> but seems not support multiple storage types, maybe we should do even more to 
> aware the storage type of disk resource?
> * Node labels or node constraints can also make a higher chance to satisfy 
> the requirement of specified storage type.
> * Fallback strategy still needs to be concerned. Certain applications might 
> not work well when the requirement of storage type is not satisfied. When 
> none of desired storage type disk are available, should container launching 
> be failed? let AM handle?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5683) Support specifying storage type for per-application local dirs

[
https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tao Yang updated YARN-5683:
---
Description:
h1. Introduction
* Some applications of various frameworks (Flink, Spark and MapReduce etc)
using local storage (checkpoint, shuffle etc) might require high IO
performance. It's useful to allocate local directories to high performance
storage media for these applications on heterogeneous clusters.
* YARN does not distinguish different storage types and hence applications
cannot selectively use storage media with different performance
characteristics. Adding awareness of storage media can allow YARN to make
better decisions about the placement of local/log directories with input from
applications. An application can choose the desired storage media by
configuration based on its performance and requirements.

h1. Approach
* NodeManager will distinguish storage types for local directories.
** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
should allow the cluster administrator to optionally specify the storage type
for each local directories. Example:
[SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
[SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
** StorageType defines DISK/SSD storage types and takes DISK as the default
storage type.
** StorageLocation separates storage type and directory path, used by
LocalDirAllocator to aware the types of local dirs, the default storage type is
DISK.
** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the
local directory of the specified storage type, and will fallback to not care
storage type if the requirement can not be satisfied.
** Support for container related local/log directories by ContainerLaunch. All
application frameworks can set the environment variables (LOCAL_STORAGE_TYPE
and LOG_STORAGE_TYPE) to specified the desired storage type of local/log
directories.
* Allow specified storage type for various frameworks (Take MapReduce as an
example)
** Add new configurations should allow application administrator to optionally
specify the storage type of local/log directories. (MapReduce add
configurations: mapreduce.job.local-storage-type and
mapreduce.job.log-storage-type)
** Support for container work directories. Set the environment variables
includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations
above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce
should update YARNRunner and TaskAttemptImpl)
** Add storage type prefix for request path to support for other local
directories of frameworks (such as shuffle directories for MapReduce).
(MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to
support for output/work directories)

h1. Further Discussion
* The requirement of storage type for local/log directories may not be
satisfied on heterogeneous clusters. To achieve global optimum, scheduler
should aware and management disk resources to.
[YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that
but seems not support multiple storage types, maybe we should do even more to
aware the storage type of disk resource?
* Node labels or node constraints can also make a higher chance to satisfy the
requirement of specified storage type.
* Fallback strategy still needs to be concerned. Certain applications might not
work well when the requirement of storage type is not satisfied. When none of
desired storage type disk are available, should container launching be failed?
let AM handle?

was:
# Introduction
* Some applications of various frameworks (Flink, Spark and MapReduce etc)
using local storage (checkpoint, shuffle etc) might require high IO
performance. It's useful to allocate local directories to high performance
storage media for these applications on heterogeneous clusters.
* YARN does not distinguish different storage types and hence applications
cannot selectively use storage media with different performance
characteristics. Adding awareness of storage media can allow YARN to make
better decisions about the placement of local/log directories with input from
applications. An application can choose the desired storage media by
configuration based on its performance and requirements.

# Approach
* NodeManager will distinguish storage types for local directories.
** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration
should allow the cluster administrator to optionally specify the storage type
for each local directories. Example:
[SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to
[SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
** StorageType defines DISK/SSD storage types and takes DISK as the default
storage type.
** StorageLocation separates storage type

[jira] [Created] (YARN-5683) Support specifying storage type for per-application local dirs

Tao Yang created YARN-5683:
--

 Summary: Support specifying storage type for per-application local 
dirs
 Key: YARN-5683
 URL: https://issues.apache.org/jira/browse/YARN-5683
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: nodemanager
Affects Versions: 3.0.0-alpha2
Reporter: Tao Yang
 Fix For: 3.0.0-alpha2


# Introduction
* Some applications of various frameworks (Flink, Spark and MapReduce etc) 
using local storage (checkpoint, shuffle etc) might require high IO 
performance. It's useful to allocate local directories to high performance 
storage media for these applications on heterogeneous clusters.
* YARN does not distinguish different storage types and hence applications 
cannot selectively use storage media with different performance 
characteristics. Adding awareness of storage media can allow YARN to make 
better decisions about the placement of local/log directories with input from 
applications. An application can choose the desired storage media by 
configuration based on its performance and requirements.

# Approach
* NodeManager will distinguish storage types for local directories.
** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration 
should allow the cluster administrator to optionally specify the storage type 
for each local directories. Example: 
[SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to 
[SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
** StorageType defines DISK/SSD storage types and takes DISK as the default 
storage type. 
** StorageLocation separates storage type and directory path, used by 
LocalDirAllocator to aware the types of local dirs, the default storage type is 
DISK.
** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the 
local directory of the specified storage type, and will fallback to not care 
storage type if the requirement can not be satisfied.
** Support for container related local/log directories by ContainerLaunch. All 
application frameworks can set the environment variables (LOCAL_STORAGE_TYPE 
and LOG_STORAGE_TYPE) to specified the desired storage type of local/log 
directories.
* Allow specified storage type for various frameworks (Take MapReduce as an 
example)
** Add new configurations should allow application administrator to optionally 
specify the storage type of local/log directories. (MapReduce add 
configurations: mapreduce.job.local-storage-type and 
mapreduce.job.log-storage-type)
** Support for container work directories. Set the environment variables 
includes LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE according to configurations 
above for ContainerLaunchContext and ApplicationSubmissionContext. (MapReduce 
should update YARNRunner and TaskAttemptImpl)
** Add storage type prefix for request path to support for other local 
directories of frameworks (such as shuffle directories for MapReduce). 
(MapReduce should update YarnOutputFiles, MROutputFiles and YarnChild to 
support for output/work directories)

# Further Discussion
* The requirement of storage type for local/log directories may not be 
satisfied on heterogeneous clusters. To achieve global optimum, scheduler 
should aware and management disk resources to. 
[YARN-2139|https://issues.apache.org/jira/browse/YARN-2139] is close to that 
but seems not support multiple storage types, maybe we should do even more to 
aware the storage type of disk resource?
* Node labels or node constraints can also make a higher chance to satisfy the 
requirement of specified storage type.
* Fallback strategy still needs to be concerned. Certain applications might not 
work well when the requirement of storage type is not satisfied. When none of 
desired storage type disk are available, should container launching be failed? 
let AM handle?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5662) Provide an option to enable ContainerMonitor

2016-09-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529266#comment-15529266
 ] 

Hudson commented on YARN-5662:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10506 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10506/])
YARN-5662. Provide an option to enable ContainerMonitor. Contributed by 
(vvasudev: rev bc2656f09f857fdbc39da6b060cee453d2dec4dc)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerStatusPBImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


> Provide an option to enable ContainerMonitor 
> -
>
> Key: YARN-5662
> URL: https://issues.apache.org/jira/browse/YARN-5662
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5662.1.patch, YARN-5662.2.patch, YARN-5662.3.patch
>
>
> Currently, if vmem/pmem check is not enabled, ContainerMonitor would not run. 
>  In certain cases, ContainerMonitor also needs to run to monitor things like 
> container-metrics. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5599) Publish AM launch command to ATS

2016-09-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529267#comment-15529267
 ] 

Hudson commented on YARN-5599:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10506 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10506/])
YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun 
(varunsaxena: rev 9b0fd01d2ee002ac4c30c2862e18ca8f1626fa8d)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV1Publisher.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV2Publisher.java


> Publish AM launch command to ATS
> 
>
> Key: YARN-5599
> URL: https://issues.apache.org/jira/browse/YARN-5599
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, 
> 0003-YARN-5599.patch
>
>
> To aid in debugging launch failures, it would be valuable to have an 
> application's launch script and logs posted to ATS.  Because the 
> application's command line may contain private credentials or other secure 
> information, access to the data in ATS should be restricted to the job owner, 
> including the at-rest data.
> Along with making the data available through ATS, the configuration parameter 
> introduced in YARN-5549 and the log line that it guards should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4855) Should check if node exists when replace nodelabels

2016-09-28 Thread Naganarasimha G R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529000#comment-15529000
 ] 

Naganarasimha G R commented on YARN-4855:
-

bq. so it will be better to not add the hyphen-connected option.
agree...

bq.  will strongly suggest to use the cliParser, but that can come in a 
separated patch.
Ok In that case lets try to target this at the earliest and will not commit 
this patch to 2.8 until the followup jira comes in.

[~Tao Jie], hope you can quickly create a patch to modify the option to 
"failOnUnknownNodes" and raise a new jira to use CliParser's options where ever 
required.

> Should check if node exists when replace nodelabels
> ---
>
> Key: YARN-4855
> URL: https://issues.apache.org/jira/browse/YARN-4855
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Minor
> Attachments: YARN-4855.001.patch, YARN-4855.002.patch, 
> YARN-4855.003.patch, YARN-4855.004.patch, YARN-4855.005.patch, 
> YARN-4855.006.patch, YARN-4855.007.patch, YARN-4855.008.patch, 
> YARN-4855.009.patch, YARN-4855.010.patch, YARN-4855.011.patch, 
> YARN-4855.012.patch
>
>
> Today when we add nodelabels to nodes, it would succeed even if nodes are not 
> existing NodeManger in cluster without any message.
> It could be like this:
> When we use *yarn rmadmin -replaceLabelsOnNode --fail-on-unkown-nodes 
> "node1=label1"* , it would be denied if node is unknown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5682) [YARN-3368] Fix maven build to keep all generated or downloaded files in target folder


[ 
https://issues.apache.org/jira/browse/YARN-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528995#comment-15528995
 ] 

Sunil G commented on YARN-5682:
---

Hi [~wangda]

I ran {{mvn install -Pyarn-ui}} with and w/o {{mvn clean}} after applying this 
patch. i think we are taking more time now.

{noformat}
mvn clean; mvn install -Pyarn-ui
[INFO] Apache Hadoop YARN UI .. SUCCESS [01:56 min]

mvn install -Pyarn-ui
[INFO] Apache Hadoop YARN UI .. SUCCESS [01:51 min]
{noformat}

I tested {{mvn install -Pyarn-ui}} w/o this patch. It is taking lesser time.
I think we need not have to clean temp dir's.

{noformat}
[INFO] Total time: 29.515 s
{noformat}


> [YARN-3368] Fix maven build to keep all generated or downloaded files in 
> target folder
> --
>
> Key: YARN-5682
> URL: https://issues.apache.org/jira/browse/YARN-5682
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-5682-YARN-3368.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4743) ResourceManager crash because TimSort


[ 
https://issues.apache.org/jira/browse/YARN-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528957#comment-15528957
 ] 

Yufei Gu edited comment on YARN-4743 at 9/28/16 9:06 AM:
-

Hi [~gzh1992n], thanks for working on this. Some thoughts about the patch. Both 
0.5(weight value less than 1.0) or 0.0 are valid value for weights in fair 
scheduler. Once use case of zero-weight would be that user uses the zero-weight 
queue to run jobs when there is no jobs for other non-zero-weight queues. So it 
make no sense to me to enforce weight larger than 1.0. 
If NaN affects the transitive, we can avoid NaN by other ways. For example, if 
the first weight is 0.0 and the second is bigger than 0.0, obviously, the 
second one is needier than the first one.


was (Author: yufeigu):
Hi [~gzh1992n], thanks for working on this. Some thoughts about the patch. Both 
0.5(weight value less than 1.0) or 0.0 are valid value for weights in fair 
scheduler. Once use case of zero-weight would be that user uses the zero-weight 
queue to run jobs when there is no jobs for other non-zero-weight queues. So it 
make no sense to me to enforce weight larger than 1.0. 

> ResourceManager crash because TimSort
> -
>
> Key: YARN-4743
> URL: https://issues.apache.org/jira/browse/YARN-4743
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.6.4
>Reporter: Zephyr Guo
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-4743-v1.patch, YARN-CDH5.4.7.patch, timsort.log
>
>
> {code}
> 2016-02-26 14:08:50,821 FATAL 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
> handling event type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Comparison method violates its general 
> contract!
>  at java.util.TimSort.mergeHi(TimSort.java:868)
>  at java.util.TimSort.mergeAt(TimSort.java:485)
>  at java.util.TimSort.mergeCollapse(TimSort.java:410)
>  at java.util.TimSort.sort(TimSort.java:214)
>  at java.util.TimSort.sort(TimSort.java:173)
>  at java.util.Arrays.sort(Arrays.java:659)
>  at java.util.Collections.sort(Collections.java:217)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:316)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:240)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1091)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:989)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1185)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:112)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:684)
>  at java.lang.Thread.run(Thread.java:745)
> 2016-02-26 14:08:50,822 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..
> {code}
> Actually, this issue found in 2.6.0-cdh5.4.7.
> I think the cause is that we modify {{Resouce}} while we are sorting 
> {{runnableApps}}.
> {code:title=FSLeafQueue.java}
> Comparator comparator = policy.getComparator();
> writeLock.lock();
> try {
>   Collections.sort(runnableApps, comparator);
> } finally {
>   writeLock.unlock();
> }
> readLock.lock();
> {code}
> {code:title=FairShareComparator}
> public int compare(Schedulable s1, Schedulable s2) {
> ..
>   s1.getResourceUsage(), minShare1);
>   boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
>   s2.getResourceUsage(), minShare2);
>   minShareRatio1 = (double) s1.getResourceUsage().getMemory()
>   / Resources.max(RESOURCE_CALCULATOR, null, minShare1, 
> ONE).getMemory();
>   minShareRatio2 = (double) s2.getResourceUsage().getMemory()
>   / Resources.max(RESOURCE_CALCULATOR, null, minShare2, 
> ONE).getMemory();
> ..
> {code}
> {{getResourceUsage}} will return current Resource. The current Resource is 
> unstable. 
> {code:title=FSAppAttempt.java}
> @Override
>   public Resource getResourceUsage() {
> // Here the getPreemptedResources() always return zero, except in
> // a preemption round
> return Resources.subtract(getCurrentConsumption(), 
> getPreemptedResources());
>   }
> {code}
> {code:title=SchedulerApplicationAttempt}
>  public Resource getCurrentConsumption() {
> return currentConsumption;
>   }
> // This method may modify

[jira] [Commented] (YARN-4743) ResourceManager crash because TimSort


[ 
https://issues.apache.org/jira/browse/YARN-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528957#comment-15528957
 ] 

Yufei Gu commented on YARN-4743:


Hi [~gzh1992n], thanks for working on this. Some thoughts about the patch. Both 
0.5(weight value less than 1.0) or 0.0 are valid value for weights in fair 
scheduler. Once use case of zero-weight would be that user uses the zero-weight 
queue to run jobs when there is no jobs for other non-zero-weight queues. So it 
make no sense to me to enforce weight larger than 1.0. 

> ResourceManager crash because TimSort
> -
>
> Key: YARN-4743
> URL: https://issues.apache.org/jira/browse/YARN-4743
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.6.4
>Reporter: Zephyr Guo
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-4743-v1.patch, YARN-CDH5.4.7.patch, timsort.log
>
>
> {code}
> 2016-02-26 14:08:50,821 FATAL 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in 
> handling event type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Comparison method violates its general 
> contract!
>  at java.util.TimSort.mergeHi(TimSort.java:868)
>  at java.util.TimSort.mergeAt(TimSort.java:485)
>  at java.util.TimSort.mergeCollapse(TimSort.java:410)
>  at java.util.TimSort.sort(TimSort.java:214)
>  at java.util.TimSort.sort(TimSort.java:173)
>  at java.util.Arrays.sort(Arrays.java:659)
>  at java.util.Collections.sort(Collections.java:217)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:316)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.assignContainer(FSParentQueue.java:240)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1091)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:989)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1185)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:112)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:684)
>  at java.lang.Thread.run(Thread.java:745)
> 2016-02-26 14:08:50,822 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..
> {code}
> Actually, this issue found in 2.6.0-cdh5.4.7.
> I think the cause is that we modify {{Resouce}} while we are sorting 
> {{runnableApps}}.
> {code:title=FSLeafQueue.java}
> Comparator comparator = policy.getComparator();
> writeLock.lock();
> try {
>   Collections.sort(runnableApps, comparator);
> } finally {
>   writeLock.unlock();
> }
> readLock.lock();
> {code}
> {code:title=FairShareComparator}
> public int compare(Schedulable s1, Schedulable s2) {
> ..
>   s1.getResourceUsage(), minShare1);
>   boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
>   s2.getResourceUsage(), minShare2);
>   minShareRatio1 = (double) s1.getResourceUsage().getMemory()
>   / Resources.max(RESOURCE_CALCULATOR, null, minShare1, 
> ONE).getMemory();
>   minShareRatio2 = (double) s2.getResourceUsage().getMemory()
>   / Resources.max(RESOURCE_CALCULATOR, null, minShare2, 
> ONE).getMemory();
> ..
> {code}
> {{getResourceUsage}} will return current Resource. The current Resource is 
> unstable. 
> {code:title=FSAppAttempt.java}
> @Override
>   public Resource getResourceUsage() {
> // Here the getPreemptedResources() always return zero, except in
> // a preemption round
> return Resources.subtract(getCurrentConsumption(), 
> getPreemptedResources());
>   }
> {code}
> {code:title=SchedulerApplicationAttempt}
>  public Resource getCurrentConsumption() {
> return currentConsumption;
>   }
> // This method may modify current Resource.
> public synchronized void recoverContainer(RMContainer rmContainer) {
> ..
> Resources.addTo(currentConsumption, rmContainer.getContainer()
>   .getResource());
> ..
>   }
> {code}
> I suggest that use stable Resource in comparator.
> Is there something i think wrong？



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5682) [YARN-3368] Fix maven build to keep all generated or downloaded files in target folder


[ 
https://issues.apache.org/jira/browse/YARN-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528887#comment-15528887
 ] 

Hadoop QA commented on YARN-5682:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m 58s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
45s {color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s 
{color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 9s 
{color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
42s {color} | {color:green} YARN-3368 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s 
{color} | {color:green} YARN-3368 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 25s {color} 
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 7s 
{color} | {color:green} hadoop-yarn-ui in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 4s {color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer |
|   | hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
|   | 
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9baccb9 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12830637/YARN-5682-YARN-3368.001.patch
 |
| JIRA Issue | YARN-5682 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  |
| uname | Linux a6bb981b5b67 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-3368 / 9ef8291 |
| Default Java | 1.8.0_101 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/13234/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-YARN-Build/13234/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/13234/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: 
hadoop-yarn-project/hadoop-yarn |
| Console output |

[jira] [Commented] (YARN-5554) MoveApplicationAcrossQueues does not check user permission on the target queue


[ 
https://issues.apache.org/jira/browse/YARN-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528729#comment-15528729
 ] 

Yufei Gu commented on YARN-5554:


The latest patch looks good to me. +1(non-binding). 

> MoveApplicationAcrossQueues does not check user permission on the target queue
> --
>
> Key: YARN-5554
> URL: https://issues.apache.org/jira/browse/YARN-5554
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.2
>Reporter: Haibo Chen
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-5554.2.patch, YARN-5554.3.patch, YARN-5554.4.patch, 
> YARN-5554.5.patch, YARN-5554.6.patch, YARN-5554.7.patch
>
>
> moveApplicationAcrossQueues operation currently does not check user 
> permission on the target queue. This incorrectly allows one user to move 
> his/her own applications to a queue that the user has no access to



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5599) Publish AM launch command to ATS

2016-09-28 Thread Varun Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528688#comment-15528688
 ] 

Varun Saxena commented on YARN-5599:


+1.
Will commit shortly.

> Publish AM launch command to ATS
> 
>
> Key: YARN-5599
> URL: https://issues.apache.org/jira/browse/YARN-5599
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, 
> 0003-YARN-5599.patch
>
>
> To aid in debugging launch failures, it would be valuable to have an 
> application's launch script and logs posted to ATS.  Because the 
> application's command line may contain private credentials or other secure 
> information, access to the data in ATS should be restricted to the job owner, 
> including the at-rest data.
> Along with making the data available through ATS, the configuration parameter 
> introduced in YARN-5549 and the log line that it guards should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5599) Publish AM launch command to ATS

2016-09-28 Thread Varun Saxena (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-5599:
---
Summary: Publish AM launch command to ATS  (was: Post AM launcher artifacts 
to ATS)

> Publish AM launch command to ATS
> 
>
> Key: YARN-5599
> URL: https://issues.apache.org/jira/browse/YARN-5599
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-5599.patch, 0002-YARN-5599.patch, 
> 0003-YARN-5599.patch
>
>
> To aid in debugging launch failures, it would be valuable to have an 
> application's launch script and logs posted to ATS.  Because the 
> application's command line may contain private credentials or other secure 
> information, access to the data in ATS should be restricted to the job owner, 
> including the at-rest data.
> Along with making the data available through ATS, the configuration parameter 
> introduced in YARN-5549 and the log line that it guards should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5677) RM can be in active-active state for an extended period