[jira] [Commented] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982364#comment-15982364
 ] 

John Zhuge commented on YARN-5007:
--

If 3.0 is not a good time to remove deprecated method, when is a good time?

Maybe the method should not have been marked as {{deprecated}}?

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>  Labels: oct16-easy
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-6467) CSQueueMetrics needs to update the current metrics for default partition only

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-6467:
---

Assignee: Manikandan R  (was: Naganarasimha G R)

> CSQueueMetrics needs to update the current metrics for default partition only
> -
>
> Key: YARN-6467
> URL: https://issues.apache.org/jira/browse/YARN-6467
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2
>Reporter: Naganarasimha G R
>Assignee: Manikandan R
> Attachments: YARN-6467.001.patch
>
>
> As a followup to YARN-6195, we need to update existing metrics to only 
> default Partition.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6467) CSQueueMetrics needs to update the current metrics for default partition only

[Manikandan R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982343#comment-15982343
 ] 

Manikandan R commented on YARN-6467:


[~Naganarasimha]

I am working on it, should be able to submit patch very soon. Please assign.

> CSQueueMetrics needs to update the current metrics for default partition only
> -
>
> Key: YARN-6467
> URL: https://issues.apache.org/jira/browse/YARN-6467
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6467.001.patch
>
>
> As a followup to YARN-6195, we need to update existing metrics to only 
> default Partition.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6519:
--
Attachment: YARN-6519.002.patch

> Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager
> 
>
> Key: YARN-6519
> URL: https://issues.apache.org/jira/browse/YARN-6519
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6519.001.patch, YARN-6519.002.patch
>
>
> There is 8 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager$1.compare(CSQueue,
>  CSQueue) incorrectly handles float value
> # org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType.index 
> field is public and mutable
> # 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.queueMetrics
>  is a mutable collection
> # 
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.cleanupStaledPreemptionCandidates(long)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.transferStateFromAttempt(RMAppAttempt)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.cleanupPreemptionList()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> See more from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982320#comment-15982320
 ] 

Weiwei Yang commented on YARN-6519:
---

Hi [~Naganarasimha]

Thanks for the review

bq. NodeType ln no 29, no need to change the access specifier of the constructor

Intellij gives a warning {{Modifier private is redundant for enum 
constructors}} if I keep the private modifier. This is because 
[http://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.9.2] says 
{noformat}
In an enum declaration, a constructor declaration with no access modifiers is 
private.
{noformat}

bq. EMPTY_CONTAINER_LIST can be private

Sure, fixed.

bq. QueueMetrics ln no 151, i was of the opinion that we use naming convention 
of field as its not exactly a string constant, thoughts?

This is using name convention for {{final}} and {{static}} field, if not name 
it this way, we will get checkstyle warnings, similar names can be found a lot 
of places, such as

{noformat}
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java:98:
  public static final Map DEFAULT_CODERS_MAP = ImmutableMap.of(
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:465:
private static final Map USER_ID_NAME_CACHE =
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:468:
private static final Map GROUP_ID_NAME_CACHE =
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:702:
  private static final Map uidCache =
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ObjectWritable.java:85:
  private static final Map PRIMITIVE_NAMES = new 
HashMap();
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/WritableFactories.java:33:
  private static final Map CLASS_TO_FACTORY =
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java:190:
  private static final Map PROTOCOL_ENGINES
{noformat}

4. is it possible for concurrent modification exception ?

Good catch! Thanks! Fixed with following

{code}
preemptionCandidates.entrySet()
.removeIf(candidate ->
candidate.getValue() + 2 * maxWaitTime < currentTime);
{code}

5. FSSchedulerNode ln no 157-165, i presume for removal we better not use entry 
set,

Also a good catch, thanks. In order to fix the findbugs warning, we can still 
use entry set, just to iterate over this entry set 

{code}
Iterator> iterator =
resourcesPreemptedForApp.entrySet().iterator();
{code}

Thanks for all the comments! Appreciate your review.

> Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager
> 
>
> Key: YARN-6519
> URL: https://issues.apache.org/jira/browse/YARN-6519
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6519.001.patch
>
>
> There is 8 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager$1.compare(CSQueue,
>  CSQueue) incorrectly handles float value
> # org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType.index 
> field is public and mutable
> # 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.queueMetrics
>  is a mutable collection
> # 
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.cleanupStaledPreemptionCandidates(long)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.transferStateFromAttempt(RMAppAttempt)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.cleanupPreemptionList()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> See more from 
> 

[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second

[JackZhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982319#comment-15982319
 ] 

JackZhou commented on YARN-6516:


[~yufeigu] I am test in a real cluster which have about 2500 nodes. I have 
already set continuous scheduling on but I set the 
yarn.scheduler.fair.continuous-scheduling-sleep-ms to 500, so it is run per 500 
ms. There is about 80 parent queues in my scheduler and about 200 queues total.
I think the scheduler assign a thousand containers per second is a pretty ideal 
scenario, because if my queue is very empty 
it will cost 1ms to assign a container for scheduler. 
But in my test, I have two queues,  the queue information as blow:
Used Resources: 
Num Active Applications:19
Num Pending Applications:   1057
Min Resources:  
Max Resources:  
Max Running Applications:   4000
Steady Fair Share:  
Instantaneous Fair Share:   

Used Resources: 
Num Active Applications:20
Num Pending Applications:   781
Min Resources:  
Max Resources:  
Max Running Applications:   4000
Steady Fair Share:  
Instantaneous Fair Share:   


The cost to assign a container up to about 3ms, and the scheduler only can 
scheduler about 40 containers.
It is so slow!

Apps Submitted  Apps PendingApps RunningApps Completed  Containers 
Running  Memory Used Memory TotalMemory Reserved VCores Used 
VCores TotalVCores Reserved Active NodesDecommissioned NodesLost 
Nodes  Unhealthy Nodes Rebooted Nodes
10268   183839  839139  39 GB   95 TB   0 B 39  97280   
0   24322   64  0   0
User Metrics for hadoop

> FairScheduler:the algorithm of assignContainer is so slow for it only can 
> assign a thousand containers per second
> -
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: JackZhou
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6524) Avoid storing unnecessary information in the Memory for the finished apps

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982309#comment-15982309
 ] 

Naganarasimha G R commented on YARN-6524:
-

IMO we can reset the unused/unnecessary fields (not used in any of the API's or 
UI) to null on application finished or recovery of finished apps  to avoid this 
issue.

> Avoid storing unnecessary information in the Memory for the finished apps
> -
>
> Key: YARN-6524
> URL: https://issues.apache.org/jira/browse/YARN-6524
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: RM
>Affects Versions: 2.7.3
>Reporter: Naganarasimha G R
>
> Avoid storing unnecessary information in the Memory for the finished apps
> In case of cluster with large number of finished apps, more memory is 
> required to store the unused information i.e. related AM's Container launch 
> like Localization resources, tokens etc. 
> In one such scenario we had around 9k finished apps each with 257 
> LocalResource amounting to 108 kbytes per app and just for 9k apps it was 
> nearly taking ~ 0.8 GB of memory. In Low end machines this would create 
> resource crunch in RM



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6524) Avoid storing unnecessary information in the Memory for the finished apps

[Naganarasimha G R (JIRA)]

Naganarasimha G R created YARN-6524:
---

 Summary: Avoid storing unnecessary information in the Memory for 
the finished apps
 Key: YARN-6524
 URL: https://issues.apache.org/jira/browse/YARN-6524
 Project: Hadoop YARN
  Issue Type: Bug
  Components: RM
Affects Versions: 2.7.3
Reporter: Naganarasimha G R


Avoid storing unnecessary information in the Memory for the finished apps
In case of cluster with large number of finished apps, more memory is required 
to store the unused information i.e. related AM's Container launch like 
Localization resources, tokens etc. 
In one such scenario we had around 9k finished apps each with 257 LocalResource 
amounting to 108 kbytes per app and just for 9k apps it was nearly taking ~ 0.8 
GB of memory. In Low end machines this would create resource crunch in RM




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982284#comment-15982284
 ] 

Naganarasimha G R commented on YARN-6523:
-

Approach depends on why we are sending credentials for all apps which i am not 
completely clear. IMO it should be sufficient to send the tokens for the apps 
(containers) active on the node.
Possible solutions :
# Send only app credentials related to the node on each heartbeat
# Send only app credentials related to the node on each heartbeat and also 
delta modifications for the node since the last heartbeat.
# Cache SystemCredentialsForAppsProto objects itself and reuse them rather than 
recreating for each node's heartbeat.(if require to send all the apps token to 
the node)

P.S. credit goes to [~gu chi] for analysis of this issue.


> RM requires large memory in sending out security tokens as part of Node 
> Heartbeat in large cluster
> --
>
> Key: YARN-6523
> URL: https://issues.apache.org/jira/browse/YARN-6523
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: RM
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Critical
>
> Currently as part of heartbeat response RM sets all application's tokens 
> though all applications might not be active on the node. On top of it 
> NodeHeartbeatResponsePBImpl converts tokens for each app into 
> SystemCredentialsForAppsProto. Hence for each node and each heartbeat too 
> many SystemCredentialsForAppsProto objects were getting created.
> We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 
> 8GB RAM configured for RM



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster

[Naganarasimha G R (JIRA)]

Naganarasimha G R created YARN-6523:
---

 Summary: RM requires large memory in sending out security tokens 
as part of Node Heartbeat in large cluster
 Key: YARN-6523
 URL: https://issues.apache.org/jira/browse/YARN-6523
 Project: Hadoop YARN
  Issue Type: Bug
  Components: RM
Affects Versions: 2.7.3, 2.8.0
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Priority: Critical


Currently as part of heartbeat response RM sets all application's tokens though 
all applications might not be active on the node. On top of it 
NodeHeartbeatResponsePBImpl converts tokens for each app into 
SystemCredentialsForAppsProto. Hence for each node and each heartbeat too many 
SystemCredentialsForAppsProto objects were getting created.
We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 
8GB RAM configured for RM




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)

[Carlo Curino (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982263#comment-15982263
 ] 

Carlo Curino commented on YARN-3663:


[~giovanni.fumarola] thanks for addressing my comments. I understand all your 
answers, and I am ok to postpone some of the refactorings I asked (we should 
think a bit more about especially for {{SQLFederationStateStore}} point 7).

Overall the patch is +1, but please:
 # Open all the follow-up JIRAs we mentioned as sub-tasks of YARN-5597 
(federation v2)
 # Address the first 2 checkstyle ({{UTC_CALENDAR}})... I think it should be 
{{private}}, and the naming could be a standard {{utcCalendar}}.

Patch is good to commit once that is done. If you get it done tonight or 
tomorrow morning, I will commit it tomorrow late morning, otherwise I asked 
[~subru] to take-over reviewing and commit,
 as I will be traveling with limited connectivity for a few days.


> Federation State and Policy Store (DBMS implementation)
> ---
>
> Key: YARN-3663
> URL: https://issues.apache.org/jira/browse/YARN-3663
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Affects Versions: YARN-2915
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-3663-YARN-2915.v1.patch, 
> YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and 
> Policy Store, which implements YARN-3662 APIs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6512) Fix for FindBugs getProcessList() possible NPE

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982207#comment-15982207
 ] 

Naganarasimha G R commented on YARN-6512:
-

 [~wilfreds], 
[~cheersyang] was already working on findbugs for yarn under umbrella jira  
HADOOP-14336 , just yesterday we concluded that he split yarn jira into project 
specific jira's and hence he created YARN-6517 - YARN-6520 and split his 
existing patch and uploaded. So as WeiWei yang had started first i thought of 
making this jira duplicate of YARN-6517. thoughts?

> Fix for FindBugs getProcessList() possible NPE
> --
>
> Key: YARN-6512
> URL: https://issues.apache.org/jira/browse/YARN-6512
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-6512.01.patch
>
>
> Findbugs output:
> {code}
> Possible null pointer dereference in 
> org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessList() due to 
> return value of called method
> Bug type NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE
> In class org.apache.hadoop.yarn.util.ProcfsBasedProcessTree
> In method org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessList()
> Value loaded from processDirs
> Dereferenced at ProcfsBasedProcessTree.java:[line 487]
> Known null at ProcfsBasedProcessTree.java:[line 484]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6513) Fix for FindBugs getPendingLogFilesToUpload() possible NPE

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982205#comment-15982205
 ] 

Naganarasimha G R commented on YARN-6513:
-

[~haibochen] & [~wilfreds], 
[~cheersyang] was already working on findbugs for yarn under umbrella jira  
HADOOP-14336 , just yesterday we concluded that he split yarn jira into project 
specific jira's and hence he created YARN-6517 - YARN-6520 and split his 
existing patch and uploaded. So as WeiWei yang had started first i thought of 
making this jira duplicate of YARN-6517. thoughts?


> Fix for FindBugs getPendingLogFilesToUpload() possible NPE
> --
>
> Key: YARN-6513
> URL: https://issues.apache.org/jira/browse/YARN-6513
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-6513.01.patch
>
>
> {code}
> Possible null pointer dereference in 
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue.getPendingLogFilesToUpload(File)
>  due to return value of called method
> Bug type NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE (click for details) 
> In class org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue
> In method 
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue.getPendingLogFilesToUpload(File)
> Local variable stored in JVM register ?
> Method invoked at AggregatedLogFormat.java:[line 314]
> Known null at AggregatedLogFormat.java:[line 314]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6467) CSQueueMetrics needs to update the current metrics for default partition only

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982187#comment-15982187
 ] 

Naganarasimha G R commented on YARN-6467:
-

Hi [~maniraj...@gmail.com], any update one this ? i forgot to assign the issue 
to your name, if you still plan to do it. Will assign it.

> CSQueueMetrics needs to update the current metrics for default partition only
> -
>
> Key: YARN-6467
> URL: https://issues.apache.org/jira/browse/YARN-6467
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6467.001.patch
>
>
> As a followup to YARN-6195, we need to update existing metrics to only 
> default Partition.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6202) Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982159#comment-15982159
 ] 

Junping Du commented on YARN-6202:
--

bq. Let's just revert the removals in the patch and instead deprecate the 
config as well as the constants.
+1. That sounds like a good plan.

> Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded
> -
>
> Key: YARN-6202
> URL: https://issues.apache.org/jira/browse/YARN-6202
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6202.001.patch, YARN-6202.002.patch, 
> YARN-6202.003.patch, YARN-6202.004.patch
>
>
> Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY (yarn.dispatcher.exit-on-error) 
> always be true no matter what value in configuration files. This misleads 
> users. Two solutions: 
> # Remove the configuration item and provide a method to allow 
> {{exitOnDispatchException}}/{{shouldExitOnError}} to be false to enable 
> related unit tests. There is no need for false value in a real daemon since 
> daemons should crash if its dispatcher quit.
> # Make it default true instead of false, so that we don't need to hard code 
> it to be true in RM and NM, it is still configurable, and also provide method 
> to enable related unit tests.
> Other than that, the code around it needs to refactor. {{public static 
> final}} for a variable of interface isn't necessary, and YARN related 
> configure item should be in class YarnConfiguration. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6423) Queue metrics doesn't work for Fair Scheduler in SLS

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6423:
---
Attachment: YARN-6423.004.patch

Rebase since YARN-6363 is in.

> Queue metrics doesn't work for Fair Scheduler in SLS
> 
>
> Key: YARN-6423
> URL: https://issues.apache.org/jira/browse/YARN-6423
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6423.001.patch, YARN-6423.002.patch, 
> YARN-6423.003.patch, YARN-6423.004.patch
>
>
> Queue allocated memory and vcores always be 0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6202) Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982137#comment-15982137
 ] 

Vinod Kumar Vavilapalli commented on YARN-6202:
---

bq. Option 1 sounds good to me. Any thoughts? Vinod Kumar Vavilapalli
bq. I saw other projects use these configuration items, better to not commit 
this to branch2.
bq. Agree. An example is: TEZ-2049. Let's keep the patch in trunk only.
Folks, a bunch of downstream are using this. This is not the kind of cleanup 
that will balance out code-cleanup vs downstream pain. Let's just revert the 
removals in the patch and instead deprecate the config as well as the constants.


> Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded
> -
>
> Key: YARN-6202
> URL: https://issues.apache.org/jira/browse/YARN-6202
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6202.001.patch, YARN-6202.002.patch, 
> YARN-6202.003.patch, YARN-6202.004.patch
>
>
> Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY (yarn.dispatcher.exit-on-error) 
> always be true no matter what value in configuration files. This misleads 
> users. Two solutions: 
> # Remove the configuration item and provide a method to allow 
> {{exitOnDispatchException}}/{{shouldExitOnError}} to be false to enable 
> related unit tests. There is no need for false value in a real daemon since 
> daemons should crash if its dispatcher quit.
> # Make it default true instead of false, so that we don't need to hard code 
> it to be true in RM and NM, it is still configurable, and also provide method 
> to enable related unit tests.
> Other than that, the code around it needs to refactor. {{public static 
> final}} for a variable of interface isn't necessary, and YARN related 
> configure item should be in class YarnConfiguration. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-3774) ZKRMStateStore should use CuratorOp when we upgrade to Curator 3

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter reassigned YARN-3774:
---

Assignee: (was: Robert Kanter)
Target Version/s:   (was: 3.0.0-alpha3)
Priority: Minor  (was: Critical)
 Summary: ZKRMStateStore should use CuratorOp when we upgrade to 
Curator 3  (was: ZKRMStateStore should use CuratorOp)

That seems reasonable.  I've updated the JIRA.

> ZKRMStateStore should use CuratorOp when we upgrade to Curator 3
> 
>
> Key: YARN-3774
> URL: https://issues.apache.org/jira/browse/YARN-3774
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Priority: Minor
> Attachments: YARN-3774.001.patch
>
>
> YARN-2716 changes ZKRMStateStore to use Curator. Transactions added there are 
> somewhat involved, and could be improved using CuratorOp introduced in 
> Curator 3.0. Hadoop 3.0.0 would be a good time to upgrade the Curator version 
> and make this change. 
> Curator is considering shading guava through CURATOR-200. In Hadoop 3, we 
> should upgrade to the next Curator version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982090#comment-15982090
 ] 

Junping Du commented on YARN-5007:
--

As [~aplusplus] said, this break Tez build against Hadoop 3 and some other 
downstream projects could also get affected seriously (in case it will use 
MiniYARNCluster). I am going to revert YARN-3573 (deprecated method) and this 
(completed removal). Let me know if you have some concerns.

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>  Labels: oct16-easy
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du reopened YARN-5007:
--

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>  Labels: oct16-easy
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3573) MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked deprecated

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982086#comment-15982086
 ] 

Junping Du commented on YARN-3573:
--

I still doubt the necessary of this deprecation especially I learnt from 
TEZ-3694 where we build Tez against Hadoop 3. The constructor could be widely 
used by downstream projects and deprecation on these methods doesn't make sense 
unless we have significant gains here.

> MiniMRYarnCluster constructor that starts the timeline server using a boolean 
> should be marked deprecated
> -
>
> Key: YARN-3573
> URL: https://issues.apache.org/jira/browse/YARN-3573
> Project: Hadoop YARN
>  Issue Type: Test
>  Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: YARN-3573-002.patch, YARN-3573.patch
>
>
> {code}MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS){code}
> starts the timeline server using *boolean enableAHS*. It is better to have 
> the timelineserver started based on the config value.
> We should mark this constructor as deprecated to avoid its future use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[Zhiyuan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982066#comment-15982066
 ] 

Zhiyuan Yang edited comment on YARN-5007 at 4/24/17 10:58 PM:
--

This change breaks Tez and possibly many other downstream projects. In 2.7.0, 
the only way to enable ATS is through ctor parameters; now the only way is 
configuration. How do you expect downstream projects to support both 2.7.0 and 
3.0.0?


was (Author: aplusplus):
This change breaks Tez and possibly many other downstream projects. In 2.7.0, 
the only way to enable ATS is through ctor parameters; now the only way is 
configuration. How do you expect downstream projects to support both support 
both 2.7.0 and 3.0.0?

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>  Labels: oct16-easy
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[Zhiyuan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982066#comment-15982066
 ] 

Zhiyuan Yang edited comment on YARN-5007 at 4/24/17 10:58 PM:
--

This change breaks Tez and possibly many other downstream projects. In 2.7.0, 
the only way to enable ATS is through ctor parameters; now the only way is 
configuration. How do you expect downstream projects to support both 2.7.0 and 
3.0.0? Even a hadoop shim layer cannot fix the problem.


was (Author: aplusplus):
This change breaks Tez and possibly many other downstream projects. In 2.7.0, 
the only way to enable ATS is through ctor parameters; now the only way is 
configuration. How do you expect downstream projects to support both 2.7.0 and 
3.0.0?

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>  Labels: oct16-easy
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-5473) Expose over-allocation in the Resource Manager UI

[Haibo Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen reassigned YARN-5473:


Assignee: Haibo Chen

> Expose over-allocation in the Resource Manager UI
> -
>
> Key: YARN-5473
> URL: https://issues.apache.org/jira/browse/YARN-5473
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Inigo Goiri
>Assignee: Haibo Chen
>
> When enabling over-allocation of nodes, the resources in the cluster change. 
> We need to surface this information for users to understand these changes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster

[Zhiyuan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982066#comment-15982066
 ] 

Zhiyuan Yang commented on YARN-5007:


This change breaks Tez and possibly many other downstream projects. In 2.7.0, 
the only way to enable ATS is through ctor parameters; now the only way is 
configuration. How do you expect downstream projects to support both support 
both 2.7.0 and 3.0.0?

> Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
> ---
>
> Key: YARN-5007
> URL: https://issues.apache.org/jira/browse/YARN-5007
> Project: Hadoop YARN
>  Issue Type: Test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>  Labels: oct16-easy
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-5007.01.patch, YARN-5007.02.patch, 
> YARN-5007.03.patch
>
>
> MiniYarnCluster has a deprecated constructor which is called by the other 
> constructors and it causes javac warnings during the build.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6392) Add submit time to Application Summary log

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982007#comment-15982007
 ] 

Hudson commented on YARN-6392:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11626 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11626/])
YARN-6392. Add submit time to Application Summary log. (Zhihai Xu via (wangda: 
rev 2ba21d63767e11535d3210dc58a03b41e83df949)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java


> Add submit time to Application Summary log
> --
>
> Key: YARN-6392
> URL: https://issues.apache.org/jira/browse/YARN-6392
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6392.000.patch
>
>
> add submit time to Application Summary log, application submit time will be 
> passed to Application Master in env variable "APP_SUBMIT_TIME_ENV". It is a 
> very important parameter, So it will be useful to log it in Application 
> Summary.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6493) Print node partition in assignContainer logs

[Jonathan Hung (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981996#comment-15981996
 ] 

Jonathan Hung commented on YARN-6493:
-

I see...seems requested label is most useful, so that we can track demand per 
partition.

In this case the branch-2.8 patch needs to be changed. Uploaded branch-2.8.002 
for this. 

> Print node partition in assignContainer logs
> 
>
> Key: YARN-6493
> URL: https://issues.apache.org/jira/browse/YARN-6493
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6493.001.patch, YARN-6493.002.patch, 
> YARN-6493-branch-2.7.001.patch, YARN-6493-branch-2.8.001.patch, 
> YARN-6493-branch-2.8.002.patch
>
>
> It would be useful to have the node's partition when logging a container 
> allocation, for tracking purposes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6493) Print node partition in assignContainer logs

[Jonathan Hung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hung updated YARN-6493:

Attachment: YARN-6493-branch-2.8.002.patch

> Print node partition in assignContainer logs
> 
>
> Key: YARN-6493
> URL: https://issues.apache.org/jira/browse/YARN-6493
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6493.001.patch, YARN-6493.002.patch, 
> YARN-6493-branch-2.7.001.patch, YARN-6493-branch-2.8.001.patch, 
> YARN-6493-branch-2.8.002.patch
>
>
> It would be useful to have the node's partition when logging a container 
> allocation, for tracking purposes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981974#comment-15981974
 ] 

Wangda Tan commented on YARN-5892:
--

[~eepayne], [~jlowe], ideally MULP is guaranteed resource for users in the 
queue.

>From 
>https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
bq. Each queue enforces a limit on the percentage of resources allocated to a 
user at any given time, if there is demand for resources. The user limit can 
vary between a minimum and maximum value. The former (the minimum value) is set 
to this property value and the latter (the maximum value) depends on the number 
of users who have submitted applications. 

 In reality there could be more than 100/MULP users run their application in 
the queue, so some users cannot get the minimum resource. But to me it doesn't 
make sense to say a user's minimum resource can be more than queue's configured 
capacity. Previously we didn't enforce MULP <= 100, it was a mistake to me, we 
should fix it.

> Capacity Scheduler: Support user-specific minimum user limit percent
> 
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch, 
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch, 
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch, 
> YARN-5892.008.patch, YARN-5892.009.patch, YARN-5892.010.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} 
> property is per queue. A cluster admin should be able to set the minimum user 
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled 
> (YARN-4945 / YARN-2113), some users can be deemed as more important than 
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user 
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed 
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like 
> this:
> {code}
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>   
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>   
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6510) Fix warning - procfs stat file is not in the expected format: YARN-3344 is not enough

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981955#comment-15981955
 ] 

Haibo Chen commented on YARN-6510:
--

Thanks [~wilfreds] for the patch!  Can the process name be empty string? If 
not, I think we should probably do '.+' instead of '.*'

> Fix warning - procfs stat file is not in the expected format: YARN-3344 is 
> not enough
> -
>
> Key: YARN-6510
> URL: https://issues.apache.org/jira/browse/YARN-6510
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-6510.01.patch
>
>
> Even with the fix for YARN-3344 we still have issues with the procfs format.
> This is the case that is causing issues:
> {code}
> [user@nm1 ~]$ cat /proc/2406/stat
> 2406 (ib_fmr(mlx4_0)) S 2 0 0 0 -1 2149613632 0 0 0 0 166 126908 0 0 20 0 1 0 
> 4284 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 18446744073709551615 
> 0 0 17 6 0 0 0 0 0
> {code}
> We do not handle the parenthesis in the name which causes the pattern 
> matching to fail



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981950#comment-15981950
 ] 

Sean Busbey commented on YARN-6509:
---

Minor point of correction, 2^63 -1 is something like 8.3 million TiB.

I'd like it if we could find some operational users to weigh in on wether 10GiB 
is enough for a default. could we ask on the yarn user list?

> Add a size threshold beyond which yarn logs will require a force option
> ---
>
> Key: YARN-6509
> URL: https://issues.apache.org/jira/browse/YARN-6509
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-6509.1.patch
>
>
> An accidental fetch for a long running application can lead to scenario which 
> the large size of log can fill up a disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5949) Add pluggable configuration policy interface as a component of MutableCSConfigurationProvider

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981951#comment-15981951
 ] 

Wangda Tan commented on YARN-5949:
--

Thanks [~jhung] for your patch, overall looks good to me as well, few 
comments/suggestions:

- Not caused by this patch, there's some place of 
{{RMWebServices#updateSchedulerConfiguration}} hardcoded to use 
CapacityScheduler, could you update to MutableConfScheduler instead? Similarly, 
we should avoid directly call CapacityScheduler in 
removeQueue/addQueue/updateQueue as well. There're some cleanups can be done 
for emoveQueue/addQueue/updateQueue, there're some duplicated logics.
- QueueAdminConfigurationMutationPolicy use regex to parse the configs and get 
existing queue names, we should avoid doing this. YarnScheduler has a 
getQueueInfo API, by using that we can get all queues. 
- Also we should get existing queues when necessary, I think we can add an API 
to reinitialize ConfigurationMutationPolicy.
- Please make sure QueueAdminConfigurationMutationPolicy can handle invalid 
configs (like "x.y.z" and x is not a root queue)
- Is it possible to change a global config (such as 
{{yarn.scheduler.capacity.maximum-applications}}), if so, is following check 
enough?
{code}
  while (queue == null) {
// We are adding a queue (whose parent we are possibly also adding).
// Check ACL of lowest parent queue which already exists.
parentPath = parentPath.substring(0, parentPath.lastIndexOf('.'));
String parentName = parentPath.lastIndexOf('.') != -1 ?
parentPath.substring(parentPath.lastIndexOf('.') + 1) : parentPath;
queue = ((CapacityScheduler) rmContext.getScheduler())
.getQueue(parentName);
  }
  if (!queue.hasAccess(QueueACL.ADMINISTER_QUEUE, user)) {
return false;
  }
{code}
If a global config is updated by admin, should we request admin permission?

> Add pluggable configuration policy interface as a component of 
> MutableCSConfigurationProvider
> -
>
> Key: YARN-5949
> URL: https://issues.apache.org/jira/browse/YARN-5949
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-5949-YARN-5734.001.patch
>
>
> This will allow different policies to customize how/if configuration changes 
> should be applied (for example, a policy might restrict whether a 
> configuration change by a certain user is allowed). This will be enforced by 
> the MutableCSConfigurationProvider.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981939#comment-15981939
 ] 

Eric Payne commented on YARN-5892:
--

bq. I don't understand imposing a hard limit of weight < 100/MULP.
[~jlowe], this was in response to [~leftnoteasy]'s comment 
[here|https://issues.apache.org/jira/browse/YARN-5892?focusedCommentId=15951539=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15951539].
 I would be interested to hear his thoughts regarding this.

BTW, the findbugs issues from the last pre-commit build are all in code that 
was not changed by this patch.

> Capacity Scheduler: Support user-specific minimum user limit percent
> 
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch, 
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch, 
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch, 
> YARN-5892.008.patch, YARN-5892.009.patch, YARN-5892.010.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} 
> property is per queue. A cluster admin should be able to set the minimum user 
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled 
> (YARN-4945 / YARN-2113), some users can be deemed as more important than 
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user 
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed 
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like 
> this:
> {code}
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>   
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>   
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981933#comment-15981933
 ] 

Junping Du commented on YARN-6509:
--

bq. The prior released behavior is to read the entire log file. YARN-5088 added 
the option to limit that, but defaulted it to Long.MAX_VALUE. This now defaults 
the behavior to the last ~4KiB of logs.
I see. I agree with [~sseth] that we shouldn't break incompatibility here given 
previous default behavior (fetch entire log) is very useful and setting 
limitation is only addressing corner case.

bq. YARN-5088 added the cli option that allowed folks to ask for either the 
head or the tail of the logs. It defaulted to fetching Long.MAX_VALUE, or the 
first 2^63 -1 bytes.
Even 2^63 -1 (~8MB) is very small and may not be useful in production cluster.

IMO, the priority work here is to have an extra large value as default, 
something like 10 GB, to allow user to download. Any log cmd in previous usage 
(without specifying size and ignore_size_limit) beyond that size should fail 
fast and no need for partial download, so user can either to chose either 
specifying a limited size for partial log or to put "ignore_size_limits" to 
ignore size limit and fetch all (extra large) logs. Thoughts?

> Add a size threshold beyond which yarn logs will require a force option
> ---
>
> Key: YARN-6509
> URL: https://issues.apache.org/jira/browse/YARN-6509
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-6509.1.patch
>
>
> An accidental fetch for a long running application can lead to scenario which 
> the large size of log can fill up a disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6405) Improve configuring services through REST API

[Billie Rinaldi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981930#comment-15981930
 ] 

Billie Rinaldi commented on YARN-6405:
--

+1, I like the new approach. [~jianhe], this patch looks good to me. We will 
need to reintroduce DEFAULT_DATA_DIR, but can do that in another patch. It 
looks like the precommit build got lost, so I've kicked it off and will commit 
the patch after it finishes.

> Improve configuring services through REST API
> -
>
> Key: YARN-6405
> URL: https://issues.apache.org/jira/browse/YARN-6405
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: unique-names-test.patch, 
> YARN-6405.yarn-native-services.01.patch, 
> YARN-6405.yarn-native-services.02.patch, 
> YARN-6405.yarn-native-services.03.patch, 
> YARN-6405.yarn-native-services.04.patch, 
> YARN-6405.yarn-native-services.05.patch, 
> YARN-6405.yarn-native-services.06.patch, 
> YARN-6405.yarn-native-services.07.patch
>
>
> YARN-4793 defined various ways that user can config services through the 
> configuration section of the REST API.  But, some semantics are not yet 
> supported in the back-end server (AM).  YARN-6255 has done some work, this 
> jira is to complete this task -  support configuring services through REST API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6513) Fix for FindBugs getPendingLogFilesToUpload() possible NPE

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981855#comment-15981855
 ] 

Haibo Chen commented on YARN-6513:
--

Thanks [~wilfreds] for the patch. One nit on the patch, we pass 0 to the 
HashSet constructor given that we  don't reuse  it.

> Fix for FindBugs getPendingLogFilesToUpload() possible NPE
> --
>
> Key: YARN-6513
> URL: https://issues.apache.org/jira/browse/YARN-6513
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-6513.01.patch
>
>
> {code}
> Possible null pointer dereference in 
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue.getPendingLogFilesToUpload(File)
>  due to return value of called method
> Bug type NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE (click for details) 
> In class org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue
> In method 
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue.getPendingLogFilesToUpload(File)
> Local variable stored in JVM register ?
> Method invoked at AggregatedLogFormat.java:[line 314]
> Known null at AggregatedLogFormat.java:[line 314]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6306) NMClient API change for container upgrade

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-6306:
--
Description: This JIRA is track the addition of Upgrade API (Re-Initialize, 
Restart, Rollback and Commit) to the NMClient and NMClientAsync

> NMClient API change for container upgrade
> -
>
> Key: YARN-6306
> URL: https://issues.apache.org/jira/browse/YARN-6306
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Arun Suresh
> Attachments: YARN-6306.001.patch
>
>
> This JIRA is track the addition of Upgrade API (Re-Initialize, Restart, 
> Rollback and Commit) to the NMClient and NMClientAsync



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981830#comment-15981830
 ] 

Eric Payne commented on YARN-2113:
--

[~sunilg],
It looks like 
{{IntraQueueCandidatesSelector#initializeUsageAndUserLimitForCompute}} should 
be cloning the used resources from leafqueue:
{code}

   // Initialize used resource of a given user for rolling computation.
   rollingResourceUsagePerUser.put(user,
-  leafQueue.getUser(user).getResourceUsage().getUsed(partition));
+Resources.clone(
+  leafQueue.getUser(user).getResourceUsage().getUsed(partition)));
   if (LOG.isDebugEnabled()) {
 LOG.debug("Rolling resource usage for user:" + user + " is : "
 + rollingResourceUsagePerUser.get(user));
{code}


> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: 
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
>  YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, 
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, 
> YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, 
> YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5067) Support specifying resources for AM containers in SLS

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981811#comment-15981811
 ] 

Haibo Chen commented on YARN-5067:
--

[~yufeigu] Looks like the patch needs to be rebased now.

> Support specifying resources for AM containers in SLS
> -
>
> Key: YARN-5067
> URL: https://issues.apache.org/jira/browse/YARN-5067
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Yufei Gu
> Attachments: YARN-5067.001.patch
>
>
> Now resource of application masters in SLS is hardcoded to mem=1024 vcores=1.
> We should be able to specify AM resources from trace input file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6392) Add submit time to Application Summary log

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6392:
-
Summary: Add submit time to Application Summary log  (was: add submit time 
to Application Summary log)

> Add submit time to Application Summary log
> --
>
> Key: YARN-6392
> URL: https://issues.apache.org/jira/browse/YARN-6392
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: YARN-6392.000.patch
>
>
> add submit time to Application Summary log, application submit time will be 
> passed to Application Master in env variable "APP_SUBMIT_TIME_ENV". It is a 
> very important parameter, So it will be useful to log it in Application 
> Summary.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6493) Print node partition in assignContainer logs

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981808#comment-15981808
 ] 

Wangda Tan commented on YARN-6493:
--

[~jhung], not sure which partition you want to print, requested, or allocated, 
since non-exclusive node label could give you different partition other than 
you requested.

> Print node partition in assignContainer logs
> 
>
> Key: YARN-6493
> URL: https://issues.apache.org/jira/browse/YARN-6493
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6493.001.patch, YARN-6493.002.patch, 
> YARN-6493-branch-2.7.001.patch, YARN-6493-branch-2.8.001.patch
>
>
> It would be useful to have the node's partition when logging a container 
> allocation, for tracking purposes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6493) Print node partition in assignContainer logs

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981807#comment-15981807
 ] 

Wangda Tan commented on YARN-6493:
--

[~sunilg], no, it will give you the original requested node label expression. 
It will be useful depends on the use case.

> Print node partition in assignContainer logs
> 
>
> Key: YARN-6493
> URL: https://issues.apache.org/jira/browse/YARN-6493
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6493.001.patch, YARN-6493.002.patch, 
> YARN-6493-branch-2.7.001.patch, YARN-6493-branch-2.8.001.patch
>
>
> It would be useful to have the node's partition when logging a container 
> allocation, for tracking purposes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6291) Introduce query parameters (sort, filter, etc.) for tables to keep state on refresh/navigation in new YARN UI

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981799#comment-15981799
 ] 

Hudson commented on YARN-6291:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11625 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11625/])
YARN-6291. Introduce query parameters (sort, filter, etc.) for tables to 
(sunilg: rev 561718e05de51c0cb7c17295d7713d52408918eb)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-apps/services.hbs
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/services.js
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/apps.js
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-nodes/table.hbs
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-nodes/table.js
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-apps/apps.hbs


> Introduce query parameters (sort, filter, etc.) for tables to keep state on 
> refresh/navigation in new YARN UI
> -
>
> Key: YARN-6291
> URL: https://issues.apache.org/jira/browse/YARN-6291
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Novák
>Assignee: Gergely Novák
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6291.001.patch, YARN-6291.002.patch, 
> YARN-6291.003.patch
>
>
> On Applications / Nodes pages the user can define filters, sort the list on 
> selected columns, etc. But all these settings are lost on a refresh, or 
> navigation, which is very inconvenient.
> For example a possible use case might be watching the Applications table 
> while running a lot of YARN applications and continuously pressing Refresh to 
> see their progress. In this case, the user have to set any filter / sorting 
> manually after every single refresh. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5617) AMs only intended to run one attempt can be run more than once

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981800#comment-15981800
 ] 

Hudson commented on YARN-5617:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11625 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11625/])
YARN-5617. AMs only intended to run one attempt can be run more than (epayne: 
rev 52adf719143c20f4f2af369c6c40dd98677e7410)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java


> AMs only intended to run one attempt can be run more than once
> --
>
> Key: YARN-5617
> URL: https://issues.apache.org/jira/browse/YARN-5617
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.5.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-5617.001.patch, YARN-5617.002.patch, 
> YARN-5617.003.patch
>
>
> There are times when a user only wants to run an application with one 
> attempt.  Examples would be cases where the second AM attempt is not prepared 
> to handle recovery or will accidentally corrupt state (e.g.: by re-executing 
> something from scratch that should not be).  Prior to YARN-614 setting the 
> max attempts to 1 would guarantee the app ran at most one attempt, but now it 
> can run more than one attempt if the attempts fail due to a fault not 
> attributed to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981801#comment-15981801
 ] 

Hudson commented on YARN-6500:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11625 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11625/])
YARN-6500. Do not mount inaccessible cgroups directories in (haibochen: rev 
8ac50e1322cb3f84bd998635924d85846aa47c94)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java


> Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
> ---
>
> Key: YARN-6500
> URL: https://issues.apache.org/jira/browse/YARN-6500
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6500.000.patch, YARN-6500.001.patch, 
> YARN-6500.002.patch, YARN-6500.003.patch
>
>
> Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to 
> CgroupsLCEResourcesHandler



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981780#comment-15981780
 ] 

Eric Payne commented on YARN-2113:
--

[~sunilg], YARN-2113.0009.patch has a problem when {{preemption-order}} is 
{{USERLIMIT_FIRST}}.

Cluster looks like:
||*Queue Capacity*||*Queue Max Capacity*||*Minimum User Limit 
Percent*||*Default Container Size*||*Vcores Per Container*||
|10GB|20GB|25%|0.5GB|1|

 With one user taking up the whole queue and cluster, the {{Active Users Info}} 
section looks like this (which seems accurate except I would have expected the 
{{Max Resource}} to be 5120, but since it's not active, it probably doesn't 
matter):
||*User Name*||*Max Resource*||*Used Resource*||
|hadoop1|||

 When the second user starts, some containers get preempted, but preemption 
stops early and the {{Active Users Info}} section looks like this (with 
inaccurate values for {{hadoop1}}:
||*User Name*||*Max Resource*||*Used Resource*||
|hadoop2|||
|hadoop1|||

- The {{Used Resource}} for {{hadoop2}} is accurate, but {{Used Resource}} for 
{{hadoop1}} is incorrect. {{hadoop1}} is actually using 17.5GB, while the above 
says it is using 5GB.

Perhaps it's as a result of the following code?
{code:title=IntraQueueCandidatesSelector#preemptFromLeastStarvedApp}
  if (ret && preemptionContext.getIntraQueuePreemptionOrder()
  .equals(IntraQueuePreemptionOrder.USERLIMIT_FIRST)) {
Resources.subtractFrom(userResourceUsage, c.getAllocatedResource());
  }
{code}


> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: 
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
>  YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, 
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, 
> YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, 
> YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem

[Subru Krishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981760#comment-15981760
 ] 

Subru Krishnan commented on YARN-5331:
--

Thanks [~ajsangeetha] for working on this. I looked at the latest patch (v8) 
and have a few minor comments:
  * I feel it's better if {{PeriodicRLESparseResourceAllocation}} extends 
{{RLESparseResourceAllocation}} as it's a specialization and not composition.
  * Following the above comment, we should rename *getMaxPeriodicCapacity* to 
*getMaximumCapacityInInterval* and use the interval length for period 
implicitly (call out in Javadoc). This will also align with 
*getMinimumCapacityInInterval*.
  * To improve efficiency, the specified *timePeriod* should be the 
*maxTimePeriod*. This will allow us to auto extend period till we hit the max 
which I feel is reasonable as otherwise we'll be unbounded if user decides to 
input prime numbers as currently we'll be storing multiple instances if the 
user requested periods are small. For e.g: 24 instances of an hourly job if 
timePeriod is one day.

> Extend RLESparseResourceAllocation with period for supporting recurring 
> reservations in YARN ReservationSystem
> --
>
> Key: YARN-5331
> URL: https://issues.apache.org/jira/browse/YARN-5331
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Sangeetha Abdu Jyothi
>  Labels: oct16-medium
> Attachments: YARN-5331.001.patch, YARN-5331.002.patch, 
> YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch, 
> YARN-5331.006.patch, YARN-5331.007.patch, YARN-5331.008.patch
>
>
> YARN-5326 proposes adding native support for recurring reservations in the 
> YARN ReservationSystem. This JIRA is a sub-task to add a 
> PeriodicRLESparseResourceAllocation. Please refer to the design doc in the 
> parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981731#comment-15981731
 ] 

Sean Busbey commented on YARN-6509:
---

YARN-5088 added the cli option that allowed folks to ask for either the head or 
the tail of the logs. It defaulted to fetching Long.MAX_VALUE, or the first 
2^63 -1 bytes.

This JIRA adds an additional flag that says "ignore any size limits and give me 
the full logs". It also changes the default fetch to "the last 4KiB bytes", so 
if you want more you have to either start using the option added in YARN-5088 
or the new flag.

> Add a size threshold beyond which yarn logs will require a force option
> ---
>
> Key: YARN-6509
> URL: https://issues.apache.org/jira/browse/YARN-6509
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-6509.1.patch
>
>
> An accidental fetch for a long running application can lead to scenario which 
> the large size of log can fill up a disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6306) NMClient API change for container upgrade

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-6306:
--
Attachment: YARN-6306.001.patch

Attaching initial patch. Will add some tests shortly..

[~jianhe], For the NMAsyncClient, I introduced a new handler, which extends the 
current one. This is so that existing clients can continue to use the old 
handler without changing code.

> NMClient API change for container upgrade
> -
>
> Key: YARN-6306
> URL: https://issues.apache.org/jira/browse/YARN-6306
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Arun Suresh
> Attachments: YARN-6306.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4166) Support changing container cpu resource

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981709#comment-15981709
 ] 

Naganarasimha G R commented on YARN-4166:
-

Thanks for the patch [~fly_in_gis], Overall approach seems to be fine and 
almost similar as what i intended to do, assigning this issue to you and will 
further help in getting it committed. Yes you were right that the findbugs 
issue is not related to the patch and have raised a separate issue(YARN-6515) 
for the same.

Few comments on the patch:
# ContainerExecutor ln no 183: {{updateContainerResource}} can be abstract like 
others and need not throw ResourceHandlerException.
# ContainerExecutor ln no 183: {{ResourceHandlerException}} requires to be 
thrown for any other case ? May be captured in the javadocs on what scenarios 
this exception will be thrown..
# ContainerManagerImpl ln no 1238 : We need to invoke 
{{ContainerExecutor().updateContainerResource}}, this will ensure that on 
recovery proper resource information are assigned to containers else there is 
possibility that the resource update fails before cgroups modifications are 
done but on restart NM will report higher/lower cpu ...
# CGroupsCpuResourceHandlerImpl ln no 246, invoking 
{{cGroupsHandler.deleteCGroup}} in setupLimits makes sense for the creating new 
container, but on update if there is excpetion just deleting the cgroup is not 
ideal and as well we might require to propagate the update failure to RM or 
fail the container, need to see how its handled (may be this error handling if 
not done can be done in other).
# ResourceHandler ln no 72: IIUC no need to return List 
as we are not taking any decision based on it. Here we just send null in most 
of the flows. IIUC PrivilegedOperation is used to squash such operations 
related optimization of container launch. And currently they support only for 
ADD_PID_TO_CGROUP, hence we can have the signature as void. And if required we 
can implement the stack properly when required.
# May be once failure to update scenario handling is finalized we might need to 
add test case for the same. 
# can you please check whether the checkstyle is applicable to the patch ?

> Support changing container cpu resource
> ---
>
> Key: YARN-4166
> URL: https://issues.apache.org/jira/browse/YARN-4166
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, nodemanager, resourcemanager
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Jian He
>Assignee: Yang Wang
> Attachments: YARN-4166.001.patch, YARN-4166.002.patch, 
> YARN-4166-branch2.8-001.patch
>
>
> Memory resizing is now supported, we need to support the same for cpu.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981703#comment-15981703
 ] 

Siddharth Seth commented on YARN-6509:
--

Is the current proposal to change the default to fetch the last 4K ?
Can we please not make this change. It is definitely incompatible, and I'd 
argue that it's not very useful.

The intent of the jira is to protect users against log downloads which could 
otherwise take hours and fill up the local fs - apps which generate large logs.

> Add a size threshold beyond which yarn logs will require a force option
> ---
>
> Key: YARN-6509
> URL: https://issues.apache.org/jira/browse/YARN-6509
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-6509.1.patch
>
>
> An accidental fetch for a long running application can lead to scenario which 
> the large size of log can fill up a disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-4166) Support changing container cpu resource

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-4166:
---

Assignee: Yang Wang  (was: Naganarasimha G R)

> Support changing container cpu resource
> ---
>
> Key: YARN-4166
> URL: https://issues.apache.org/jira/browse/YARN-4166
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api, nodemanager, resourcemanager
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Jian He
>Assignee: Yang Wang
> Attachments: YARN-4166.001.patch, YARN-4166.002.patch, 
> YARN-4166-branch2.8-001.patch
>
>
> Memory resizing is now supported, we need to support the same for cpu.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6522) Container host should be optional in SLS JSON input file format

[Yufei Gu (JIRA)]

Yufei Gu created YARN-6522:
--

 Summary: Container host should be optional in SLS JSON input file 
format
 Key: YARN-6522
 URL: https://issues.apache.org/jira/browse/YARN-6522
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: scheduler-load-simulator
Reporter: Yufei Gu
Assignee: Yufei Gu


Container host is useful for locality testing. It is obnoxious to specify 
container host for each task for tests unrelated to locality. We would like to 
make it optional.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981666#comment-15981666
 ] 

Rushabh S Shah commented on YARN-6516:
--

[~zhouyunfan]: do you mind changing your visible name ?

> FairScheduler:the algorithm of assignContainer is so slow for it only can 
> assign a thousand containers per second
> -
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: GirlKiller
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6520) Fix warnings from Spotbugs in hadoop-yarn-client

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981653#comment-15981653
 ] 

Naganarasimha G R commented on YARN-6520:
-

thanks for the patch [~cheersyang],
Modifications are fine with me and will wait for the jenkins build to pass then 
will commit it.


> Fix warnings from Spotbugs in hadoop-yarn-client
> 
>
> Key: YARN-6520
> URL: https://issues.apache.org/jira/browse/YARN-6520
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6520.001.patch
>
>
> There 2 findbugs warning in hadoop-yarn-client since switched to spotbugs
> # Possible exposure of partially initialized object in 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken()
> # Useless condition: it's known that isAppFinished == false at this point
> See more info from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981651#comment-15981651
 ] 

Haibo Chen commented on YARN-6500:
--

Thanks [~miklos.szeg...@cloudera.com] for your contribution, [~templedf] and 
[~rkanter] for additional reviews! I committed the patch to trunk and branch-2.

> Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
> ---
>
> Key: YARN-6500
> URL: https://issues.apache.org/jira/browse/YARN-6500
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6500.000.patch, YARN-6500.001.patch, 
> YARN-6500.002.patch, YARN-6500.003.patch
>
>
> Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to 
> CgroupsLCEResourcesHandler



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5949) Add pluggable configuration policy interface as a component of MutableCSConfigurationProvider

[Jonathan Hung (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981639#comment-15981639
 ] 

Jonathan Hung commented on YARN-5949:
-

Thanks [~xgong] for the review.

Right now the default policy is DefaultConfigurationMutationPolicy. I was 
thinking the policy would be enforced in MutableCSConfigurationProvider instead 
of the REST API, so if they want to use the queue admin-based policy they can 
just configure {{yarn.scheduler.configuration.mutation.policy.class}} to 
QueueAdminConfigurationMutationPolicy. 

> Add pluggable configuration policy interface as a component of 
> MutableCSConfigurationProvider
> -
>
> Key: YARN-5949
> URL: https://issues.apache.org/jira/browse/YARN-5949
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-5949-YARN-5734.001.patch
>
>
> This will allow different policies to customize how/if configuration changes 
> should be applied (for example, a policy might restrict whether a 
> configuration change by a certain user is allowed). This will be enforced by 
> the MutableCSConfigurationProvider.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6445) Improve YARN-3926 performance with respect to SLS

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981635#comment-15981635
 ] 

Wangda Tan commented on YARN-6445:
--

Latest patch looks good to me, +1. Thanks [~vvasudev] and reviews from 
[~sunilg]/[~dan...@cloudera.com], looking forward a SLS perf report of the 
resource profile includes this improvement.

> Improve YARN-3926 performance with respect to SLS
> -
>
> Key: YARN-6445
> URL: https://issues.apache.org/jira/browse/YARN-6445
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: YARN-6445-YARN-3926.001.patch, 
> YARN-6445-YARN-3926.002.patch, YARN-6445-YARN-3926.003.patch, 
> YARN-6445-YARN-3926.004.patch
>
>
> As part of the SLS runs on YARN-3926, we discovered a bunch of bottlenecks 
> around object creation and garbage collection. This JIRA is to apply a fix 
> for those bottlenecks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981633#comment-15981633
 ] 

Naganarasimha G R commented on YARN-6519:
-


Hi [~cheersyang], 
few nits in the patch as follows :
# NodeType ln no 29, no need to change the access specifier of the constructor
# ApplicationMasterService ln no 396, {{EMPTY_CONTAINER_LIST}} can be private 
and if required made default access in future
#AbstractYarnScheduler ln no 135, {{EMPTY_CONTAINER_LIST}} as above can be 
private 
# QueueMetrics ln no 151, i was of the opinion that we use naming convention of 
field as its not exactly a string constant, thoughts?
# ProportionalCapacityPreemptionPolicy ln no 311-315, as we are modifying by 
invoking {{preemptionCandidates.remove}} within the iterator whether is it 
possible for concurrent modification exception ?
# FSSchedulerNode ln no 157-165, i presume for removal we better not use entry 
set, as there is Concurrent Modification Exception possible
{code}

public static void main(String args[]){
  Map testMap = new HashMap<>();
  testMap.put("1", "one");
  testMap.put("2", "two");
  testMap.put("3", "three");
  testMap.put("4", "four");
  
  for (Entry entry : testMap.entrySet()) {
if (entry.getKey().equals("2")){
  testMap.remove(entry.getKey());
}
  }
}
{code}

> Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager
> 
>
> Key: YARN-6519
> URL: https://issues.apache.org/jira/browse/YARN-6519
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6519.001.patch
>
>
> There is 8 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager$1.compare(CSQueue,
>  CSQueue) incorrectly handles float value
> # org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType.index 
> field is public and mutable
> # 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.queueMetrics
>  is a mutable collection
> # 
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.cleanupStaledPreemptionCandidates(long)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.transferStateFromAttempt(RMAppAttempt)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.cleanupPreemptionList()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> See more from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981583#comment-15981583
 ] 

Junping Du commented on YARN-679:
-

Sorry for coming late on this. The latest patch looks OK to me in general. From 
Steve, this effort has been pending for years and he has been using it in 
Slider for a long while. Why not we just go ahead to commit it and address 
small issues/enhancements in separated JIRAs?

> add an entry point that can start any Yarn service
> --
>
> Key: YARN-679
> URL: https://issues.apache.org/jira/browse/YARN-679
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: api
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: org.apache.hadoop.servic...mon 3.0.0-SNAPSHOT API).pdf, 
> YARN-679-001.patch, YARN-679-002.patch, YARN-679-002.patch, 
> YARN-679-003.patch, YARN-679-004.patch, YARN-679-005.patch, 
> YARN-679-006.patch, YARN-679-007.patch, YARN-679-008.patch, 
> YARN-679-009.patch, YARN-679-010.patch, YARN-679-011.patch, YARN-679-013.patch
>
>  Time Spent: 72h
>  Remaining Estimate: 0h
>
> There's no need to write separate .main classes for every Yarn service, given 
> that the startup mechanism should be identical: create, init, start, wait for 
> stopped -with an interrupt handler to trigger a clean shutdown on a control-c 
> interrupt.
> Provide one that takes any classname, and a list of config files/options



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981575#comment-15981575
 ] 

Sean Busbey commented on YARN-6509:
---

That's not correct. The prior released behavior is to read the entire log file. 
YARN-5088 added the option to limit that, but defaulted it to Long.MAX_VALUE. 
This now defaults the behavior to the last ~4KiB of logs. I suppose one could 
argue about wether or not getting the last Long.MAX_VALUE worth of logs instead 
of all of them should be considered an incompatible change. For just about 
everyone I'm guessing the two conditions are equivalent. The number of folks 
impacted by changing to 4KiB is much larger.

> Add a size threshold beyond which yarn logs will require a force option
> ---
>
> Key: YARN-6509
> URL: https://issues.apache.org/jira/browse/YARN-6509
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-6509.1.patch
>
>
> An accidental fetch for a long running application can lead to scenario which 
> the large size of log can fill up a disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3053) [Security] Review and implement authentication in ATS v.2

[Varun Saxena (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981565#comment-15981565
 ] 

Varun Saxena commented on YARN-3053:


Thanks [~rkanter] for your comments. Sorry was on leave so could not reply.

The reason I had chosen approach 1 is due to minimum amount of change required 
for it. We already have client and server side filter code for ATSv1 which 
could be reused with approach 1. Also the con I pointed out for approach 1 i.e. 
AM having to get the token from Allocate Response, I thought would be fine 
because AM would anyways have to change to publish entities to ATSv2 as the 
APIs' are new. 

With approach 2, we would have to still pass on the token from RM-> NM-> 
Collector as in the end entities would be directly published by AM to 
Collector. This would mean introduction of a new message in Collector Manager 
protocol.
The design for offline collectors is not yet decided but in future, we would 
probably let clients ask for token directly from Collector as well. The issue I 
pointed out with clash of IDs' would mean that we would have to probably 
differentiate between token generated by collector itself and one generated by 
RM. Probably differentiate on the basis of token kind. This, however doable, 
would mean additional changes at both the client and server side.
Moreover, we would need to also store the token in a state store even for 
managed apps to ensure app token is available across collector restarts.

Do you see any major issues with approach 1?

> [Security] Review and implement authentication in ATS v.2
> -
>
> Key: YARN-3053
> URL: https://issues.apache.org/jira/browse/YARN-3053
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>  Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: ATSv2Authentication(draft).pdf, 
> ATSv2Authentication.v01.pdf
>
>
> Per design in YARN-2928, we want to evaluate and review the system for 
> security, and ensure proper security in the system.
> This includes proper authentication, token management, access control, and 
> any other relevant security aspects.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6202) Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981561#comment-15981561
 ] 

Junping Du commented on YARN-6202:
--

Agree. An example is: TEZ-2049. Let's keep the patch in trunk only.

> Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded
> -
>
> Key: YARN-6202
> URL: https://issues.apache.org/jira/browse/YARN-6202
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6202.001.patch, YARN-6202.002.patch, 
> YARN-6202.003.patch, YARN-6202.004.patch
>
>
> Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY (yarn.dispatcher.exit-on-error) 
> always be true no matter what value in configuration files. This misleads 
> users. Two solutions: 
> # Remove the configuration item and provide a method to allow 
> {{exitOnDispatchException}}/{{shouldExitOnError}} to be false to enable 
> related unit tests. There is no need for false value in a real daemon since 
> daemons should crash if its dispatcher quit.
> # Make it default true instead of false, so that we don't need to hard code 
> it to be true in RM and NM, it is still configurable, and also provide method 
> to enable related unit tests.
> Other than that, the code around it needs to refactor. {{public static 
> final}} for a variable of interface isn't necessary, and YARN related 
> configure item should be in class YarnConfiguration. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6475) Fix some long function checkstyle issues

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981547#comment-15981547
 ] 

Miklos Szegedi commented on YARN-6475:
--

Thank you, [~soumabrata]. I think you can address them as this jira.

> Fix some long function checkstyle issues
> 
>
> Key: YARN-6475
> URL: https://issues.apache.org/jira/browse/YARN-6475
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Soumabrata Chakraborty
>Priority: Trivial
>  Labels: newbie
>
> I am talking about these two:
> {code}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java:441:
>   @Override:3: Method length is 176 lines (max allowed is 150). [MethodLength]
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java:159:
>   @Override:3: Method length is 158 lines (max allowed is 150). [MethodLength]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5617) AMs only intended to run one attempt can be run more than once

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981542#comment-15981542
 ] 

Eric Payne commented on YARN-5617:
--

Thanks, [~jlowe]. I committed this patch to trunk and branch 2. However, it 
does not cherry-pick cleanly back to branch-2.8. If this patch should go back 
into branch-2.8, please proved a patch.

> AMs only intended to run one attempt can be run more than once
> --
>
> Key: YARN-5617
> URL: https://issues.apache.org/jira/browse/YARN-5617
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.5.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-5617.001.patch, YARN-5617.002.patch, 
> YARN-5617.003.patch
>
>
> There are times when a user only wants to run an application with one 
> attempt.  Examples would be cases where the second AM attempt is not prepared 
> to handle recovery or will accidentally corrupt state (e.g.: by re-executing 
> something from scratch that should not be).  Prior to YARN-614 setting the 
> max attempts to 1 would guarantee the app ran at most one attempt, but now it 
> can run more than one attempt if the attempts fail due to a fault not 
> attributed to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3742) YARN RM will shut down if ZKClient creation times out

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981514#comment-15981514
 ] 

Daniel Templeton commented on YARN-3742:


{quote}When creating RMFatalEvent, start the message with capital letters{quote}

I did that on purpose because the {{RMFatalEvent}} will prepend the message 
with "RMFatalEvent of type %s, caused by %s".  There were some cases where the 
message was logged in addition to being passed to the event constructor, and in 
those cases I started with a capital.

{quote}Would "Critical thread, %s, exited unexpectedly" be a grammatically 
better message?{quote}

Again, prepend "RMFatalEvent of type %s, caused by %s" => "RMFatalEvent of type 
%s, caused by a critical thread, %s, that exited unexpectedly".

> YARN RM  will shut down if ZKClient creation times out 
> ---
>
> Key: YARN-3742
> URL: https://issues.apache.org/jira/browse/YARN-3742
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Daniel Templeton
> Attachments: YARN-3742.001.patch, YARN-3742.002.patch
>
>
> The RM goes down showing the following stacktrace if the ZK client connection 
> fails to be created. We should not exit but transition to StandBy and stop 
> doing things and let the other RM take over.
> {code}
> 2015-04-19 01:22:20,513  FATAL 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a 
> org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type 
> STATE_STORE_OP_FAILED. Cause:
> java.io.IOException: Wait for ZKClient creation timed out
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1066)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1090)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:996)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationStateInternal(ZKRMStateStore.java:643)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore.java:162)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore.java:147)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>   at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:806)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:879)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:874)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6516:
---
Priority: Major  (was: Blocker)

> FairScheduler:the algorithm of assignContainer is so slow for it only can 
> assign a thousand containers per second
> -
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: GirlKiller
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981478#comment-15981478
 ] 

Yufei Gu commented on YARN-6516:


Can you provide more details about test environment? Is this tested in a real 
cluster? How many nodes are there? Was continuous scheduling on? What exact 
version did you test?

Besides, I don't think it is blocker. 

> FairScheduler:the algorithm of assignContainer is so slow for it only can 
> assign a thousand containers per second
> -
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: GirlKiller
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6164) Expose Queue Configurations per Node Label through YARN client api

[Benson Qiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981416#comment-15981416
 ] 

Benson Qiu commented on YARN-6164:
--

Thanks [~sunilg] for committing this patch! Thanks [~bibinchundatt], 
[~leftnoteasy], [~rohithsharma], [~churromorales]!

> Expose Queue Configurations per Node Label through YARN client api
> --
>
> Key: YARN-6164
> URL: https://issues.apache.org/jira/browse/YARN-6164
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Benson Qiu
>Assignee: Benson Qiu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6164.001.patch, YARN-6164.002.patch, 
> YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch, 
> YARN-6164.006.patch, YARN-6164.007.patch, YARN-6164.008.patch, 
> YARN-6164-branch-2.001.patch
>
>
> `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the 
> [Cluster Scheduler 
> API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API],
>  but not through 
> [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html].
> Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by 
> default), it would be nice to expose `maximum-am-resource-percent` in 
> YarnClient as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6518) Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981358#comment-15981358
 ] 

Naganarasimha G R commented on YARN-6518:
-

Thanks  [~cheersyang] for the patch,
Simple modifications and Overall changes looks good enough to go in. Will 
wait for the jenkins report before committing.

> Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice
> 
>
> Key: YARN-6518
> URL: https://issues.apache.org/jira/browse/YARN-6518
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6518.001.patch
>
>
> There is 1 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntities(File,
>  String, TimelineEntityFilters, TimelineDataToRetrieve) due to return value 
> of called method
> See more in 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6517) Fix warnings from Spotbugs in hadoop-yarn-common

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981354#comment-15981354
 ] 

Naganarasimha G R commented on YARN-6517:
-

Thanks  [~cheersyang] for the patch,
 Overall changes looks good enough to go in. Will wait for the jenkins 
report before committing.

> Fix warnings from Spotbugs in hadoop-yarn-common
> 
>
> Key: YARN-6517
> URL: https://issues.apache.org/jira/browse/YARN-6517
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6517.001.patch
>
>
> There are 2 findbugs warnings in hadoop-yarn-common project since switched to 
> spotbugs,
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue.getPendingLogFilesToUpload(File)
>  due to return value of called method
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessList() due to 
> return value of called method
> see more in 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981332#comment-15981332
 ] 

Steve Loughran commented on YARN-679:
-

[~templedf]

bq. Signalled" is misspelled several times. Should be "signaled"

no, Signalled is spelt correctly. However, to avoid confusing people in the 
EN_US locale who contain invalid assumptions about which locale of the english 
language SHALL be considered normative, I shall change it.




> add an entry point that can start any Yarn service
> --
>
> Key: YARN-679
> URL: https://issues.apache.org/jira/browse/YARN-679
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: api
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: org.apache.hadoop.servic...mon 3.0.0-SNAPSHOT API).pdf, 
> YARN-679-001.patch, YARN-679-002.patch, YARN-679-002.patch, 
> YARN-679-003.patch, YARN-679-004.patch, YARN-679-005.patch, 
> YARN-679-006.patch, YARN-679-007.patch, YARN-679-008.patch, 
> YARN-679-009.patch, YARN-679-010.patch, YARN-679-011.patch, YARN-679-013.patch
>
>  Time Spent: 72h
>  Remaining Estimate: 0h
>
> There's no need to write separate .main classes for every Yarn service, given 
> that the startup mechanism should be identical: create, init, start, wait for 
> stopped -with an interrupt handler to trigger a clean shutdown on a control-c 
> interrupt.
> Provide one that takes any classname, and a list of config files/options



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers

[luhuichun (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981311#comment-15981311
 ] 

luhuichun commented on YARN-4266:
-

[~ebadger] Hi Eric, sorry for late response, we are working on another JIRA 
last week,  here is the first patch

> Allow whitelisted users to disable user re-mapping/squashing when launching 
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, 
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify 
> the user the container processes should run as. We use this mechanism today 
> when launching docker containers . In non-secure mode, we run the docker 
> container based on 
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in 
> secure mode, as the submitting user. However, this mechanism breaks down with 
> a large number of 'pre-created' images which don't necessarily have the users 
> available within the image. Examples of such images include shared images 
> that need to be used by multiple users. We need a way in which we can allow a 
> pre-defined set of users to run containers based on existing images, without 
> using the --user switch. There are some implications of disabling this user 
> squashing that we'll need to work through : log aggregation, artifact 
> deletion etc.,



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers

[luhuichun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

luhuichun reassigned YARN-4266:
---

Assignee: luhuichun  (was: Zhankun Tang)

> Allow whitelisted users to disable user re-mapping/squashing when launching 
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, 
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify 
> the user the container processes should run as. We use this mechanism today 
> when launching docker containers . In non-secure mode, we run the docker 
> container based on 
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in 
> secure mode, as the submitting user. However, this mechanism breaks down with 
> a large number of 'pre-created' images which don't necessarily have the users 
> available within the image. Examples of such images include shared images 
> that need to be used by multiple users. We need a way in which we can allow a 
> pre-defined set of users to run containers based on existing images, without 
> using the --user switch. There are some implications of disabling this user 
> squashing that we'll need to work through : log aggregation, artifact 
> deletion etc.,



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6419) Support to launch native-service deployment from new YARN UI

[Akhil PB (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6419:
---
Attachment: YARN-6419-yarn-native-services.001.patch

Latest YARN-6419 patch rebased based on {{yarn-native-services}} branch and 
includes fixes for comments from [~sunilg]

> Support to launch native-service deployment from new YARN UI
> 
>
> Key: YARN-6419
> URL: https://issues.apache.org/jira/browse/YARN-6419
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6419.001.patch, YARN-6419.002.patch, 
> YARN-6419.003.patch, YARN-6419.004.patch, 
> YARN-6419-yarn-native-services.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers

[luhuichun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

luhuichun updated YARN-4266:

Attachment: YARN-4266.001.patch

update

> Allow whitelisted users to disable user re-mapping/squashing when launching 
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, 
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, 
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify 
> the user the container processes should run as. We use this mechanism today 
> when launching docker containers . In non-secure mode, we run the docker 
> container based on 
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in 
> secure mode, as the submitting user. However, this mechanism breaks down with 
> a large number of 'pre-created' images which don't necessarily have the users 
> available within the image. Examples of such images include shared images 
> that need to be used by multiple users. We need a way in which we can allow a 
> pre-defined set of users to run containers based on existing images, without 
> using the --user switch. There are some implications of disabling this user 
> squashing that we'll need to work through : log aggregation, artifact 
> deletion etc.,



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981164#comment-15981164
 ] 

Jason Lowe commented on YARN-5892:
--

I don't understand imposing a hard limit of weight < 100/MULP.  For example, 
take a queue that has a max capacity far larger than normal capacity and the 
MULP is 100%.  With that hard limit, nobody can have a weight > 1.  However two 
or three separate users can all fit in that queue with their minimum limit due 
to queue growth beyond normal capacity, but we're not going to let a single, 
weighted user do the same?  I don't see why there needs to be an upper limit on 
the weight -- it's the same as that many separate users showing up.  At some 
point, yes, the weight could be high enough that the queue could never meet the 
minimum limit of that user, but the same occurs when that many users show up at 
the same time as well.  Some users do not get their minimum because there's too 
much demand.


> Capacity Scheduler: Support user-specific minimum user limit percent
> 
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch, 
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch, 
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch, 
> YARN-5892.008.patch, YARN-5892.009.patch, YARN-5892.010.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}} 
> property is per queue. A cluster admin should be able to set the minimum user 
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled 
> (YARN-4945 / YARN-2113), some users can be deemed as more important than 
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user 
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed 
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like 
> this:
> {code}
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>   
>   
> 
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>   
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6398) Implement a new native-service UI

[Akhil PB (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6398:
---
Attachment: YARN-6398.003.patch

v3 patch: handling scenarios where timeline server gives empty payload

> Implement a new native-service UI
> -
>
> Key: YARN-6398
> URL: https://issues.apache.org/jira/browse/YARN-6398
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Sunil G
>Assignee: Akhil PB
> Attachments: YARN-6398.001.patch, YARN-6398.002.patch, 
> YARN-6398.003.patch
>
>
> Create a new and advanced native service UI which can co-exist with the new 
> Yarn UI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980954#comment-15980954
 ] 

Rohith Sharma K S commented on YARN-1593:
-

Thanks [~vvasudev] for detailed doc and thanks to other folks for design 
discussions. 

The doc talks about both system container and system services. IIUC, these 2 
use cases and scopes are different.  System_services are admin configuration 
and managed via native service layer along with AppMaster. And 
system_containers are special containers requested by individual NodeManager to 
replace auxiliary services. These are independent containers  which do not need 
app-master.

Could we separate system_containers and system_services into separate JIRA? 
P.S : For flow-collector launching we are looking for system services so that 
any arbitrary client should be able to publish data into ATSv2. 

> support out-of-proc AuxiliaryServices
> -
>
> Key: YARN-1593
> URL: https://issues.apache.org/jira/browse/YARN-1593
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager, rolling upgrade
>Reporter: Ming Ma
>Assignee: Varun Vasudev
> Attachments: SystemContainersandSystemServices.pdf
>
>
> AuxiliaryServices such as ShuffleHandler currently run in the same process as 
> NM. There are some benefits to host them in dedicated processes.
> 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the 
> ShuffleHandler restart. If ShuffleHandler runs as a separate process, 
> ShuffleHandler can continue to run during NM restart. NM can reconnect the 
> the running ShuffleHandler after restart.
> 2. Resource management. It is possible another type of AuxiliaryServices will 
> be implemented. AuxiliaryServices are considered YARN application specific 
> and could consume lots of resources. Running AuxiliaryServices in separate 
> processes allow easier resource management. NM could potentially stop a 
> specific AuxiliaryServices process from running if it consumes resource way 
> above its allocation.
> Here are some high level ideas:
> 1. NM provides a hosting process for each AuxiliaryService. Existing 
> AuxiliaryService API doesn't change.
> 2. The hosting process provides RPC server for AuxiliaryService proxy object 
> inside NM to connect to.
> 3. When we rolling restart NM, the existing AuxiliaryService processes will 
> continue to run. NM could reconnect to the running AuxiliaryService processes 
> upon restart.
> 4. Policy and resource management of AuxiliaryServices. So far we don't have 
> immediate need for this. AuxiliaryService could run inside a container and 
> its resource utilization could be taken into account by RM and RM could 
> consider a specific type of applications overutilize cluster resource.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6493) Print node partition in assignContainer logs

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980951#comment-15980951
 ] 

Sunil G commented on YARN-6493:
---

[~leftnoteasy]A quick question. For non-exclusive labels, 
{{RMContainer.getNodeLabelExpression}} will give correct label to which 
container is allocated, correct?

> Print node partition in assignContainer logs
> 
>
> Key: YARN-6493
> URL: https://issues.apache.org/jira/browse/YARN-6493
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-6493.001.patch, YARN-6493.002.patch, 
> YARN-6493-branch-2.7.001.patch, YARN-6493-branch-2.8.001.patch
>
>
> It would be useful to have the node's partition when logging a container 
> allocation, for tracking purposes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6419) Support to launch native-service deployment from new YARN UI

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980888#comment-15980888
 ] 

Sunil G edited comment on YARN-6419 at 4/24/17 8:58 AM:


Few major comments:
# Rebase this patch on top of to "yarn-native-services" branch
# Add "New Service" button in Services home page.
# Remove Action column name in NewService wizard.


was (Author: sunilg):
Few major comments:
# Rebase this patch on top of to "yarn-native-services" branch
# Add "New Service" button in Services home page.

> Support to launch native-service deployment from new YARN UI
> 
>
> Key: YARN-6419
> URL: https://issues.apache.org/jira/browse/YARN-6419
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6419.001.patch, YARN-6419.002.patch, 
> YARN-6419.003.patch, YARN-6419.004.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6419) Support to launch native-service deployment from new YARN UI

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980888#comment-15980888
 ] 

Sunil G commented on YARN-6419:
---

Few major comments:
# Rebase this patch on top of to "yarn-native-services" branch
# Add "New Service" button in Services home page.

> Support to launch native-service deployment from new YARN UI
> 
>
> Key: YARN-6419
> URL: https://issues.apache.org/jira/browse/YARN-6419
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
> Attachments: YARN-6419.001.patch, YARN-6419.002.patch, 
> YARN-6419.003.patch, YARN-6419.004.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6521) Yarn Log-aggregation Transform Enable not to Spam the NameNode

[zhangyubiao (JIRA)]

zhangyubiao created YARN-6521:
-

 Summary: Yarn Log-aggregation Transform Enable not to Spam the 
NameNode
 Key: YARN-6521
 URL: https://issues.apache.org/jira/browse/YARN-6521
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: zhangyubiao


Nowdays We have large cluster,with the apps incr and container incr ,We split 
an  namespace to store applogs. But the  log-aggregation /tmp/app-logs incr 
also make the namespace responce slow.  
We want to chang yarn.log-aggregation-enable true -> false,But Transform the
 the yarn log cli service also can get the app-logs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6521) Yarn Log-aggregation Transform Enable not to Spam the NameNode

[zhangyubiao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated YARN-6521:
--
Description: 
Nowdays We have large cluster,with the apps incr and container incr ,We split 
an  namespace to store applogs. 
But the  log-aggregation /tmp/app-logs incr also make the namespace responce 
slow.  
We want to chang yarn.log-aggregation-enable true -> false,But Transform the 
the yarn log cli service also can get the app-logs.

  was:
Nowdays We have large cluster,with the apps incr and container incr ,We split 
an  namespace to store applogs. But the  log-aggregation /tmp/app-logs incr 
also make the namespace responce slow.  
We want to chang yarn.log-aggregation-enable true -> false,But Transform the
 the yarn log cli service also can get the app-logs.


> Yarn Log-aggregation Transform Enable not to Spam the NameNode
> --
>
> Key: YARN-6521
> URL: https://issues.apache.org/jira/browse/YARN-6521
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: zhangyubiao
>
> Nowdays We have large cluster,with the apps incr and container incr ,We split 
> an  namespace to store applogs. 
> But the  log-aggregation /tmp/app-logs incr also make the namespace responce 
> slow.  
> We want to chang yarn.log-aggregation-enable true -> false,But Transform the 
> the yarn log cli service also can get the app-logs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6398) Implement a new native-service UI

[Akhil PB (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6398:
---
Attachment: YARN-6398.002.patch

v2 patch

> Implement a new native-service UI
> -
>
> Key: YARN-6398
> URL: https://issues.apache.org/jira/browse/YARN-6398
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Sunil G
>Assignee: Akhil PB
> Attachments: YARN-6398.001.patch, YARN-6398.002.patch
>
>
> Create a new and advanced native service UI which can co-exist with the new 
> Yarn UI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6514) Fail to launch container when distributed scheduling is enabled

[Zheng Lv (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980878#comment-15980878
 ] 

Zheng Lv commented on YARN-6514:


[~asuresh], I set this option on NM nodes, and keep {{yarn-site.xml}} on the RM 
machine unchanged. The same error occurred.

The last INFO before the AMRMToken error is 
{{org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at 
h3master/192.168.0.165:8030}}. So I think AM did connect to the Proxy instead 
of directly to the RM.

> Fail to launch container when distributed scheduling is enabled
> ---
>
> Key: YARN-6514
> URL: https://issues.apache.org/jira/browse/YARN-6514
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: distributed-scheduling, yarn
>Affects Versions: 3.0.0-alpha2
> Environment: Ubuntu Linux 4.4.0-72-generic with java-8-openjdk-amd64 
> 1.8.0_121
>Reporter: Zheng Lv
>
> When yarn.nodemanager.distributed-scheduling.enabled is set to true, 
> mapreduce fails to launch with Invalid AMRMToken errors.
> This error does not occur when the distributed scheduling option is disabled.
> {code:title=yarn-site.xml|borderStyle=solid}
> 
> 
> 
> 
> 
> yarn.resourcemanager.hostname
> h3master
> 
> 
> yarn.nodemanager.aux-services
> mapreduce_shuffle
> 
> 
> yarn.nodemanager.env-whitelist
> 
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
> 
> 
> yarn.nodemanager.aux-services
> mapreduce_shuffle
> 
> 
> yarn.nodemanager.vmem-check-enabled
> false
> 
> 
> 
> yarn.resourcemanager.opportunistic-container-allocation.enabled
> true
> 
> 
> 
> yarn.nodemanager.opportunistic-containers-max-queue-length
> 10
> 
> 
> yarn.nodemanager.distributed-scheduling.enabled
> true
> 
> 
> yarn.nodemanager.amrmproxy.enable
> true
> 
> 
> 
> yarn.resourcemanager.opportunistic-container-allocation.enabled
> true
> 
> 
> yarn.nodemanager.resource.memory-mb
> 4096
> 
> 
> {code}
> {code:title=Container Log|borderStyle=solid}
> 2017-04-23 05:17:50,324 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1492953411349_0001_02
> 2017-04-23 05:17:51,625 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: 
> /
> [system properties]
> os.name: Linux
> os.version: 4.4.0-72-generic
> java.home: /usr/lib/jvm/java-8-openjdk-amd64/jre
> java.runtime.version: 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13
> java.vendor: Oracle Corporation
> java.version: 1.8.0_121
> java.vm.name: OpenJDK 64-Bit Server VM
> java.class.path: 
> 

[jira] [Commented] (YARN-6515) Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980870#comment-15980870
 ] 

Weiwei Yang commented on YARN-6515:
---

Revised the title a bit for consistency :)

> Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager
> 
>
> Key: YARN-6515
> URL: https://issues.apache.org/jira/browse/YARN-6515
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6515.001.patch
>
>
> 5 find bugs issue was reported NM project as part of the YARN-4166 [build| 
> https://builds.apache.org/job/PreCommit-YARN-Build/15694/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html]
> Issue 1: 
>   
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
>  is a mutable collection which should be package protected
> Bug type MS_MUTABLE_COLLECTION_PKGPROTECT (click for details) 
> In class 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics
> Field 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
> At ContainerMetrics.java:\[line 134\]
> Issue 2:
>   
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> Bug type WMI_WRONG_MAP_ITERATOR (click for details) 
> In class 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer
> In method 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
> Field 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.pendingResources
> At ContainerLocalizer.java:\[line 334\]
> Issue 3: 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> Bug type WMI_WRONG_MAP_ITERATOR (click for details) 
> In class org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl
> In method 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
> Field 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.recentlyStoppedContainers
> At NodeStatusUpdaterImpl.java:\[line 721\]
> Issue 4: 
> Hard coded reference to an absolute pathname in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
> Bug type DMI_HARDCODED_ABSOLUTE_FILENAME (click for details) 
> In class 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime
> In method 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
> File name /sys/fs/cgroup
> At DockerLinuxContainerRuntime.java:\[line 455\]
> Useless object stored in variable removedNullContainers of method 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
> Bug type UC_USELESS_OBJECT (click for details) 
> In class org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl
> In method 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
> Value removedNullContainers
> Type java.util.HashSet
> At NodeStatusUpdaterImpl.java:\[line 644\]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6515) Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6515:
--
Summary: Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager  
(was: Findbugs issues in Nodemanager project)

> Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager
> 
>
> Key: YARN-6515
> URL: https://issues.apache.org/jira/browse/YARN-6515
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-6515.001.patch
>
>
> 5 find bugs issue was reported NM project as part of the YARN-4166 [build| 
> https://builds.apache.org/job/PreCommit-YARN-Build/15694/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html]
> Issue 1: 
>   
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
>  is a mutable collection which should be package protected
> Bug type MS_MUTABLE_COLLECTION_PKGPROTECT (click for details) 
> In class 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics
> Field 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
> At ContainerMetrics.java:\[line 134\]
> Issue 2:
>   
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> Bug type WMI_WRONG_MAP_ITERATOR (click for details) 
> In class 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer
> In method 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
> Field 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.pendingResources
> At ContainerLocalizer.java:\[line 334\]
> Issue 3: 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> Bug type WMI_WRONG_MAP_ITERATOR (click for details) 
> In class org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl
> In method 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
> Field 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.recentlyStoppedContainers
> At NodeStatusUpdaterImpl.java:\[line 721\]
> Issue 4: 
> Hard coded reference to an absolute pathname in 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
> Bug type DMI_HARDCODED_ABSOLUTE_FILENAME (click for details) 
> In class 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime
> In method 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
> File name /sys/fs/cgroup
> At DockerLinuxContainerRuntime.java:\[line 455\]
> Useless object stored in variable removedNullContainers of method 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
> Bug type UC_USELESS_OBJECT (click for details) 
> In class org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl
> In method 
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
> Value removedNullContainers
> Type java.util.HashSet
> At NodeStatusUpdaterImpl.java:\[line 644\]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6517) Fix warnings from Spotbugs in hadoop-yarn-common

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6517:
--
Attachment: YARN-6517.001.patch

> Fix warnings from Spotbugs in hadoop-yarn-common
> 
>
> Key: YARN-6517
> URL: https://issues.apache.org/jira/browse/YARN-6517
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6517.001.patch
>
>
> There are 2 findbugs warnings in hadoop-yarn-common project since switched to 
> spotbugs,
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogValue.getPendingLogFilesToUpload(File)
>  due to return value of called method
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessList() due to 
> return value of called method
> see more in 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6518) Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6518:
--
Attachment: YARN-6518.001.patch

> Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice
> 
>
> Key: YARN-6518
> URL: https://issues.apache.org/jira/browse/YARN-6518
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6518.001.patch
>
>
> There is 1 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntities(File,
>  String, TimelineEntityFilters, TimelineDataToRetrieve) due to return value 
> of called method
> See more in 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6520) Fix warnings from Spotbugs in hadoop-yarn-client

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6520:
--
Attachment: YARN-6520.001.patch

> Fix warnings from Spotbugs in hadoop-yarn-client
> 
>
> Key: YARN-6520
> URL: https://issues.apache.org/jira/browse/YARN-6520
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6520.001.patch
>
>
> There 2 findbugs warning in hadoop-yarn-client since switched to spotbugs
> # Possible exposure of partially initialized object in 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken()
> # Useless condition: it's known that isAppFinished == false at this point
> See more info from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second

[GirlKiller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

GirlKiller updated YARN-6516:
-
Priority: Blocker  (was: Major)

> FairScheduler:the algorithm of assignContainer is so slow for it only can 
> assign a thousand containers per second
> -
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: GirlKiller
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second

[GirlKiller (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980852#comment-15980852
 ] 

GirlKiller commented on YARN-6516:
--

is the avg value

> FairScheduler:the algorithm of assignContainer is so slow for it only can 
> assign a thousand containers per second
> -
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 2.7.2
>Reporter: GirlKiller
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6519:
--
Attachment: YARN-6519.001.patch

> Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager
> 
>
> Key: YARN-6519
> URL: https://issues.apache.org/jira/browse/YARN-6519
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6519.001.patch
>
>
> There is 8 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager$1.compare(CSQueue,
>  CSQueue) incorrectly handles float value
> # org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType.index 
> field is public and mutable
> # 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.queueMetrics
>  is a mutable collection
> # 
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.cleanupStaledPreemptionCandidates(long)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.transferStateFromAttempt(RMAppAttempt)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.cleanupPreemptionList()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> See more from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6518) Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-6518:
--
Summary: Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice  
(was: Fix warnings from Spotbugs in hadoop-yarn-common)

> Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice
> 
>
> Key: YARN-6518
> URL: https://issues.apache.org/jira/browse/YARN-6518
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
>
> There is 1 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntities(File,
>  String, TimelineEntityFilters, TimelineDataToRetrieve) due to return value 
> of called method
> See more in 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6520) Fix warnings from Spotbugs in hadoop-yarn-client

[Weiwei Yang (JIRA)]

Weiwei Yang created YARN-6520:
-

 Summary: Fix warnings from Spotbugs in hadoop-yarn-client
 Key: YARN-6520
 URL: https://issues.apache.org/jira/browse/YARN-6520
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Weiwei Yang
Assignee: Weiwei Yang


There 2 findbugs warning in hadoop-yarn-client since switched to spotbugs

# Possible exposure of partially initialized object in 
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken()
# Useless condition: it's known that isAppFinished == false at this point

See more info from 
[https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html]




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Weiwei Yang (JIRA)]

Weiwei Yang created YARN-6519:
-

 Summary: Fix warnings from Spotbugs in 
hadoop-yarn-server-resourcemanager
 Key: YARN-6519
 URL: https://issues.apache.org/jira/browse/YARN-6519
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Weiwei Yang
Assignee: Weiwei Yang


There is 8 findbugs warning in hadoop-yarn-server-timelineservice since 
switched to spotbugs

# 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager$1.compare(CSQueue,
 CSQueue) incorrectly handles float value
# org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType.index field 
is public and mutable
# 
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.EMPTY_CONTAINER_LIST
 is a mutable collection which should be package protected
# 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.EMPTY_CONTAINER_LIST
 is a mutable collection which should be package protected
# 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.queueMetrics
 is a mutable collection
# 
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.cleanupStaledPreemptionCandidates(long)
 makes inefficient use of keySet iterator instead of entrySet iterator
# 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.transferStateFromAttempt(RMAppAttempt)
 makes inefficient use of keySet iterator instead of entrySet iterator
# 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.cleanupPreemptionList()
 makes inefficient use of keySet iterator instead of entrySet iterator

See more from 
[https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html]





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-6518) Fix warnings from Spotbugs in hadoop-yarn-common

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang reassigned YARN-6518:
-

Assignee: Weiwei Yang

> Fix warnings from Spotbugs in hadoop-yarn-common
> 
>
> Key: YARN-6518
> URL: https://issues.apache.org/jira/browse/YARN-6518
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
>
> There is 1 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # Possible null pointer dereference in 
> org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntities(File,
>  String, TimelineEntityFilters, TimelineDataToRetrieve) due to return value 
> of called method
> See more in 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6518) Fix warnings from Spotbugs in hadoop-yarn-common

[Weiwei Yang (JIRA)]

Weiwei Yang created YARN-6518:
-

 Summary: Fix warnings from Spotbugs in hadoop-yarn-common
 Key: YARN-6518
 URL: https://issues.apache.org/jira/browse/YARN-6518
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Weiwei Yang


There is 1 findbugs warning in hadoop-yarn-server-timelineservice since 
switched to spotbugs

# Possible null pointer dereference in 
org.apache.hadoop.yarn.server.timelineservice.storage.FileSystemTimelineReaderImpl.getEntities(File,
 String, TimelineEntityFilters, TimelineDataToRetrieve) due to return value of 
called method

See more in 
[https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6291) Introduce query parameters (sort, filter, etc.) for tables to keep state on refresh/navigation in new YARN UI

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-6291:
--
Summary: Introduce query parameters (sort, filter, etc.) for tables to keep 
state on refresh/navigation in new YARN UI  (was: [YARN-3368] Introduce query 
parameters (sort, filter, etc.) for tables to keep state on refresh/navigation)

> Introduce query parameters (sort, filter, etc.) for tables to keep state on 
> refresh/navigation in new YARN UI
> -
>
> Key: YARN-6291
> URL: https://issues.apache.org/jira/browse/YARN-6291
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Novák
>Assignee: Gergely Novák
> Attachments: YARN-6291.001.patch, YARN-6291.002.patch, 
> YARN-6291.003.patch
>
>
> On Applications / Nodes pages the user can define filters, sort the list on 
> selected columns, etc. But all these settings are lost on a refresh, or 
> navigation, which is very inconvenient.
> For example a possible use case might be watching the Applications table 
> while running a lot of YARN applications and continuously pressing Refresh to 
> see their progress. In this case, the user have to set any filter / sorting 
> manually after every single refresh. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org