[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323706#comment-16323706
]
genericqa commented on YARN-7738:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 4s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 413 unchanged - 0 fixed = 416 total (was 413) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
50s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905812/YARN-7738.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux dfe31b26722f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / addbcd8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbu
[jira] [Commented] (YARN-7672) hadoop-sls can not simulate huge scale of YARN
[
https://issues.apache.org/jira/browse/YARN-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323728#comment-16323728
]
stefanlee commented on YARN-7672:
-
[~zsl2007] thanks for this jira. i have merged this patch to my hadoop version
and there is a problem occurred during my testing.
{code:java}
1. RM1 is active ,RM2 is standby
2. i run SLSRunnerForRealRM and my jobs will running in my cluster with correct
user name and queue name.
then:
1. RM1 is standby , RM2 is active
2. i run SLSRunnerForRealRM and my jobs will failover to RM2, then them will
running in my cluster with the user who
run SLSRunnerForRealRM. that is ,them will running in one queue.
{code}
i review the hadoop resource and found this prolem occurred in
*ConfiguredRMFailoverProxyProivder.getProxyInternal->RMProxy.getProxy*
{code:java}
static T getProxy(final Configuration conf,
final Class protocol, final InetSocketAddress rmAddress)
throws IOException {
return UserGroupInformation.getCurrentUser().doAs(
new PrivilegedAction() {
@Override
public T run() {
return (T) YarnRPC.create(conf).getProxy(protocol, rmAddress, conf);
}
});
{code}
here, it will *getCurrentUser()*, so we should come up with a solution to
resolve it.
> hadoop-sls can not simulate huge scale of YARN
> --
>
> Key: YARN-7672
> URL: https://issues.apache.org/jira/browse/YARN-7672
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: zhangshilong
>Assignee: zhangshilong
> Attachments: YARN-7672.patch
>
>
> Our YARN cluster scale to nearly 10 thousands nodes. We need to do scheduler
> pressure test.
> Using SLS,we start 2000+ threads to simulate NM and AM. But cpu.load very
> high to 100+. I thought that will affect performance evaluation of
> scheduler.
> So I thought to separate the scheduler from the simulator.
> I start a real RM. Then SLS will register nodes to RM,And submit apps to RM
> using RM RPC.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7672) hadoop-sls can not simulate huge scale of YARN
[
https://issues.apache.org/jira/browse/YARN-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323728#comment-16323728
]
stefanlee edited comment on YARN-7672 at 1/12/18 9:18 AM:
--
[~zsl2007] thanks for this jira. i have merged this patch to my hadoop version
and there is a problem occurred during my testing.
{code:java}
1. RM1 is active ,RM2 is standby
2. i run SLSRunnerForRealRM and my jobs will running in my cluster with correct
user name and queue name.
then:
1. RM1 is standby , RM2 is active
2. i run SLSRunnerForRealRM and my jobs will failover to RM2, then them will
running in my cluster with the user who
run SLSRunnerForRealRM. that is ,them will running in one queue.
{code}
i review the hadoop resource and found this prolem occurred in
*ConfiguredRMFailoverProxyProivder.getProxyInternal->RMProxy.getProxy*
{code:java}
static T getProxy(final Configuration conf,
final Class protocol, final InetSocketAddress rmAddress)
throws IOException {
return UserGroupInformation.getCurrentUser().doAs(
new PrivilegedAction() {
@Override
public T run() {
return (T) YarnRPC.create(conf).getProxy(protocol, rmAddress, conf);
}
});
{code}
here, it will *getCurrentUser()*, so we should come up with a solution to
resolve it.
but if we have only one RM, it will run well.:D
was (Author: imstefanlee):
[~zsl2007] thanks for this jira. i have merged this patch to my hadoop version
and there is a problem occurred during my testing.
{code:java}
1. RM1 is active ,RM2 is standby
2. i run SLSRunnerForRealRM and my jobs will running in my cluster with correct
user name and queue name.
then:
1. RM1 is standby , RM2 is active
2. i run SLSRunnerForRealRM and my jobs will failover to RM2, then them will
running in my cluster with the user who
run SLSRunnerForRealRM. that is ,them will running in one queue.
{code}
i review the hadoop resource and found this prolem occurred in
*ConfiguredRMFailoverProxyProivder.getProxyInternal->RMProxy.getProxy*
{code:java}
static T getProxy(final Configuration conf,
final Class protocol, final InetSocketAddress rmAddress)
throws IOException {
return UserGroupInformation.getCurrentUser().doAs(
new PrivilegedAction() {
@Override
public T run() {
return (T) YarnRPC.create(conf).getProxy(protocol, rmAddress, conf);
}
});
{code}
here, it will *getCurrentUser()*, so we should come up with a solution to
resolve it.
> hadoop-sls can not simulate huge scale of YARN
> --
>
> Key: YARN-7672
> URL: https://issues.apache.org/jira/browse/YARN-7672
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: zhangshilong
>Assignee: zhangshilong
> Attachments: YARN-7672.patch
>
>
> Our YARN cluster scale to nearly 10 thousands nodes. We need to do scheduler
> pressure test.
> Using SLS,we start 2000+ threads to simulate NM and AM. But cpu.load very
> high to 100+. I thought that will affect performance evaluation of
> scheduler.
> So I thought to separate the scheduler from the simulator.
> I start a real RM. Then SLS will register nodes to RM,And submit apps to RM
> using RM RPC.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323743#comment-16323743
]
Sunil G commented on YARN-7738:
---
Thanks [~leftnoteasy]. Fix seems fine to me
Few comments:
# {{Resource#setResourceValue}} will internally take care of checking whether
resource name is memory or vcores and accordingly call setMemorySize or
setVirtualCores. Hence we could avoid
{{setMaximumAllocationByConfiguredResourceInformation}} in ResourceUtils.
# {{if (!Resources.fitsIn(oldMax, newMax))}} could we make its like {{if
(Resources.fitsIn(newMax, oldMax))}}. fitsIn checks smaller as first arg.
TO check test cases, i ll try attach a patch with this changes.
> CapacityScheduler: Support refresh maximum allocation for multiple resource
> types
> -
>
> Key: YARN-7738
> URL: https://issues.apache.org/jira/browse/YARN-7738
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Fix For: 3.1.0, 3.0.1
>
> Attachments: YARN-7738.001.patch, YARN-7738.002.patch
>
>
> Currently CapacityScheduler fails to refresh maximum allocation for multiple
> resource types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7624) RM gives YARNFeatureNotEnabledException even when resource profile feature is not enabled
[
https://issues.apache.org/jira/browse/YARN-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323745#comment-16323745
]
Sunil G commented on YARN-7624:
---
[~maniraj...@gmail.com], could u pls share a patch based on your analysis. I
might not have enough bandwidth for this. Thanks.
> RM gives YARNFeatureNotEnabledException even when resource profile feature is
> not enabled
> -
>
> Key: YARN-7624
> URL: https://issues.apache.org/jira/browse/YARN-7624
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Weiwei Yang
>Assignee: Sunil G
>
> A single node setup, I haven't enabled resource profile feature. Property
> {{yarn.resourcemanager.resource-profiles.enabled}} was not set. Start yarn,
> launch a job, I got following error message in RM log
> {noformat}
> org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
> profile is not enabled, please enable resource profile feature before using
> its functions. (by setting yarn.resourcemanager.resource-profiles.enabled to
> true)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1822)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:657)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:617)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {noformat}
> this is confusing because I did not enable this feature, why I still get this
> error?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-6736: Attachment: YARN-6736.001.patch The patch looks approach looks good to me! It also maintaining compatibility which is good. I attached the rebased patch to run jenkins. > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch, YARN-6736-YARN-5355.005.patch, > YARN-6736.001.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7738: -- Attachment: YARN-7738.003.patch attaching patch for jenkins. > CapacityScheduler: Support refresh maximum allocation for multiple resource > types > - > > Key: YARN-7738 > URL: https://issues.apache.org/jira/browse/YARN-7738 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Tan, Wangda >Priority: Blocker > Fix For: 3.1.0, 3.0.1 > > Attachments: YARN-7738.001.patch, YARN-7738.002.patch, > YARN-7738.003.patch > > > Currently CapacityScheduler fails to refresh maximum allocation for multiple > resource types. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323778#comment-16323778 ] Rohith Sharma K S commented on YARN-6736: - [~varun_saxena] [~vrushalic] do you have any comments on the patch? > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch, YARN-6736-YARN-5355.005.patch, > YARN-6736.001.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-6736: Target Version/s: 3.1.0, 3.0.1 Description: When the cluster is being upgraded from atsv1 to v2, it may be good to have a brief time period during which RM writes to both atsv1 and v2. This will help frameworks like Tez migrate more smoothly. (was: When the cluster is being upgraded from atsv1 to v2, it may be good to have a brief time period during which RM writes to both atsv1 and v2. This will help frameworks like Tez migrate more smoothly. ) > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch, YARN-6736-YARN-5355.005.patch, > YARN-6736.001.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323787#comment-16323787 ] Sunil G commented on YARN-6736: --- Thanks [~agresch] and [~rohithsharma]. This makes sense to have it can where we want to have upgrade seamlessly from ATS1.5 to 2 to avoid data loss. > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch, YARN-6736-YARN-5355.005.patch, > YARN-6736.001.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7727) Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector
[
https://issues.apache.org/jira/browse/YARN-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7727:
--
Summary: Incorrect log levels in few logs with
QueuePriorityContainerCandidateSelector (was: Logging issues with
QueuePriorityContainerCandidateSelector)
> Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector
> -
>
> Key: YARN-7727
> URL: https://issues.apache.org/jira/browse/YARN-7727
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Labels: supportability
> Attachments: YARN-7727.1.patch
>
>
> 1. RM logs are verbose with below log message repeats for every fraction of
> second from QueuePriorityContainerCandidateSelector
> 2. And there is a minor issue where we log info message after checking debug
> enabled.
> {code}
> 2018-01-10 05:46:07,834 INFO
> capacity.QueuePriorityContainerCandidateSelector
> (QueuePriorityContainerCandidateSelector.java:intializePriorityDigraph(122))
> - Initializing priority preemption directed graph:
> 2018-01-10 05:46:10,835 INFO
> capacity.QueuePriorityContainerCandidateSelector
> (QueuePriorityContainerCandidateSelector.java:intializePriorityDigraph(122))
> - Initializing priority preemption directed graph:
> 2018-01-10 05:46:13,836 INFO
> capacity.QueuePriorityContainerCandidateSelector
> (QueuePriorityContainerCandidateSelector.java:intializePriorityDigraph(122))
> - Initializing priority preemption directed graph:
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[
https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323791#comment-16323791
]
genericqa commented on YARN-6736:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 6m
59s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-api in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-server-tests in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-client in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-applications-distributedshell in trunk
failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
2m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
10s{color} | {color:red} hadoop-yarn-api in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} hadoop-yarn-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-client in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
12s{color} | {color:red} hadoop-yarn-applications-distributedshell in trunk
failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-api in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-yarn-server-tests in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-yarn-client in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-yarn-applications-distributedshell in trunk
failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
13s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} hadoop-yarn-applications-distributedshell in the patch
failed. {color} |
| {color:red}-1{color} | {color:red}
[jira] [Comment Edited] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323755#comment-16323755 ] Rohith Sharma K S edited comment on YARN-6736 at 1/12/18 10:30 AM: --- The patch approach looks good to me! It also maintaining compatibility which is good. I attached the rebased patch to run jenkins. was (Author: rohithsharma): The patch looks approach looks good to me! It also maintaining compatibility which is good. I attached the rebased patch to run jenkins. > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch, YARN-6736-YARN-5355.003.patch, > YARN-6736-YARN-5355.004.patch, YARN-6736-YARN-5355.005.patch, > YARN-6736.001.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7727) Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector
[
https://issues.apache.org/jira/browse/YARN-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323803#comment-16323803
]
Hudson commented on YARN-7727:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13484 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13484/])
YARN-7727. Incorrect log levels in few logs with (sunilg: rev
128d773a2315fa6baaa3a52b13c53c77e741b69c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/QueuePriorityContainerCandidateSelector.java
> Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector
> -
>
> Key: YARN-7727
> URL: https://issues.apache.org/jira/browse/YARN-7727
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Labels: supportability
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: YARN-7727.1.patch
>
>
> 1. RM logs are verbose with below log message repeats for every fraction of
> second from QueuePriorityContainerCandidateSelector
> 2. And there is a minor issue where we log info message after checking debug
> enabled.
> {code}
> 2018-01-10 05:46:07,834 INFO
> capacity.QueuePriorityContainerCandidateSelector
> (QueuePriorityContainerCandidateSelector.java:intializePriorityDigraph(122))
> - Initializing priority preemption directed graph:
> 2018-01-10 05:46:10,835 INFO
> capacity.QueuePriorityContainerCandidateSelector
> (QueuePriorityContainerCandidateSelector.java:intializePriorityDigraph(122))
> - Initializing priority preemption directed graph:
> 2018-01-10 05:46:13,836 INFO
> capacity.QueuePriorityContainerCandidateSelector
> (QueuePriorityContainerCandidateSelector.java:intializePriorityDigraph(122))
> - Initializing priority preemption directed graph:
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323832#comment-16323832 ] Sunil G commented on YARN-7159: --- bq.Here, value has been configured as 10 with "" units, but ResourceUtils#createResourceTypesArray triggered from FairSchedulerConfiguration#getIncrementAllocation returns array of RI's containing A_CUSTOM_RESOURCE with value as 10k, as it simply copies the object and modifies only the value. Is this correct behaviour? I dont think its correct. [~dan...@cloudera.com] could u pls help to check this behavior. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch, YARN-7159.017.patch, YARN-7159.018.patch, > YARN-7159.019.patch, YARN-7159.020.patch, YARN-7159.021.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7728) Expose and expand container preemptions in Capacity Scheduler queue metrics
[ https://issues.apache.org/jira/browse/YARN-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323837#comment-16323837 ] Sunil G commented on YARN-7728: --- Thanks [~eepayne], this will be really helpful when preemption happens. I have doubt w.r.t multiple resources scenario. In 3.0, we support multiple types, and this covers only cpu and memory. So could we cover preemption metrics also in case of multi resources. One more doubt is with aggregateVcoreSecondsPreempted. MutableCounterLong is used for this. But under one queue, we ll have multiple containers gets preempted and each container resource size vary drastically. So are we looking for an aggregate resource among all preempted containers in a given time ? Minor Nits: aggregateMegabyteSecondsPreempted: MegaByte seems a bit confusing, MemoryMB is used in another places as well. Could we use something similar (like prepending memory) > Expose and expand container preemptions in Capacity Scheduler queue metrics > --- > > Key: YARN-7728 > URL: https://issues.apache.org/jira/browse/YARN-7728 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0, 2.8.3, 3.0.0 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: YARN-7728.001.patch > > > YARN-1047 exposed queue metrics for the number of preempted containers to the > fair scheduler. I would like to also expose these to the capacity scheduler > and add metrics for the amount of lost memory seconds and vcore seconds. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7742) [UI2] Duplicated containers are rendered per attempt
Rohith Sharma K S created YARN-7742: --- Summary: [UI2] Duplicated containers are rendered per attempt Key: YARN-7742 URL: https://issues.apache.org/jira/browse/YARN-7742 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S Attachments: Screen Shot 2018-01-12 at 5.10.48 PM.png In UI2, containers are rendered twice with different start and end time. Attached the screen shot of UI2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323865#comment-16323865
]
Gergo Repas commented on YARN-7451:
---
Thanks [~snemeth] for the patch. A couple of comments:
# You documented the motivation behind the serialization-related changes at
some locations (etc. in ResourceInfoWithCustomResourceTypes), but would be
useful to add some explanation to other locations. For example:
## ResourceManager.startWepApp() when setting the IteratorProvider
## JAXBContextResolver.ECLIPSELINK_SERIALIZED_TYPES - some comment would be
useful why these classes need special handling (maybe even a short example).
## in the resourcemanager's pom.xml - where you're adding the new dependency
# SchedulerInfo.getMaxClusterLevelAppPriority() - please add a comment what's
the reason for adding @XmlTransient
# In some classes imports are changed without any logic change, etc: RMWebApp,
TestRMWebServicesConfigurationMutation
# ArrayElementPathSegment.extractIndex() - when input does not match pattern -
this case is not handled
# There are outstanding checkstyle, license and javadoc issues
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
>
[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323888#comment-16323888
]
genericqa commented on YARN-7738:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 33s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestZKConfigurationStore
|
| | hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors |
| | hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler |
| | hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
| |
hadoop.yarn.server.resourcemanager.scheduler.capa
[jira] [Assigned] (YARN-7742) [UI2] Duplicated containers are rendered per attempt
[ https://issues.apache.org/jira/browse/YARN-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-7742: - Assignee: Vasudevan Skm > [UI2] Duplicated containers are rendered per attempt > > > Key: YARN-7742 > URL: https://issues.apache.org/jira/browse/YARN-7742 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Vasudevan Skm > Attachments: Screen Shot 2018-01-12 at 5.10.48 PM.png > > > In UI2, containers are rendered twice with different start and end time. > Attached the screen shot of UI2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-7451:
-
Attachment: YARN-7451.006.patch
Thanks [~grepas] for your comments.
All of them are fixed except
org.apache.hadoop.yarn.server.resourcemanager.webapp.representationhelper.path.ArrayElementPathSegment#extractIndex
since it is already "handled".
in the line "return Integer.parseInt(matcher.group(1));" the matcher group call
would throw an IllegalStateException which is perfectly reasonable in tests I
think.
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
> ]
> {noformat}
> However, the web UI shows resource types usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323935#comment-16323935
]
Szilard Nemeth edited comment on YARN-7451 at 1/12/18 1:10 PM:
---
Thanks [~grepas] for your comments.
All of them are fixed except
org.apache.hadoop.yarn.server.resourcemanager.webapp.representationhelper.path.ArrayElementPathSegment#extractIndex
since it is already "handled".
in the line "return Integer.parseInt(matcher.group(1));" the matcher group call
would throw an IllegalStateException which is perfectly reasonable in tests I
think.
Also fixed a bunch of checkstyle warnings.
was (Author: snemeth):
Thanks [~grepas] for your comments.
All of them are fixed except
org.apache.hadoop.yarn.server.resourcemanager.webapp.representationhelper.path.ArrayElementPathSegment#extractIndex
since it is already "handled".
in the line "return Integer.parseInt(matcher.group(1));" the matcher group call
would throw an IllegalStateException which is perfectly reasonable in tests I
think.
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
>
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323938#comment-16323938
]
genericqa commented on YARN-7451:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7451 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905862/YARN-7451.006.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19219/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
>
[jira] [Comment Edited] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323935#comment-16323935
]
Szilard Nemeth edited comment on YARN-7451 at 1/12/18 1:11 PM:
---
Thanks [~grepas] for your comments.
All of them are fixed except
org.apache.hadoop.yarn.server.resourcemanager.webapp.representationhelper.path.ArrayElementPathSegment#extractIndex
since it is already "handled".
See the line "return Integer.parseInt(matcher.group(1));" ,
the matcher group call would throw an IllegalStateException which is perfectly
reasonable in tests I suppose.
Also fixed a bunch of checkstyle warnings.
was (Author: snemeth):
Thanks [~grepas] for your comments.
All of them are fixed except
org.apache.hadoop.yarn.server.resourcemanager.webapp.representationhelper.path.ArrayElementPathSegment#extractIndex
since it is already "handled".
in the line "return Integer.parseInt(matcher.group(1));" the matcher group call
would throw an IllegalStateException which is perfectly reasonable in tests I
think.
Also fixed a bunch of checkstyle warnings.
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
>
[jira] [Updated] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-7738:
--
Attachment: YARN-7738.004.patch
{{fitsIn}} logic was originally correct. Reattaching the patch.
> CapacityScheduler: Support refresh maximum allocation for multiple resource
> types
> -
>
> Key: YARN-7738
> URL: https://issues.apache.org/jira/browse/YARN-7738
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Fix For: 3.1.0, 3.0.1
>
> Attachments: YARN-7738.001.patch, YARN-7738.002.patch,
> YARN-7738.003.patch, YARN-7738.004.patch
>
>
> Currently CapacityScheduler fails to refresh maximum allocation for multiple
> resource types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323947#comment-16323947
]
genericqa commented on YARN-7451:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 7m
8s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7451 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905862/YARN-7451.006.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19220/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
>
[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323949#comment-16323949
]
genericqa commented on YARN-7738:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 6m
59s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19221/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> CapacityScheduler: Support refresh maximum allocation for multiple resource
> types
> -
>
> Key: YARN-7738
> URL: https://issues.apache.org/jira/browse/YARN-7738
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Fix For: 3.1.0, 3.0.1
>
> Attachments: YARN-7738.001.patch, YARN-7738.002.patch,
> YARN-7738.003.patch, YARN-7738.004.patch
>
>
> Currently CapacityScheduler fails to refresh maximum allocation for multiple
> resource types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323964#comment-16323964
]
genericqa commented on YARN-7738:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 5m
38s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19222/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> CapacityScheduler: Support refresh maximum allocation for multiple resource
> types
> -
>
> Key: YARN-7738
> URL: https://issues.apache.org/jira/browse/YARN-7738
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Tan, Wangda
>Priority: Blocker
> Fix For: 3.1.0, 3.0.1
>
> Attachments: YARN-7738.001.patch, YARN-7738.002.patch,
> YARN-7738.003.patch, YARN-7738.004.patch
>
>
> Currently CapacityScheduler fails to refresh maximum allocation for multiple
> resource types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[
https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323965#comment-16323965
]
genericqa commented on YARN-6736:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 353 unchanged - 0 fixed = 355 total (was 353) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m
23s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 57s{color}
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
57s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {
[jira] [Updated] (YARN-6492) Generate queue metrics for each partition
[ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-6492: -- Target Version/s: 3.1.0 > Generate queue metrics for each partition > - > > Key: YARN-6492 > URL: https://issues.apache.org/jira/browse/YARN-6492 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Manikandan R > Attachments: YARN-6492.001.patch, YARN-6492.002.patch, > partition_metrics.txt > > > We are interested in having queue metrics for all partitions. Right now each > queue has one QueueMetrics object which captures metrics either in default > partition or across all partitions. (After YARN-6467 it will be in default > partition) > But having the partition metrics would be very useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7739) Revisit scheduler resource normalization behavior for max allocation
[ https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324014#comment-16324014 ] Jason Lowe commented on YARN-7739: -- I'm not a fan of silently capping the app's request. If the app says it needs 12GB then it needs 12GB. I think it is unhelpful more often than not to assume the app can work with less. In the case where an app wants "the biggest you can offer up to this amount" kind of allocations then it should be able to query the RM for the current maximum capability. It's already told the max allocation during app registration, but currently this can be dynamically updated (e.g.: queue refresh) without the app's knowledge. As far as dynamically setting the maximum allocation, some of that stems from the desire to keep apps from hanging forever if they ask for a container that is bigger than any node can satisfy. See YARN-2604. I'm personally torn on the behavior. In many cases it could be very useful to proactively tell an app that its container cannot be satisfied by any node in the cluster, but on the other hand we don't know if such requests would be satisfied just a little bit later because a large node that was temporarily offline rejoins the cluster. If we do allow the maximum allocation to fluctuate based on current node capabilities then I think there needs to be a way for the AM to either query for the current max allocation or be proactively told about the max allocation change in the allocation response. > Revisit scheduler resource normalization behavior for max allocation > > > Key: YARN-7739 > URL: https://issues.apache.org/jira/browse/YARN-7739 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Priority: Critical > > Currently, YARN Scheduler normalizes requested resource based on the maximum > allocation derived from configured maximum allocation and maximum registered > node resources. Basically, the scheduler will silently cap asked resource by > maximum allocation. > This could cause issues for applications, for example, a Spark job which > needs 12 GB memory to run, however in the cluster, registered NMs have at > most 8 GB mem on each node. So scheduler allocates 8GB memory container to > the requested application. > Once app receives containers from RM, if it doesn't double check allocated > resources, it will lead to OOM and hard to debug because scheduler silently > caps maximum allocation. > When non-mandatory resources introduced, this becomes worse. For resources > like GPU, we typically set minimum allocation to 0 since not all nodes have > GPU devices. So it is possible that application asks 4 GPUs but get 0 GPU, it > gonna be a big problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3895) Support ACLs in ATSv2
[ https://issues.apache.org/jira/browse/YARN-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324140#comment-16324140 ] Jason Lowe commented on YARN-3895: -- I think Application ACLs could work fine for the straightforward case of a user running their own app. As you mentioned, it already reflects how YARN handles the ACLs for the AHS and log server today. It's less clear to me how this is going to work for the case of an AM running as one user but working on behalf of multiple other users across multiple sub-apps. The YARN application only has one set of ACLs, set when it is submitted by the service user. Those permissions are going to be restricted to just the service user, most likely. Then the service user runs a sub-app (e.g.: a DAG) on behalf of another user. In that case the ACLs may need to change (e.g.: be permissive to more groups, etc.). The YARN app ACL isn't changing at this point, it was set at time of submit, so how does the AM inform the collector of the ACL change? Similarly, even if the AM wrapped some of its execution in a doAs for the other user, how does the collector know the user has changed? Did the AM somehow disconnect and reconnect to the collector? How does the collector authenticate that the AM is allowed to proxy as that user, or can any AM forge data as other users simply by stating the data is from so-and-so? I'm not that familiar with HBase, but it looks like the ACLs are per cell and then it seems pretty straightforward how ACLs could change across sub-apps and implement the proper restrictions on the read path. It's the write path in the multiple-sub-apps-for-multiple-users-by-one-service-user case that I'm not seeing how the security works. If we're basing it on the YARN app ACL, that isn't changing across sub-apps but in many cases will need to do so. > Support ACLs in ATSv2 > - > > Key: YARN-3895 > URL: https://issues.apache.org/jira/browse/YARN-3895 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355 > > This JIRA is to keep track of authorization support design discussions for > both readers and collectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7724) yarn application status should support application name
[
https://issues.apache.org/jira/browse/YARN-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324156#comment-16324156
]
Billie Rinaldi commented on YARN-7724:
--
The changes to ApplicationCLI look good. The changes to ServiceClient and
AppAdminClient are no longer needed. The usage statement should also be
updated. Here is the current usage; I suggest changing status to have something
similar to the stop command:
{noformat}
-status Prints the status of the
application.
-stopStops application gracefully
(may be started again later). If
name is provided, appType must
be provided unless it is the
default yarn-service. If ID is
provided, the appType will be
looked up. Supports -appTypes
option to specify which client
implementation to use.
{noformat}
Except for the part about "If ID is provided, the appType will be looked up,"
because that isn't true for status. See ApplicationCLI.java#L140-L141 and
ApplicationCLI.java#L193. When usage is updated,
TestYarnCLI.createApplicationCLIHelpMessage and YarnCommands.md will also need
to be updated.
> yarn application status should support application name
> ---
>
> Key: YARN-7724
> URL: https://issues.apache.org/jira/browse/YARN-7724
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Reporter: Yesha Vora
>Assignee: Jian He
> Attachments: YARN-7724.01.patch, YARN-7724.02.patch,
> YARN-7724.03.patch
>
>
> Yarn Service application are tied with app name. Thus, yarn application
> -status should be able to take yarn service name as argument such as
> yarn application -status
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7728) Expose and expand container preemptions in Capacity Scheduler queue metrics
[
https://issues.apache.org/jira/browse/YARN-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324165#comment-16324165
]
Eric Payne commented on YARN-7728:
--
Thanks a lot for the comments, [~sunilg].
bq. n 3.0, we support multiple types, and this covers only cpu and memory. So
could we cover preemption metrics also in case of multi resources.
I agree with this in principle. However, I made a conscious decision not to do
this. There are a couple of difficulties that I see. First, this is not done
for other resource metrics in QueueMetrics (or any of the other system metrics
I could find). The resource metrics only cover memory and vcores. Second,
making the metric names match the resource names is a little difficult if the
resource names could be dynamic. Because of these two things, I feel that
solving this should be done all at the same time in a more general JIRA.
{quote}
One more doubt is with aggregateVcoreSecondsPreempted. MutableCounterLong is
used for this. But under one queue, we ll have multiple containers gets
preempted and each container resource size vary drastically. So are we looking
for an aggregate resource among all preempted containers in a given time ?
{quote}
I don't think I understand the question. The metrics are updated when each
container is preempted, and the value keeps increasing over time. Similar to
memory, it's basically a metric of total lost (virtual) cpu cycles due to
preemption since the RM was started.
{quote}
aggregateMegabyteSecondsPreempted: MegaByte seems a bit confusing, MemoryMB is
used in another places as well. Could we use something similar (like prepending
memory)
{quote}
Good point. I will update a new patch.
> Expose and expand container preemptions in Capacity Scheduler queue metrics
> ---
>
> Key: YARN-7728
> URL: https://issues.apache.org/jira/browse/YARN-7728
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.9.0, 2.8.3, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: YARN-7728.001.patch
>
>
> YARN-1047 exposed queue metrics for the number of preempted containers to the
> fair scheduler. I would like to also expose these to the capacity scheduler
> and add metrics for the amount of lost memory seconds and vcore seconds.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324185#comment-16324185
]
Szilard Nemeth commented on YARN-7451:
--
Unfortunately, I got an unfavorable response for the legal jira, see:
https://issues.apache.org/jira/browse/LEGAL-359, so I cannot include the
ServiceFinder as a source file, apparently.
Any ideas how to resolve this?
thanks!
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
> ]
> {noformat}
> However, the web UI shows resource types usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324185#comment-16324185
]
Szilard Nemeth edited comment on YARN-7451 at 1/12/18 4:21 PM:
---
Unfortunately, I got an unfavorable response for the legal jira, see:
https://issues.apache.org/jira/browse/LEGAL-359, so I cannot include the
ServiceFinder as a source file, apparently.
[~rkanter] Any ideas how to resolve this situation?
thanks!
was (Author: snemeth):
Unfortunately, I got an unfavorable response for the legal jira, see:
https://issues.apache.org/jira/browse/LEGAL-359, so I cannot include the
ServiceFinder as a source file, apparently.
Any ideas how to resolve this?
thanks!
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
> ]
> {noformat}
> However, the web UI shows resource types usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
[jira] [Updated] (YARN-7729) Add support for setting the PID namespace mode
[
https://issues.apache.org/jira/browse/YARN-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billie Rinaldi updated YARN-7729:
-
Attachment: YARN-7729.001.patch
Attaching first draft. This patch adds a yarn-site.xml property
yarn.nodemanager.runtime.linux.docker.host-pid-namespace.allowed (valued true
or false), an environment variable
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_PID_NAMESPACE for the docker runtime
(value would be "host" to request the host pid namespace), and a docker
container-executor.cfg property docker.pid-host.enabled (valued 0 or 1).
> Add support for setting the PID namespace mode
> --
>
> Key: YARN-7729
> URL: https://issues.apache.org/jira/browse/YARN-7729
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Billie Rinaldi
> Attachments: YARN-7729.001.patch
>
>
> Docker has support for allowing containers to share the PID namespace with
> the host or other containers via the {{docker run --pid}} flag.
> There are a number of use cases where this is desirable:
> * Monitoring tools running in containers that need access to the host level
> PIDs.
> * Debug containers that can attach to another container to run strace, gdb,
> etc.
> * Testing Docker on YARN in a container, where the docker socket is bind
> mounted.
> Enabling this feature should be considered privileged as it exposes host
> details inside the container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7064) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324221#comment-16324221 ] Haibo Chen commented on YARN-7064: -- bq. I think the logging inside isAvailable should be enough. I am not in favor of logging the same thing duplicated. Ah, I missed the logging in isAvailable(). Agree with you the current logging is sufficient. bq. Now we will send out the same error message on every tick of ContainersMonitor as you requested Help me understand this a little bit more. Under what circumstances, will getMemorySize() fail for the same reason? We have two problems here whose solution may conflict, misleading metrics because of swallowing exceptions V.S. too much logging if not swallowing exceptions. Regardless of which one we choose to favor, I think no design change to ContainersMonitorImpl is necessary, that is we just need to decide if we want to swallow exceptions in CGroupResourceCalculator.updateProcessTree(). > Use cgroup to get container resource utilization > > > Key: YARN-7064 > URL: https://issues.apache.org/jira/browse/YARN-7064 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-7064.000.patch, YARN-7064.001.patch, > YARN-7064.002.patch, YARN-7064.003.patch, YARN-7064.004.patch, > YARN-7064.005.patch, YARN-7064.007.patch, YARN-7064.008.patch, > YARN-7064.009.patch, YARN-7064.010.patch, YARN-7064.011.patch > > > This is an addendum to YARN-6668. What happens is that that jira always wants > to rebase patches against YARN-1011 instead of trunk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7731) RegistryDNS should handle upstream DNS returning CNAME
[ https://issues.apache.org/jira/browse/YARN-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324258#comment-16324258 ] Billie Rinaldi commented on YARN-7731: -- +1 for patch 003. I will commit this with whitespace=fix. > RegistryDNS should handle upstream DNS returning CNAME > -- > > Key: YARN-7731 > URL: https://issues.apache.org/jira/browse/YARN-7731 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Eric Yang > Attachments: YARN-7731.001.patch, YARN-7731.002.patch, > YARN-7731.003.patch > > > When RegistryDNS performs a lookup in an upstream DNS server and a CNAME > record is retrieved, it returns a response with only the CNAME record (there > is no A record, meaning no IP address is resolved). RegistryDNS should > perform a lookup on the new name from the CNAME record in an attempt to find > an A record, which would provide an IP address. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7064) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324269#comment-16324269 ] Miklos Szegedi commented on YARN-7064: -- The asflicense and unit test issues are unrelated. bq. Under what circumstances, will getMemorySize() fail for the same reason? The cgroup might not have been created, yet, the Linux distribution or kernel build could have a limitation like missing swap configuration, there could be a cgroup configuration change, or configuration issue, where not all subsystems are mounted. > Use cgroup to get container resource utilization > > > Key: YARN-7064 > URL: https://issues.apache.org/jira/browse/YARN-7064 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-7064.000.patch, YARN-7064.001.patch, > YARN-7064.002.patch, YARN-7064.003.patch, YARN-7064.004.patch, > YARN-7064.005.patch, YARN-7064.007.patch, YARN-7064.008.patch, > YARN-7064.009.patch, YARN-7064.010.patch, YARN-7064.011.patch > > > This is an addendum to YARN-6668. What happens is that that jira always wants > to rebase patches against YARN-1011 instead of trunk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6486) FairScheduler: Deprecate continuous scheduling
[ https://issues.apache.org/jira/browse/YARN-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324274#comment-16324274 ] Yufei Gu commented on YARN-6486: I'm not a fan of how RM_SCHEDULER_INCREMENT_ALLOCATION_* and the defaults were deprecated. It makes sense to deprecate every field in case we miss some things. It is not a big thing though. I would prefer to deprecate the ones related to continuous scheduling in this JIRA and file JIRAs for others. > FairScheduler: Deprecate continuous scheduling > -- > > Key: YARN-6486 > URL: https://issues.apache.org/jira/browse/YARN-6486 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > Attachments: YARN-6486.001.patch, YARN-6486.002.patch, > YARN-6486.003.patch, YARN-6486.004.patch, YARN-6486.005.patch > > > Mark continuous scheduling as deprecated in 2.9 and remove the code in 3.0. > Removing continuous scheduling from the code will be logged as a separate jira -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7729) Add support for setting the PID namespace mode
[
https://issues.apache.org/jira/browse/YARN-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324320#comment-16324320
]
genericqa commented on YARN-7729:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 237 unchanged - 0 fixed = 239 total (was 237) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 19s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
38s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7729 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905890/YARN-7729.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 1f44185ab8e2 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Lin
[jira] [Commented] (YARN-7731) RegistryDNS should handle upstream DNS returning CNAME
[ https://issues.apache.org/jira/browse/YARN-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324343#comment-16324343 ] Hudson commented on YARN-7731: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13486 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13486/]) YARN-7731. RegistryDNS should handle upstream DNS returning CNAME. (billie: rev 4fb1f45f21916ca1b1fc6652a2ad562ac996b7b8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/dns/RegistryDNS.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/server/dns/TestRegistryDNS.java > RegistryDNS should handle upstream DNS returning CNAME > -- > > Key: YARN-7731 > URL: https://issues.apache.org/jira/browse/YARN-7731 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Eric Yang > Attachments: YARN-7731.001.patch, YARN-7731.002.patch, > YARN-7731.003.patch > > > When RegistryDNS performs a lookup in an upstream DNS server and a CNAME > record is retrieved, it returns a response with only the CNAME record (there > is no A record, meaning no IP address is resolved). RegistryDNS should > perform a lookup on the new name from the CNAME record in an attempt to find > an A record, which would provide an IP address. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3660) [GPG] Federation Global Policy Generator (service hook only)
[ https://issues.apache.org/jira/browse/YARN-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-3660: --- Summary: [GPG] Federation Global Policy Generator (service hook only) (was: [GPG] Federation Global Policy Generator (load balancing)) > [GPG] Federation Global Policy Generator (service hook only) > > > Key: YARN-3660 > URL: https://issues.apache.org/jira/browse/YARN-3660 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Botong Huang > Labels: federation, gpg > Attachments: YARN-3660-YARN-7402.v1.patch > > > In a federated environment, local impairments of one sub-cluster might > unfairly affect users/queues that are mapped to that sub-cluster. A > centralized component (GPG) runs out-of-band and edits the policies governing > how users/queues are allocated to sub-clusters. This allows us to enforce > global invariants (by dynamically updating locally-enforced invariants). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3660) [GPG] Federation Global Policy Generator (service hook only)
[ https://issues.apache.org/jira/browse/YARN-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324362#comment-16324362 ] Carlo Curino commented on YARN-3660: [~botong] the patch seems generally ok but please: # Address checkstyle issues and write some good Javadoc for the top classes # Please provide some basic test, even just that the boot is correct or fails on disable federation, the empty Test class is not ok (better none at all and explain why when QA complaints). # This JIRA is an empty service that you will need in other patches, let's rename the JIRA (my attempt is meah if you can find a better title) # The patch is reasonably small, so please include the bash/powershel scripts to start/stop/restart the GPG (you can look at what was done for the Federation Router) > [GPG] Federation Global Policy Generator (service hook only) > > > Key: YARN-3660 > URL: https://issues.apache.org/jira/browse/YARN-3660 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Botong Huang > Labels: federation, gpg > Attachments: YARN-3660-YARN-7402.v1.patch > > > In a federated environment, local impairments of one sub-cluster might > unfairly affect users/queues that are mapped to that sub-cluster. A > centralized component (GPG) runs out-of-band and edits the policies governing > how users/queues are allocated to sub-clusters. This allows us to enforce > global invariants (by dynamically updating locally-enforced invariants). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6648) [GPG] Add FederationStateStore interfaces for Global Policy Generator
[ https://issues.apache.org/jira/browse/YARN-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324373#comment-16324373 ] Carlo Curino commented on YARN-6648: [~botong] the changes in this JIRA seem fine/harmless. However, since I don't see the code that will use them are a bit pointless as is. There is a bit of a trade off between breaking things down in small easy to review JIRAs and keeping things together so that changes are justified. In this case, I think we might have been over-zealous in keeping patches small. Please combine this with the JIRA that uses them,and mark this as duplicate. > [GPG] Add FederationStateStore interfaces for Global Policy Generator > - > > Key: YARN-6648 > URL: https://issues.apache.org/jira/browse/YARN-6648 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Labels: federation, gpg > Attachments: YARN-6648-YARN-2915.v1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7731) RegistryDNS should handle upstream DNS returning CNAME
[ https://issues.apache.org/jira/browse/YARN-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324378#comment-16324378 ] Eric Yang commented on YARN-7731: - [~billie.rinaldi] Thanks Billie. > RegistryDNS should handle upstream DNS returning CNAME > -- > > Key: YARN-7731 > URL: https://issues.apache.org/jira/browse/YARN-7731 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Billie Rinaldi >Assignee: Eric Yang > Fix For: 3.1.0 > > Attachments: YARN-7731.001.patch, YARN-7731.002.patch, > YARN-7731.003.patch > > > When RegistryDNS performs a lookup in an upstream DNS server and a CNAME > record is retrieved, it returns a response with only the CNAME record (there > is no A record, meaning no IP address is resolved). RegistryDNS should > perform a lookup on the new name from the CNAME record in an attempt to find > an A record, which would provide an IP address. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7671) Improve Diagonstic message for stop yarn native service
[
https://issues.apache.org/jira/browse/YARN-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-7671:
Attachment: YARN-7671.001.patch
Patch 1 includes the suggested fix
> Improve Diagonstic message for stop yarn native service
>
>
> Key: YARN-7671
> URL: https://issues.apache.org/jira/browse/YARN-7671
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Chandni Singh
> Attachments: YARN-7671.001.patch
>
>
> Steps:
> 1) Install Hadoop 3.0 cluster
> 2) Run Yarn service application
> {code:title=sleeper.json}{
> "name": "sleeper-service",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 1,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
>}
> }
> ]
> }{code}
> {code:title=cmd}
> yarn app -launch my-sleeper1 sleeper.json{code}
> 3) stop yarn service app
> {code:title=cmd}
> yarn app -stop my-sleeper1{code}
> On stopping yarn service, appId finishes with YarnApplicationState: FINISHED
> , FinalStatus Reported by AM: ENDED and Diagnostics: Navigate to the
> failed component for more details.
> Here, Diagnostics message should be improved. When an application is
> explicitly stopped by user, the diagnostics message should say " Application
> stopped by user"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7724) yarn application status should support application name
[ https://issues.apache.org/jira/browse/YARN-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7724: -- Attachment: YARN-7724.04.patch Thanks Billie for review, uploaded a new patch > yarn application status should support application name > --- > > Key: YARN-7724 > URL: https://issues.apache.org/jira/browse/YARN-7724 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-7724.01.patch, YARN-7724.02.patch, > YARN-7724.03.patch, YARN-7724.04.patch > > > Yarn Service application are tied with app name. Thus, yarn application > -status should be able to take yarn service name as argument such as > yarn application -status -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324398#comment-16324398 ] Hudson commented on YARN-5366: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13487 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13487/]) YARN-5366. Improve signal handling and delete delay for Docker on Yarn. (eyang: rev 3d65dbe032e202361d613344ccc6d9c5f99ba395) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerKillCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/executor/TestContainerReapContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/path-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/TestNMProtoUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerKillCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommandExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerReapContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-path-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/path-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/NMProtoUtils.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/
[jira] [Updated] (YARN-7729) Add support for setting the PID namespace mode
[
https://issues.apache.org/jira/browse/YARN-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billie Rinaldi updated YARN-7729:
-
Attachment: YARN-7729.002.patch
> Add support for setting the PID namespace mode
> --
>
> Key: YARN-7729
> URL: https://issues.apache.org/jira/browse/YARN-7729
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Shane Kumpf
>Assignee: Billie Rinaldi
> Attachments: YARN-7729.001.patch, YARN-7729.002.patch
>
>
> Docker has support for allowing containers to share the PID namespace with
> the host or other containers via the {{docker run --pid}} flag.
> There are a number of use cases where this is desirable:
> * Monitoring tools running in containers that need access to the host level
> PIDs.
> * Debug containers that can attach to another container to run strace, gdb,
> etc.
> * Testing Docker on YARN in a container, where the docker socket is bind
> mounted.
> Enabling this feature should be considered privileged as it exposes host
> details inside the container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324413#comment-16324413
]
Shane Kumpf commented on YARN-7654:
---
I can work on this as a follow on to the recovery and clean up lifecycle work.
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Shane Kumpf
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned YARN-7654:
---
Assignee: Shane Kumpf
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Shane Kumpf
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324414#comment-16324414
]
genericqa commented on YARN-7738:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 29s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 35s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7dd03ab48ebf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b278f7b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-
[jira] [Commented] (YARN-7739) Revisit scheduler resource normalization behavior for max allocation
[ https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324431#comment-16324431 ] Wangda Tan commented on YARN-7739: -- Thanks [~jlowe] for the comment, I agree with all your points. Regarding the fix of YARN-2604, I think the proper behavior is to set rejected ResourceRequest to AllocateResponse, we added rejected SchedulingRequest to AllocatedResponse in YARN-6592 branch. It doesn't look like a proper fix to automatically give app shrunk resources, many apps don't check if the allocated resource is same as requested, so we saw some issues which app requests 400 GB containers but only received 150 GB container. I personally prefer to disable this behavior by default and get rid of this in 3.1.0. By default, the maximum allocation will be determined by preconfigured cluster maximum allocation and per-queue maximum allocation. Any objections of doing this? > Revisit scheduler resource normalization behavior for max allocation > > > Key: YARN-7739 > URL: https://issues.apache.org/jira/browse/YARN-7739 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Priority: Critical > > Currently, YARN Scheduler normalizes requested resource based on the maximum > allocation derived from configured maximum allocation and maximum registered > node resources. Basically, the scheduler will silently cap asked resource by > maximum allocation. > This could cause issues for applications, for example, a Spark job which > needs 12 GB memory to run, however in the cluster, registered NMs have at > most 8 GB mem on each node. So scheduler allocates 8GB memory container to > the requested application. > Once app receives containers from RM, if it doesn't double check allocated > resources, it will lead to OOM and hard to debug because scheduler silently > caps maximum allocation. > When non-mandatory resources introduced, this becomes worse. For resources > like GPU, we typically set minimum allocation to 0 since not all nodes have > GPU devices. So it is possible that application asks 4 GPUs but get 0 GPU, it > gonna be a big problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[ https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7740: -- Attachment: YARN-7740.1.patch > Fix logging for destroy yarn service cli when app does not exist > > > Key: YARN-7740 > URL: https://issues.apache.org/jira/browse/YARN-7740 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Reporter: Yesha Vora > Attachments: YARN-7740.1.patch > > > Scenario: > Run "yarn app -destroy" cli with a application name which does not exist. > Here, The cli should return a message " Application does not exists" instead > it is returning a message "Destroyed cluster httpd-xxx" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3895) Support ACLs in ATSv2
[ https://issues.apache.org/jira/browse/YARN-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324450#comment-16324450 ] Vrushali C commented on YARN-3895: -- Thanks [~jlowe] , I will think over this and get back. > Support ACLs in ATSv2 > - > > Key: YARN-3895 > URL: https://issues.apache.org/jira/browse/YARN-3895 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355 > > This JIRA is to keep track of authorization support design discussions for > both readers and collectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[ https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reassigned YARN-7740: - Assignee: Jian He > Fix logging for destroy yarn service cli when app does not exist > > > Key: YARN-7740 > URL: https://issues.apache.org/jira/browse/YARN-7740 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-7740.1.patch > > > Scenario: > Run "yarn app -destroy" cli with a application name which does not exist. > Here, The cli should return a message " Application does not exists" instead > it is returning a message "Destroyed cluster httpd-xxx" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7671) Improve Diagonstic message for stop yarn native service
[
https://issues.apache.org/jira/browse/YARN-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-7671:
Attachment: YARN-7671.002.patch
> Improve Diagonstic message for stop yarn native service
>
>
> Key: YARN-7671
> URL: https://issues.apache.org/jira/browse/YARN-7671
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Chandni Singh
> Attachments: YARN-7671.001.patch, YARN-7671.002.patch
>
>
> Steps:
> 1) Install Hadoop 3.0 cluster
> 2) Run Yarn service application
> {code:title=sleeper.json}{
> "name": "sleeper-service",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 1,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
>}
> }
> ]
> }{code}
> {code:title=cmd}
> yarn app -launch my-sleeper1 sleeper.json{code}
> 3) stop yarn service app
> {code:title=cmd}
> yarn app -stop my-sleeper1{code}
> On stopping yarn service, appId finishes with YarnApplicationState: FINISHED
> , FinalStatus Reported by AM: ENDED and Diagnostics: Navigate to the
> failed component for more details.
> Here, Diagnostics message should be improved. When an application is
> explicitly stopped by user, the diagnostics message should say " Application
> stopped by user"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[
https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324452#comment-16324452
]
Robert Kanter commented on YARN-7451:
-
[~snemeth], you said offline that Jersey 2 fixes this issue, right? Could you
do a quick look into how much work it would be to upgrade Jersey? If it's not
a big deal, then that would be a good option.
> Resources Types should be visible in the Cluster Apps API "resourceRequests"
> section
>
>
> Key: YARN-7451
> URL: https://issues.apache.org/jira/browse/YARN-7451
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, restapi
>Affects Versions: 3.0.0
>Reporter: Grant Sohn
>Assignee: Szilard Nemeth
> Attachments: YARN-7451.001.patch, YARN-7451.002.patch,
> YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch,
> YARN-7451.006.patch,
> YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch
>
>
> When running jobs that request resource types the RM Cluster Apps API should
> include this in the "resourceRequests" object.
> Additionally, when calling the RM scheduler API it returns:
> {noformat}
> "childQueues": {
> "queue": [
> {
> "allocatedContainers": 101,
> "amMaxResources": {
> "memory": 320390,
> "vCores": 192
> },
> "amUsedResources": {
> "memory": 1024,
> "vCores": 1
> },
> "clusterResources": {
> "memory": 640779,
> "vCores": 384
> },
> "demandResources": {
> "memory": 103424,
> "vCores": 101
> },
> "fairResources": {
> "memory": 640779,
> "vCores": 384
> },
> "maxApps": 2147483647,
> "maxResources": {
> "memory": 640779,
> "vCores": 384
> },
> "minResources": {
> "memory": 0,
> "vCores": 0
> },
> "numActiveApps": 1,
> "numPendingApps": 0,
> "preemptable": true,
> "queueName": "root.users.systest",
> "reservedContainers": 0,
> "reservedResources": {
> "memory": 0,
> "vCores": 0
> },
> "schedulingPolicy": "fair",
> "steadyFairResources": {
> "memory": 320390,
> "vCores": 192
> },
> "type": "fairSchedulerLeafQueueInfo",
> "usedResources": {
> "memory": 103424,
> "vCores": 101
> }
> }
> ]
> {noformat}
> However, the web UI shows resource types usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7221) Add security check for privileged docker container
[ https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324453#comment-16324453 ] Eric Yang commented on YARN-7221: - Can someone review this patch? Thanks > Add security check for privileged docker container > -- > > Key: YARN-7221 > URL: https://issues.apache.org/jira/browse/YARN-7221 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7221.001.patch, YARN-7221.002.patch > > > When a docker is running with privileges, majority of the use case is to have > some program running with root then drop privileges to another user. i.e. > httpd to start with privileged and bind to port 80, then drop privileges to > www user. > # We should add security check for submitting users, to verify they have > "sudo" access to run privileged container. > # We should remove --user=uid:gid for privileged containers. > > Docker can be launched with --privileged=true, and --user=uid:gid flag. With > this parameter combinations, user will not have access to become root user. > All docker exec command will be drop to uid:gid user to run instead of > granting privileges. User can gain root privileges if container file system > contains files that give user extra power, but this type of image is > considered as dangerous. Non-privileged user can launch container with > special bits to acquire same level of root power. Hence, we lose control of > which image should be run with --privileges, and who have sudo rights to use > privileged container images. As the result, we should check for sudo access > then decide to parameterize --privileged=true OR --user=uid:gid. This will > avoid leading developer down the wrong path. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7671) Improve Diagonstic message for stop yarn native service
[
https://issues.apache.org/jira/browse/YARN-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-7671:
Attachment: (was: YARN-7671.002.patch)
> Improve Diagonstic message for stop yarn native service
>
>
> Key: YARN-7671
> URL: https://issues.apache.org/jira/browse/YARN-7671
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Chandni Singh
> Attachments: YARN-7671.001.patch
>
>
> Steps:
> 1) Install Hadoop 3.0 cluster
> 2) Run Yarn service application
> {code:title=sleeper.json}{
> "name": "sleeper-service",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 1,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
>}
> }
> ]
> }{code}
> {code:title=cmd}
> yarn app -launch my-sleeper1 sleeper.json{code}
> 3) stop yarn service app
> {code:title=cmd}
> yarn app -stop my-sleeper1{code}
> On stopping yarn service, appId finishes with YarnApplicationState: FINISHED
> , FinalStatus Reported by AM: ENDED and Diagnostics: Navigate to the
> failed component for more details.
> Here, Diagnostics message should be improved. When an application is
> explicitly stopped by user, the diagnostics message should say " Application
> stopped by user"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7671) Improve Diagonstic message for stop yarn native service
[
https://issues.apache.org/jira/browse/YARN-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-7671:
Attachment: YARN-7671.002.patch
Implemented changes as per [~jianhe] feedback
> Improve Diagonstic message for stop yarn native service
>
>
> Key: YARN-7671
> URL: https://issues.apache.org/jira/browse/YARN-7671
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Chandni Singh
> Attachments: YARN-7671.001.patch, YARN-7671.002.patch
>
>
> Steps:
> 1) Install Hadoop 3.0 cluster
> 2) Run Yarn service application
> {code:title=sleeper.json}{
> "name": "sleeper-service",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 1,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
>}
> }
> ]
> }{code}
> {code:title=cmd}
> yarn app -launch my-sleeper1 sleeper.json{code}
> 3) stop yarn service app
> {code:title=cmd}
> yarn app -stop my-sleeper1{code}
> On stopping yarn service, appId finishes with YarnApplicationState: FINISHED
> , FinalStatus Reported by AM: ENDED and Diagnostics: Navigate to the
> failed component for more details.
> Here, Diagnostics message should be improved. When an application is
> explicitly stopped by user, the diagnostics message should say " Application
> stopped by user"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7741) Eliminate extra log statement from yarn app -destroy cli
[ https://issues.apache.org/jira/browse/YARN-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh reassigned YARN-7741: --- Assignee: Chandni Singh > Eliminate extra log statement from yarn app -destroy cli > > > Key: YARN-7741 > URL: https://issues.apache.org/jira/browse/YARN-7741 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Chandni Singh > > "Yarn destroy -app " cli prints very long stacktrace from zookeeper client > which is not required. > This cli prints 44009 characters (38 lines and 358 words) > This api should only print the message whether app was destroyed successfully > or not. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7743) UI component placement overlaps on Safari
Yesha Vora created YARN-7743: Summary: UI component placement overlaps on Safari Key: YARN-7743 URL: https://issues.apache.org/jira/browse/YARN-7743 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0 Reporter: Yesha Vora Browser: Safari Version 9.1.1 (11601.6.17) Issue with new RM UI: The two tables on application and service page overlaps on Safari browser. Find screenshot for details -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7743) UI component placement overlaps on Safari
[ https://issues.apache.org/jira/browse/YARN-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-7743: - Attachment: Screen Shot 2018-01-11 at 5.20.48 PM.png Screen Shot 2018-01-11 at 5.17.18 PM.png > UI component placement overlaps on Safari > - > > Key: YARN-7743 > URL: https://issues.apache.org/jira/browse/YARN-7743 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Yesha Vora > Attachments: Screen Shot 2018-01-11 at 5.17.18 PM.png, Screen Shot > 2018-01-11 at 5.20.48 PM.png > > > Browser: Safari Version 9.1.1 (11601.6.17) > Issue with new RM UI: > The two tables on application and service page overlaps on Safari browser. > Find screenshot for details -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for untrusted docker image
[
https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324484#comment-16324484
]
Eric Badger commented on YARN-7516:
---
[~eyang], thanks for the patch.
{noformat}
+// Disable set privileged if image is not trusted.
+if (check_trusted_image(command_config, conf) != 0) {
+ ret = PRIVILEGED_CONTAINERS_DISABLED;
+ goto free_and_exit;
{noformat}
To be consistent with the rest of the code, this should print a message that
privileged containers are disabled.
{noformat}
+file_cmd_vec.push_back(std::make_pair(
+"[docker-command-execution]\n"
+" docker-command=run\n name=container_e1_12312_1_02_01\n
image=docker-image\n user=test\n hostname=host-id\n"
+"
ro-mounts=/var/log:/var/log,/var/lib:/lib,/usr/bin/cut:/usr/bin/cut\n
rw-mounts=/tmp:/tmp\n"
+" network=bridge\n devices=/dev/test:/dev/test\n net=bridge\n"
+" cap-add=CHOWN,SETUID\n cgroup-parent=ctr-cgroup\n
detach=true\n rm=true\n group-add=1000,1001\n"
+" launch-command=bash,test_script.sh,arg1,arg2",
+"run --name='container_e1_12312_1_02_01' --user='test' -d --rm
--net='bridge' -v '/var/log:/var/log:ro' -v '/var/lib:/lib:ro'"
+" -v '/usr/bin/cut:/usr/bin/cut:ro' -v '/tmp:/tmp'
--cgroup-parent='ctr-cgroup' --cap-drop='ALL' "
+"--cap-add='CHOWN' --cap-add='SETUID' --hostname='host-id'
--group-add '1000' --group-add '1001' "
+"--device='/dev/test:/dev/test' 'docker-image' 'bash'
'test_script.sh' 'arg1' 'arg2' "));
{noformat}
This test should fail, but doesn't. As I understand the patch, since the image
is not from a trusted registry, it should fail because of adding devices,
capabilities, or mounts. However, since it isn't asking for privilege it
bypasses the trusted registry check.
{noformat:title=check_trusted_image()}
if (value != NULL && strcmp(value, "true") == 0) {
{noformat}
I think this is the line that is causing the error. We are only actually doing
the trusted image check if we're asking for privilege.
> Security check for untrusted docker image
> -
>
> Key: YARN-7516
> URL: https://issues.apache.org/jira/browse/YARN-7516
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7516.001.patch, YARN-7516.002.patch,
> YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch,
> YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch
>
>
> Hadoop YARN Services can support using private docker registry image or
> docker image from docker hub. In current implementation, Hadoop security is
> enforced through username and group membership, and enforce uid:gid
> consistency in docker container and distributed file system. There is cloud
> use case for having ability to run untrusted docker image on the same cluster
> for testing.
> The basic requirement for untrusted container is to ensure all kernel and
> root privileges are dropped, and there is no interaction with distributed
> file system to avoid contamination. We can probably enforce detection of
> untrusted docker image by checking the following:
> # If docker image is from public docker hub repository, the container is
> automatically flagged as insecure, and disk volume mount are disabled
> automatically, and drop all kernel capabilities.
> # If docker image is from private repository in docker hub, and there is a
> white list to allow the private repository, disk volume mount is allowed,
> kernel capabilities follows the allowed list.
> # If docker image is from private trusted registry with image name like
> "private.registry.local:5000/centos", and white list allows this private
> trusted repository. Disk volume mount is allowed, kernel capabilities
> follows the allowed list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for untrusted docker image
[
https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324489#comment-16324489
]
Eric Badger commented on YARN-7516:
---
Also, going back to the registry parsing, what about the case where I build an
image on the node myself and then tag it with a name. Think about a single node
cluster or similar or manually pushing images to every node. If I do that, then
{{docker.privileged-containers.registries}} would prevent you from using the
image as privileged since it didn't come from a registry. It would just be
local to the node. I'm not sure that parsing for a {{/}} delimiter is the best
method here.
> Security check for untrusted docker image
> -
>
> Key: YARN-7516
> URL: https://issues.apache.org/jira/browse/YARN-7516
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7516.001.patch, YARN-7516.002.patch,
> YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch,
> YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch
>
>
> Hadoop YARN Services can support using private docker registry image or
> docker image from docker hub. In current implementation, Hadoop security is
> enforced through username and group membership, and enforce uid:gid
> consistency in docker container and distributed file system. There is cloud
> use case for having ability to run untrusted docker image on the same cluster
> for testing.
> The basic requirement for untrusted container is to ensure all kernel and
> root privileges are dropped, and there is no interaction with distributed
> file system to avoid contamination. We can probably enforce detection of
> untrusted docker image by checking the following:
> # If docker image is from public docker hub repository, the container is
> automatically flagged as insecure, and disk volume mount are disabled
> automatically, and drop all kernel capabilities.
> # If docker image is from private repository in docker hub, and there is a
> white list to allow the private repository, disk volume mount is allowed,
> kernel capabilities follows the allowed list.
> # If docker image is from private trusted registry with image name like
> "private.registry.local:5000/centos", and white list allows this private
> trusted repository. Disk volume mount is allowed, kernel capabilities
> follows the allowed list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7221) Add security check for privileged docker container
[ https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324496#comment-16324496 ] Eric Badger commented on YARN-7221: --- Hi, [~eyang], I will review this when I get a chance. Probably early next week. > Add security check for privileged docker container > -- > > Key: YARN-7221 > URL: https://issues.apache.org/jira/browse/YARN-7221 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7221.001.patch, YARN-7221.002.patch > > > When a docker is running with privileges, majority of the use case is to have > some program running with root then drop privileges to another user. i.e. > httpd to start with privileged and bind to port 80, then drop privileges to > www user. > # We should add security check for submitting users, to verify they have > "sudo" access to run privileged container. > # We should remove --user=uid:gid for privileged containers. > > Docker can be launched with --privileged=true, and --user=uid:gid flag. With > this parameter combinations, user will not have access to become root user. > All docker exec command will be drop to uid:gid user to run instead of > granting privileges. User can gain root privileges if container file system > contains files that give user extra power, but this type of image is > considered as dangerous. Non-privileged user can launch container with > special bits to acquire same level of root power. Hence, we lose control of > which image should be run with --privileges, and who have sudo rights to use > privileged container images. As the result, we should check for sudo access > then decide to parameterize --privileged=true OR --user=uid:gid. This will > avoid leading developer down the wrong path. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[ https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7740: -- Attachment: YARN-7740.2.patch > Fix logging for destroy yarn service cli when app does not exist > > > Key: YARN-7740 > URL: https://issues.apache.org/jira/browse/YARN-7740 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-7740.1.patch, YARN-7740.2.patch > > > Scenario: > Run "yarn app -destroy" cli with a application name which does not exist. > Here, The cli should return a message " Application does not exists" instead > it is returning a message "Destroyed cluster httpd-xxx" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7728) Expose and expand container preemptions in Capacity Scheduler queue metrics
[ https://issues.apache.org/jira/browse/YARN-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-7728: - Attachment: YARN-7728.002.patch > Expose and expand container preemptions in Capacity Scheduler queue metrics > --- > > Key: YARN-7728 > URL: https://issues.apache.org/jira/browse/YARN-7728 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0, 2.8.3, 3.0.0 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: YARN-7728.001.patch, YARN-7728.002.patch > > > YARN-1047 exposed queue metrics for the number of preempted containers to the > fair scheduler. I would like to also expose these to the capacity scheduler > and add metrics for the amount of lost memory seconds and vcore seconds. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7739) Revisit scheduler resource normalization behavior for max allocation
[ https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324511#comment-16324511 ] Jason Lowe commented on YARN-7739: -- I'm OK with getting rid of implicitly shrinking containers since I think it is causing more problems than it is solving, but it is technically an incompatible change. If there was some app out there that was relying on asking for a GB container and then expecting it to be silently truncated to the cluster/queue's max allocation then it will break after that change. >From a compatibility standpoint we could add a flag to the request indicating >whether the request is OK with silent truncation, but most apps would have to >be updated to set the flag saying it's _not_ OK to truncate to get the >behavior most apps would want which is no truncation. Not a fan of that >either. IMHO truncation is a bug, so maybe it's not so much an incompatible >change as it is a bugfix. ;-) I would recommend pinging [~vinodkv] about this since I believe he was involved in the early days discussions that led to the silent truncation to max behavior. > Revisit scheduler resource normalization behavior for max allocation > > > Key: YARN-7739 > URL: https://issues.apache.org/jira/browse/YARN-7739 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Priority: Critical > > Currently, YARN Scheduler normalizes requested resource based on the maximum > allocation derived from configured maximum allocation and maximum registered > node resources. Basically, the scheduler will silently cap asked resource by > maximum allocation. > This could cause issues for applications, for example, a Spark job which > needs 12 GB memory to run, however in the cluster, registered NMs have at > most 8 GB mem on each node. So scheduler allocates 8GB memory container to > the requested application. > Once app receives containers from RM, if it doesn't double check allocated > resources, it will lead to OOM and hard to debug because scheduler silently > caps maximum allocation. > When non-mandatory resources introduced, this becomes worse. For resources > like GPU, we typically set minimum allocation to 0 since not all nodes have > GPU devices. So it is possible that application asks 4 GPUs but get 0 GPU, it > gonna be a big problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[
https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324529#comment-16324529
]
genericqa commented on YARN-7740:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 37s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
23s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7740 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905912/YARN-7740.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2fba6edc48ac 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d65dbe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19228/testReport/ |
| Max. process+thread count | 627 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop
[jira] [Commented] (YARN-7724) yarn application status should support application name
[
https://issues.apache.org/jira/browse/YARN-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324531#comment-16324531
]
genericqa commented on YARN-7724:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 248 unchanged - 1 fixed = 257 total (was 249) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
19s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m
49s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
38s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7724 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905906/YARN-7
[jira] [Commented] (YARN-7671) Improve Diagonstic message for stop yarn native service
[
https://issues.apache.org/jira/browse/YARN-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324533#comment-16324533
]
genericqa commented on YARN-7671:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
27s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7671 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905914/YARN-7671.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux dfa27e8b6490 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d65dbe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19229/testReport/ |
| Max. process+thread count | 622 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19229/console |
| Po
[jira] [Commented] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[
https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324544#comment-16324544
]
genericqa commented on YARN-7740:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 1 new + 38 unchanged - 2 fixed = 39 total (was 40) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 37s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7740 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905919/YARN-7740.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b63d96379724 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d65dbe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19230/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| unit
[jira] [Commented] (YARN-2185) Use pipes when localizing archives
[
https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324545#comment-16324545
]
Jason Lowe commented on YARN-2185:
--
Thanks for updating the patch!
bq. I was wondering, if it should be better for OS specific archives containing
symlinks for example.
symlinks is a good example of why it is important to use {{tar}} on Linux.
However if we're going to use OS tools to support ZIP files properly then we
should not be using the {{jar}} command. {{jar}} does not properly support
symlinks in a ZIP archive but {{unzip}} does. To be honest I don't see the
appeal of using the jar command for anything since I do not see how it is going
to do anything different than the JDK builtin support since it is part of the
JDK. As with the parallel copying and timestamp ignore support, I think it may
be best to have this patch focus on using pipes rather than also changing the
semantics of what tools are used to unpack the archives. We can discuss what
should be done with jars and zips in another JIRA.
Speaking of symbolic links, now that all supported releases are on JDK7 or
later, we can update unTarUsingJava to fix the lacking symlink support. I
filed HADOOP-15170 to track that.
The failure on the path containing a single quote seems overly paranoid.
Single quotes can be escaped in the POSIX shell code path per my previous
comment, e.g.:
{code}
filename.replace("'","'\\''")
{code}
Rather than do the subshell with an extra {{cd}}, could it be simplified to
just {{ tar -C }}?
It doesn't look like the following comment was addressed as runCommandOnStream
is still creating an executor sometimes yet does not clean it up in those
cases. The Javadoc should also be updated to document the deadlock potential.
{quote}
I think we should shutdown the executor service if we created it, otherwise we
risk running out of threads before the garbage collector gets around to
noticing it can free everything up. Either that or require the caller to
provide the executor.
{quote}
Speaking of deadlock, if debug logging is not enabled then this can deadlock in
the manner I described above. That's one of the reasons why I think leveraging
Shell isn't a terrible idea. Wielding ProcessBuilder and Process properly is
trickier than most people believe. It also side-steps a lot of the executor
management stuff. I'm OK if you still think it's better to roll our own here.
"parallelVerifyAndCopy" is no longer parallel so the name and javadoc is
misleading. It also does not throw InterruptedException nor ExecutionException.
It's weird to allow the user to specify a downloader thread count yet we setup
and teardown the executor for every unpack call and the unpack call is only
going to use two threads. Essentially it shouldn't be a config since any value
but 2 is either wrong or wasteful.
Speaking of executors, would it makes more sense for FSDownload to be able to
leverage an existing executor rather than creating its own? All of the
existing usages of FSDownload are using an executor of one form or another, so
it'd be nice not to have to keep building and tearing these down for every call.
I'm still confuse about the recursive destination directory delete that was
added. It was not there before, and I don't think it's appropriate here. What
are we worried is going to be there, and if somehow something is there, are we
sure it's OK to just blindly remove it?
destionationTmp s/b destinationTmp
FSDownloadException is no longer needed.
> Use pipes when localizing archives
> --
>
> Key: YARN-2185
> URL: https://issues.apache.org/jira/browse/YARN-2185
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Miklos Szegedi
> Attachments: YARN-2185.000.patch, YARN-2185.001.patch,
> YARN-2185.002.patch, YARN-2185.003.patch, YARN-2185.004.patch
>
>
> Currently the nodemanager downloads an archive to a local file, unpacks it,
> and then removes it. It would be more efficient to stream the data as it's
> being unpacked to avoid both the extra disk space requirements and the
> additional disk activity from storing the archive.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for untrusted docker image
[
https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324568#comment-16324568
]
Eric Yang commented on YARN-7516:
-
[~ebadger] Thanks for the review.
{quote}
To be consistent with the rest of the code, this should print a message that
privileged containers are disabled.
{quote}
This will be fixed.
{quote}
This test should fail, but doesn't. As I understand the patch, since the image
is not from a trusted registry, it should fail because of adding devices,
capabilities, or mounts. However, since it isn't asking for privilege it
bypasses the trusted registry check.
{quote}
Good catch here. I was under the impression that if the image doesn't ask for
a privileged container, and running as specific user, we will let them interact
with host device. I see the flawed in my logic, they could have sticky bit
binary to gain root and make changes to attached device. Hence, we will denied
them of cap-add/cap-drop and device attachment from untrusted image even if
they didn't ask for privileged container.
{quote}
Also, going back to the registry parsing, what about the case where I build an
image on the node myself and then tag it with a name. Think about a single node
cluster or similar or manually pushing images to every node. If I do that, then
docker.privileged-containers.registries would prevent you from using the image
as privileged since it didn't come from a registry. It would just be local to
the node. I'm not sure that parsing for a / delimiter is the best method here.
{quote}
We can add docker.privileged-containers.registries=* as the default. If
wildcard is detected, then check_trusted_image check is by passed completely
and allow any image to run with privileged mode. Alternatively, user can tag
the locally build image with trusted_registry/image name. This allows user to
define docker.privileged-containers.registries=trusted_registry and use the
locally built image without push image to a trusted registry. I am kind of in
favor of second solution to prevent accidental allowance of privileged
containers. What is your preference?
> Security check for untrusted docker image
> -
>
> Key: YARN-7516
> URL: https://issues.apache.org/jira/browse/YARN-7516
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7516.001.patch, YARN-7516.002.patch,
> YARN-7516.003.patch, YARN-7516.004.patch, YARN-7516.005.patch,
> YARN-7516.006.patch, YARN-7516.007.patch, YARN-7516.008.patch
>
>
> Hadoop YARN Services can support using private docker registry image or
> docker image from docker hub. In current implementation, Hadoop security is
> enforced through username and group membership, and enforce uid:gid
> consistency in docker container and distributed file system. There is cloud
> use case for having ability to run untrusted docker image on the same cluster
> for testing.
> The basic requirement for untrusted container is to ensure all kernel and
> root privileges are dropped, and there is no interaction with distributed
> file system to avoid contamination. We can probably enforce detection of
> untrusted docker image by checking the following:
> # If docker image is from public docker hub repository, the container is
> automatically flagged as insecure, and disk volume mount are disabled
> automatically, and drop all kernel capabilities.
> # If docker image is from private repository in docker hub, and there is a
> white list to allow the private repository, disk volume mount is allowed,
> kernel capabilities follows the allowed list.
> # If docker image is from private trusted registry with image name like
> "private.registry.local:5000/centos", and white list allows this private
> trusted repository. Disk volume mount is allowed, kernel capabilities
> follows the allowed list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7724) yarn application status should support application name
[ https://issues.apache.org/jira/browse/YARN-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324571#comment-16324571 ] Billie Rinaldi commented on YARN-7724: -- I'll rebuild the precommit build; it doesn't seem like that shadedclient error is due to this patch. > yarn application status should support application name > --- > > Key: YARN-7724 > URL: https://issues.apache.org/jira/browse/YARN-7724 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-7724.01.patch, YARN-7724.02.patch, > YARN-7724.03.patch, YARN-7724.04.patch > > > Yarn Service application are tied with app name. Thus, yarn application > -status should be able to take yarn service name as argument such as > yarn application -status -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[ https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7740: -- Attachment: YARN-7740.3.patch > Fix logging for destroy yarn service cli when app does not exist > > > Key: YARN-7740 > URL: https://issues.apache.org/jira/browse/YARN-7740 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-7740.1.patch, YARN-7740.2.patch, YARN-7740.3.patch > > > Scenario: > Run "yarn app -destroy" cli with a application name which does not exist. > Here, The cli should return a message " Application does not exists" instead > it is returning a message "Destroyed cluster httpd-xxx" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7738) CapacityScheduler: Support refresh maximum allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324578#comment-16324578
]
genericqa commented on YARN-7738:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 30s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7738 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e380e3d87536 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git r
[jira] [Assigned] (YARN-7737) prelaunch.err file not found exception on container failure
[
https://issues.apache.org/jira/browse/YARN-7737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang reassigned YARN-7737:
---
Assignee: Keqiu Hu (was: Jonathan Hung)
> prelaunch.err file not found exception on container failure
> ---
>
> Key: YARN-7737
> URL: https://issues.apache.org/jira/browse/YARN-7737
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Hung
>Assignee: Keqiu Hu
>
> Hit this exception when a container failed:{noformat}2018-01-11 19:04:08,036
> ERROR
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Failed to get tail of the container's prelaunch error log file
> java.io.FileNotFoundException: File
> /grid/b/tmp/userlogs/application_1515190594800_1766/container_e39_1515190594800_1766_01_02/prelaunch.err
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:545)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:511)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:93)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){noformat}
> containerLogDir is picked on container launch via
> {{LocalDirAllocator#getLocalPathForWrite}}, which is where it looks for
> {{prelaunch.err}} when the container fails. But prelaunch.err (and
> prelaunch.out) are created in the first log dir (in {{ContainerLaunch#call}}:
> {noformat}exec.writeLaunchEnv(containerScriptOutStream, environment,
> localResources, launchContext.getCommands(),
> new Path(containerLogDirs.get(0)), user);{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7712) Add ability to ignore timestamps in localized files
[
https://issues.apache.org/jira/browse/YARN-7712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324620#comment-16324620
]
Chris Douglas commented on YARN-7712:
-
Got it. The purpose of the timestamp is not security, but correctness. It does
not support applications that might specify a region of a dependency (e.g.,
download a segment of a log file being appended to) or on a dependency that
does not exist during submission. It is sufficient for static dependencies
(e.g., jars) that are uploaded prior to submission, and to avoid the NM linking
a stale version of a resource for a new container. The only security guarantees
come from the {{FileSystem}}.
You mentioned the REST APIs a couple times. Why are those problematic?
If this is purely for testing, one could use a {{FilterFileSystem}} that
returns a constant for the modification time, rather than modifying YARN...
> Add ability to ignore timestamps in localized files
> ---
>
> Key: YARN-7712
> URL: https://issues.apache.org/jira/browse/YARN-7712
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>
> YARN currently requires and checks the timestamp of localized files and
> fails, if the file on HDFS does not match to the one requested. This jira
> adds the ability to ignore the timestamp based on the request of the client.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7728) Expose and expand container preemptions in Capacity Scheduler queue metrics
[
https://issues.apache.org/jira/browse/YARN-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324624#comment-16324624
]
genericqa commented on YARN-7728:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 144 unchanged - 0 fixed = 146 total (was 144)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m
2s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7728 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905922/YARN-7728.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 51d954c543b5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d65dbe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19231/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19231/testReport/ |
| Max. process+thread count | 846 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resou
[jira] [Commented] (YARN-7729) Add support for setting the PID namespace mode
[
https://issues.apache.org/jira/browse/YARN-7729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324627#comment-16324627
]
genericqa commented on YARN-7729:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
40s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 27m
48s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
33s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 16m
20s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
34s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
5s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7729 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905907/YARN-7729.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml cc |
| uname | Linux e8781
[jira] [Updated] (YARN-3660) [GPG] Federation Global Policy Generator (service hook only)
[ https://issues.apache.org/jira/browse/YARN-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-3660: --- Attachment: YARN-3660-YARN-7402.v2.patch > [GPG] Federation Global Policy Generator (service hook only) > > > Key: YARN-3660 > URL: https://issues.apache.org/jira/browse/YARN-3660 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Botong Huang > Labels: federation, gpg > Attachments: YARN-3660-YARN-7402.v1.patch, > YARN-3660-YARN-7402.v2.patch > > > In a federated environment, local impairments of one sub-cluster might > unfairly affect users/queues that are mapped to that sub-cluster. A > centralized component (GPG) runs out-of-band and edits the policies governing > how users/queues are allocated to sub-clusters. This allows us to enforce > global invariants (by dynamically updating locally-enforced invariants). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7740) Fix logging for destroy yarn service cli when app does not exist
[
https://issues.apache.org/jira/browse/YARN-7740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324639#comment-16324639
]
genericqa commented on YARN-7740:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 1 new + 38 unchanged - 2 fixed = 39 total (was 40) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 35s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7740 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905933/YARN-7740.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4851c4c703e3 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3d65dbe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/19233/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19233/testReport/ |
| M
[jira] [Commented] (YARN-3660) [GPG] Federation Global Policy Generator (service hook only)
[ https://issues.apache.org/jira/browse/YARN-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324644#comment-16324644 ] Botong Huang commented on YARN-3660: Thanks [~curino] for the quick review! v2 patch uploaded > [GPG] Federation Global Policy Generator (service hook only) > > > Key: YARN-3660 > URL: https://issues.apache.org/jira/browse/YARN-3660 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Botong Huang > Labels: federation, gpg > Attachments: YARN-3660-YARN-7402.v1.patch, > YARN-3660-YARN-7402.v2.patch > > > In a federated environment, local impairments of one sub-cluster might > unfairly affect users/queues that are mapped to that sub-cluster. A > centralized component (GPG) runs out-of-band and edits the policies governing > how users/queues are allocated to sub-clusters. This allows us to enforce > global invariants (by dynamically updating locally-enforced invariants). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7671) Improve Diagonstic message for stop yarn native service
[
https://issues.apache.org/jira/browse/YARN-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324661#comment-16324661
]
Hudson commented on YARN-7671:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13488 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13488/])
YARN-7671. Improve Diagonstic message for stop yarn native service. (jianhe:
rev c05b84703b7754b6c2fbcec39f22b5d7802563ec)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ClientAMService.java
> Improve Diagonstic message for stop yarn native service
>
>
> Key: YARN-7671
> URL: https://issues.apache.org/jira/browse/YARN-7671
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Chandni Singh
> Fix For: 3.1.0
>
> Attachments: YARN-7671.001.patch, YARN-7671.002.patch
>
>
> Steps:
> 1) Install Hadoop 3.0 cluster
> 2) Run Yarn service application
> {code:title=sleeper.json}{
> "name": "sleeper-service",
> "components" :
> [
> {
> "name": "sleeper",
> "number_of_containers": 1,
> "launch_command": "sleep 90",
> "resource": {
> "cpus": 1,
> "memory": "256"
>}
> }
> ]
> }{code}
> {code:title=cmd}
> yarn app -launch my-sleeper1 sleeper.json{code}
> 3) stop yarn service app
> {code:title=cmd}
> yarn app -stop my-sleeper1{code}
> On stopping yarn service, appId finishes with YarnApplicationState: FINISHED
> , FinalStatus Reported by AM: ENDED and Diagnostics: Navigate to the
> failed component for more details.
> Here, Diagnostics message should be improved. When an application is
> explicitly stopped by user, the diagnostics message should say " Application
> stopped by user"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7479) TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk
[
https://issues.apache.org/jira/browse/YARN-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324667#comment-16324667
]
Robert Kanter commented on YARN-7479:
-
Oops, ya, that should have been x 1000 :)
Would it also make sense to change the interval, which is set to 10ms to
something like 500ms, or 1000ms? As it is now, it will check 100 times a
second which seems a little overkill. It's also more in line with what most of
the other calls to {{waitFor}} are doing where I'm seeing typical values of
100, 500, and 1000.
> TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk
>
>
> Key: YARN-7479
> URL: https://issues.apache.org/jira/browse/YARN-7479
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Botong Huang
>Assignee: Akira Ajisaka
> Attachments: YARN-7479.001.patch, YARN-7479.002.patch,
> YARN-7479.003.patch
>
>
> Was waiting for container_1_0001_01_00 to get to state COMPLETE but was
> in state RUNNING after the timeout
> java.lang.AssertionError: Was waiting for container_1_0001_01_00 to get
> to state COMPLETE but was in state RUNNING after the timeout
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:431)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:360)
> at
> org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:171)
> Pasting some exception message during test run here:
> org.apache.hadoop.security.AccessControlException: SIMPLE authentication is
> not enabled. Available:[TOKEN]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Given NMToken for application : appattempt_1_0001_01 seems to have been
> generated illegally.
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Given NMToken for application : appattempt_1_0001_01 is not valid for
> current node manager.expected : localhost:46649 found : InvalidHost:1234
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7744) Fix Get status rest api response when application is destroyed
Yesha Vora created YARN-7744:
Summary: Fix Get status rest api response when application is
destroyed
Key: YARN-7744
URL: https://issues.apache.org/jira/browse/YARN-7744
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-native-services
Reporter: Yesha Vora
Priority: Critical
Steps:
1) Create a yarn service
2) Destroy a yarn service
Run get status for application using REST API:
{code}
response json = {u'diagnostics': u'Failed to retrieve service: File does not
exist:
hdfs://mycluster/user/yarn/.yarn/services/httpd-service/httpd-service.json'}
status code = 500{code}
The REST API should respond with proper json including diagnostics and HTTP
status code 404
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7696) Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases
[ https://issues.apache.org/jira/browse/YARN-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-7696: -- Attachment: YARN-7696-YARN-6592.004.patch Updating patch based on comments from [~leftnoteasy]. Keeping the old APIs to prevent unforseen downstream issues. > Add container tags to ContainerTokenIdentifier, api.Container and > NMContainerStatus to handle all recovery cases > > > Key: YARN-7696 > URL: https://issues.apache.org/jira/browse/YARN-7696 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-7696-YARN-6592.001.patch, > YARN-7696-YARN-6592.002.patch, YARN-7696-YARN-6592.003.patch, > YARN-7696-YARN-6592.004.patch > > > The NM needs to persist the Container tags so that on RM recovery, it is sent > back to the RM via the NMContainerStatus. The RM would then recover the > AllocationTagsManager using this information. > The api.Container also requires the allocationTags since after AM recovery, > we need to provide the AM with previously allocated containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7743) UI component placement overlaps on Safari
[ https://issues.apache.org/jira/browse/YARN-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-7743: - Component/s: yarn-ui-v2 > UI component placement overlaps on Safari > - > > Key: YARN-7743 > URL: https://issues.apache.org/jira/browse/YARN-7743 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.0.0 >Reporter: Yesha Vora > Attachments: Screen Shot 2018-01-11 at 5.17.18 PM.png, Screen Shot > 2018-01-11 at 5.20.48 PM.png > > > Browser: Safari Version 9.1.1 (11601.6.17) > Issue with new RM UI: > The two tables on application and service page overlaps on Safari browser. > Find screenshot for details -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7696) Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases
[ https://issues.apache.org/jira/browse/YARN-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324695#comment-16324695 ] Wangda Tan commented on YARN-7696: -- Latest patch looks good, thanks [~asuresh] for updating the patch. +1 and pending Jenkins. > Add container tags to ContainerTokenIdentifier, api.Container and > NMContainerStatus to handle all recovery cases > > > Key: YARN-7696 > URL: https://issues.apache.org/jira/browse/YARN-7696 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-7696-YARN-6592.001.patch, > YARN-7696-YARN-6592.002.patch, YARN-7696-YARN-6592.003.patch, > YARN-7696-YARN-6592.004.patch > > > The NM needs to persist the Container tags so that on RM recovery, it is sent > back to the RM via the NMContainerStatus. The RM would then recover the > AllocationTagsManager using this information. > The api.Container also requires the allocationTags since after AM recovery, > we need to provide the AM with previously allocated containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7724) yarn application status should support application name
[
https://issues.apache.org/jira/browse/YARN-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324697#comment-16324697
]
genericqa commented on YARN-7724:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new + 247 unchanged - 1 fixed = 256 total (was 248) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 12s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m
51s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
40s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7724 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12
[jira] [Commented] (YARN-7739) Revisit scheduler resource normalization behavior for max allocation
[ https://issues.apache.org/jira/browse/YARN-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324699#comment-16324699 ] Wangda Tan commented on YARN-7739: -- Thanks [~jlowe], to me it is also a bug :). I think we should get rid of this since it could badly impact users when we have multiple resources enabled. Will talk to Vinod and keep this thread updated > Revisit scheduler resource normalization behavior for max allocation > > > Key: YARN-7739 > URL: https://issues.apache.org/jira/browse/YARN-7739 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Priority: Critical > > Currently, YARN Scheduler normalizes requested resource based on the maximum > allocation derived from configured maximum allocation and maximum registered > node resources. Basically, the scheduler will silently cap asked resource by > maximum allocation. > This could cause issues for applications, for example, a Spark job which > needs 12 GB memory to run, however in the cluster, registered NMs have at > most 8 GB mem on each node. So scheduler allocates 8GB memory container to > the requested application. > Once app receives containers from RM, if it doesn't double check allocated > resources, it will lead to OOM and hard to debug because scheduler silently > caps maximum allocation. > When non-mandatory resources introduced, this becomes worse. For resources > like GPU, we typically set minimum allocation to 0 since not all nodes have > GPU devices. So it is possible that application asks 4 GPUs but get 0 GPU, it > gonna be a big problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7745) Allow DistributedShell to take a placement specification for containers it wants to launch
Arun Suresh created YARN-7745:
-
Summary: Allow DistributedShell to take a placement specification
for containers it wants to launch
Key: YARN-7745
URL: https://issues.apache.org/jira/browse/YARN-7745
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun Suresh
Assignee: Arun Suresh
This is add a '-placement_spec' option to the distributed shell client. Where
the user can specify a string-ified specification for how it wants containers
to be placed.
For eg:
{noformat}
$ yarn org.apache.hadoop.yarn.applications.distributedshell.Client –jar \
$YARN_DS/hadoop-yarn-applications-distributedshell-$YARN_VERSION.jar \
-shell_command sleep -shell_args 10 -placement_spec
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7745) Allow DistributedShell to take a placement specification for containers it wants to launch
[
https://issues.apache.org/jira/browse/YARN-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324741#comment-16324741
]
Arun Suresh commented on YARN-7745:
---
Formal specification of the *placementspec*:
{noformat}
PlacementSpec => "" | KeyVal;PlacementSpec
KeyVal => SourceTag=Constraint
SourceTag => String
Constraint => NumContainers,ConstraintType,Scope,TargetTag |
NumContainers,ConstraintType,Scope,TargetTag,MinCard,MaxCard
NumContainers => int (number of containers)
ConstraintType => "in" | "notIn" | "cardinality"
Scope => "NODE"|"RACK"
TargetTag => String (Target Tag)
MinCard=> int (min cardinality - needed if ConstraintType ==
cardinality)
MaxCard=> int (max cardinality - needed if ConstraintType ==
cardinality)
{noformat}
Event though the PlacementConstraint API has support for a Set of source and
target expressions, to avoid complicating the distributed shell, we will
support a single source and target expression per constraint.
{code}
.. -placement_spec
foo=3,notIn,NODE,foo;bar=5;in,RACK,blur;blur,6,cardinality,NODE,blur,0,3
{code}
would encode 3 constraints:
# foo : 3 containers with antiaffinity to each other at the node scope (no more
than 1 container per node)
# bar : 5 containers with affinity to a rack on which containers with tag
"blur" are running
# blur : 6 containers with cardinality constraint, such that no more than 3
containers are scheduled on a node.
> Allow DistributedShell to take a placement specification for containers it
> wants to launch
> --
>
> Key: YARN-7745
> URL: https://issues.apache.org/jira/browse/YARN-7745
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> This is add a '-placement_spec' option to the distributed shell client. Where
> the user can specify a string-ified specification for how it wants containers
> to be placed.
> For eg:
> {noformat}
> $ yarn org.apache.hadoop.yarn.applications.distributedshell.Client –jar \
> $YARN_DS/hadoop-yarn-applications-distributedshell-$YARN_VERSION.jar \
> -shell_command sleep -shell_args 10 -placement_spec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7724) yarn application status should support application name
[ https://issues.apache.org/jira/browse/YARN-7724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324740#comment-16324740 ] Billie Rinaldi commented on YARN-7724: -- +1 for patch 04. > yarn application status should support application name > --- > > Key: YARN-7724 > URL: https://issues.apache.org/jira/browse/YARN-7724 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Yesha Vora >Assignee: Jian He > Attachments: YARN-7724.01.patch, YARN-7724.02.patch, > YARN-7724.03.patch, YARN-7724.04.patch > > > Yarn Service application are tied with app name. Thus, yarn application > -status should be able to take yarn service name as argument such as > yarn application -status -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7745) Allow DistributedShell to take a placement specification for containers it wants to launch
[
https://issues.apache.org/jira/browse/YARN-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324741#comment-16324741
]
Arun Suresh edited comment on YARN-7745 at 1/12/18 11:40 PM:
-
Formal specification of the *placementspec*:
{noformat}
PlacementSpec => "" | KeyVal;PlacementSpec
KeyVal => SourceTag=Constraint
SourceTag => String
Constraint => NumContainers,ConstraintType,Scope,TargetTag |
NumContainers,ConstraintType,Scope,TargetTag,MinCard,MaxCard
NumContainers => int (number of containers)
ConstraintType => "in" | "notIn" | "cardinality"
Scope => "NODE"|"RACK"
TargetTag => String (Target Tag)
MinCard=> int (min cardinality - needed if ConstraintType ==
cardinality)
MaxCard=> int (max cardinality - needed if ConstraintType ==
cardinality)
{noformat}
Event though the PlacementConstraint API has support for a Set of source and
target expressions, to avoid complicating the distributed shell, we will
support a single source and target expression per constraint.
{code}
.. -placement_spec
foo=3,notIn,NODE,foo;bar=5;in,RACK,blur;blur,6,cardinality,NODE,blur,0,3
{code}
would encode 3 constraints:
# foo : 3 containers with antiaffinity to each other at the node scope (no more
than 1 container per node)
# bar : 5 containers with affinity to a rack on which containers with tag
"blur" are running
# blur : 6 containers with cardinality constraint, such that no more than 3
containers are scheduled on a node.
NOTE:
* since the placement spec contains the number of containers, it will override
the *-num_containers* option.
* placement specification works only for GUARANTEED containers for the time
being.
was (Author: asuresh):
Formal specification of the *placementspec*:
{noformat}
PlacementSpec => "" | KeyVal;PlacementSpec
KeyVal => SourceTag=Constraint
SourceTag => String
Constraint => NumContainers,ConstraintType,Scope,TargetTag |
NumContainers,ConstraintType,Scope,TargetTag,MinCard,MaxCard
NumContainers => int (number of containers)
ConstraintType => "in" | "notIn" | "cardinality"
Scope => "NODE"|"RACK"
TargetTag => String (Target Tag)
MinCard=> int (min cardinality - needed if ConstraintType ==
cardinality)
MaxCard=> int (max cardinality - needed if ConstraintType ==
cardinality)
{noformat}
Event though the PlacementConstraint API has support for a Set of source and
target expressions, to avoid complicating the distributed shell, we will
support a single source and target expression per constraint.
{code}
.. -placement_spec
foo=3,notIn,NODE,foo;bar=5;in,RACK,blur;blur,6,cardinality,NODE,blur,0,3
{code}
would encode 3 constraints:
# foo : 3 containers with antiaffinity to each other at the node scope (no more
than 1 container per node)
# bar : 5 containers with affinity to a rack on which containers with tag
"blur" are running
# blur : 6 containers with cardinality constraint, such that no more than 3
containers are scheduled on a node.
> Allow DistributedShell to take a placement specification for containers it
> wants to launch
> --
>
> Key: YARN-7745
> URL: https://issues.apache.org/jira/browse/YARN-7745
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> This is add a '-placement_spec' option to the distributed shell client. Where
> the user can specify a string-ified specification for how it wants containers
> to be placed.
> For eg:
> {noformat}
> $ yarn org.apache.hadoop.yarn.applications.distributedshell.Client –jar \
> $YARN_DS/hadoop-yarn-applications-distributedshell-$YARN_VERSION.jar \
> -shell_command sleep -shell_args 10 -placement_spec
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
