[jira] [Commented] (YARN-7254) UI and metrics changes related to absolute resource configuration
[
https://issues.apache.org/jira/browse/YARN-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199744#comment-16199744
]
Hadoop QA commented on YARN-7254:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5881 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
38s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
47s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} YARN-5881 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 13m
24s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} YARN-5881 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 10 new + 281 unchanged - 1 fixed = 291 total (was 282) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 11m
12s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 13s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue |
[jira] [Commented] (YARN-7202) Add UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199732#comment-16199732
]
Hadoop QA commented on YARN-7202:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
57s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
39s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 6s{color} | {color:orange} root: The patch generated 1 new + 9 unchanged -
4 fixed = 10 total (was 13) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
|
[jira] [Commented] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199728#comment-16199728 ] Naganarasimha G R commented on YARN-6523: - Thanks for looking into this [~subru] and [~jlowe] for the comment, Sorry to have missed to update the priority. Agree that this is not critical but will impact clusters with multiple long running jobs. > RM requires large memory in sending out security tokens as part of Node > Heartbeat in large cluster > -- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6523: Priority: Major (was: Critical) > RM requires large memory in sending out security tokens as part of Node > Heartbeat in large cluster > -- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-6523: Target Version/s: 3.1.0 (was: 2.9.0) > RM requires large memory in sending out security tokens as part of Node > Heartbeat in large cluster > -- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) Add UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: YARN-7202.yarn-native-services.011.patch This patch is based on 008. PUT method is no longer modified, but make correction to stopService to report BAD_REQUEST instead of NOT_FOUND for deployed service. NO_CONTENT is also replaced with OK in case more diagnostic information need to be presented in the future. > Add UT for api-server > - > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch, > YARN-7202.yarn-native-services.011.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7202) Add UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199656#comment-16199656 ] Eric Yang edited comment on YARN-7202 at 10/11/17 1:12 AM: --- Patch 011 is based on 008. PUT method is no longer modified, but make correction to stopService to report BAD_REQUEST instead of NOT_FOUND for deployed service. NO_CONTENT is also replaced with OK in case more diagnostic information need to be presented in the future. was (Author: eyang): This patch is based on 008. PUT method is no longer modified, but make correction to stopService to report BAD_REQUEST instead of NOT_FOUND for deployed service. NO_CONTENT is also replaced with OK in case more diagnostic information need to be presented in the future. > Add UT for api-server > - > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch, > YARN-7202.yarn-native-services.011.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7254) UI and metrics changes related to absolute resource configuration
[
https://issues.apache.org/jira/browse/YARN-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199651#comment-16199651
]
Wangda Tan commented on YARN-7254:
--
[~sunilg],
Took a quick look at the patch and tried in my local cluster:
In general, it's better to make sure for all user-facing places
((REST)API/UI/CLI), include the information of is the resource is a effective
or configured. And on UI/API/REST-API, it's better to include both.
For example, on REST API (should we add {{-mb}} suffix to memory as well? And
change vCores to vcores):
{code}
6000
5
6000
12
{code}
And On queue page, it says:
{code}
Configured Capacity: (100.0%)
{code}
However it is effective capacity (100.0% is calculated as well).
Since we only allow admin to configure either absolute or percentage, we can
make the default to -1 to indicate it is not set by admin.
Others:
- QueueConfigurations not set by anybody, did you forget to set it or we should
move this to a separate patch? Like I mentioned above, it's better to include
both effective/configured resources in the java API.
*Tests on a pusedo cluster:*
Configured a single queue in c-s.xml: Queue.capacity = (8192, 8)
Start cluster with one node: (8192, 8). Information shown on UI looks correct.
Updated node's resource to: (6000, 12), Information shown on UI:
{code}
- Configured Capacity: (100.0%) (Should this be
vcore=8?).
- Max capacity. (100.0%)
{code}
> UI and metrics changes related to absolute resource configuration
> -
>
> Key: YARN-7254
> URL: https://issues.apache.org/jira/browse/YARN-7254
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-7254.001.patch, YARN-7254.002.patch,
> YARN-7254.YARN-5881.002.patch
>
>
> Impact on UI and metrics related to absolute resource configuration on CS
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7254) UI and metrics changes related to absolute resource configuration
[ https://issues.apache.org/jira/browse/YARN-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7254: - Attachment: YARN-7254.YARN-5881.002.patch > UI and metrics changes related to absolute resource configuration > - > > Key: YARN-7254 > URL: https://issues.apache.org/jira/browse/YARN-7254 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7254.001.patch, YARN-7254.002.patch, > YARN-7254.YARN-5881.002.patch > > > Impact on UI and metrics related to absolute resource configuration on CS -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[
https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199593#comment-16199593
]
Hadoop QA commented on YARN-6930:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} branch-2.8 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in branch-2.8 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
14s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c2d96dd |
| JIRA Issue | YARN-6930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891373/YARN-6930.branch-2.8.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux fc55b1bfdc23 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2.8 / 108c239 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| findbugs |
[jira] [Updated] (YARN-7224) Support GPU isolation for docker container
[
https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7224:
-
Attachment: YARN-7224.003.patch
Attached ver.003 patch, major updates:
1) Instead of using {{volume-driver}}, this patch added {{docker volume
create}} command to c-e and NM Java side. The reason is {{volume-driver}} can
only use single volume driver for each launched docker container.
2) Updated {{c-e}} and Java side, if a mounted volume is a named volume in
docker, skip checking file existence. (Named-volume still need to be added to
permitted list of container-executor.cfg).
3) More tests and cleanups.
> Support GPU isolation for docker container
> --
>
> Key: YARN-7224
> URL: https://issues.apache.org/jira/browse/YARN-7224
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-7224.001.patch, YARN-7224.002-wip.patch,
> YARN-7224.003.patch
>
>
> YARN-6620 added support of GPU isolation in NM side, which only supports
> non-docker containers. We need to add support to help docker containers
> launched by YARN can utilize GPUs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7198) Add jsvc support for RegistryDNS
[ https://issues.apache.org/jira/browse/YARN-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199576#comment-16199576 ] Allen Wittenauer commented on YARN-7198: bq. but this is going too slow. A month is a relatively short time compared to other patches. > Add jsvc support for RegistryDNS > > > Key: YARN-7198 > URL: https://issues.apache.org/jira/browse/YARN-7198 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7198-yarn-native-services.01.patch, > YARN-7198-yarn-native-services.02.patch, > YARN-7198-yarn-native-services.03.patch, > YARN-7198-yarn-native-services.04.patch, > YARN-7198-yarn-native-services.05.patch > > > RegistryDNS should have jsvc support and be managed through the shell > scripts, rather than being started manually. See original comments on > YARN-7191. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7124) LogAggregationTFileController deletes/renames while file is open
[
https://issues.apache.org/jira/browse/YARN-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199570#comment-16199570
]
Hadoop QA commented on YARN-7124:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891371/YARN-7124.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 24309fc96799 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 78af6cd |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17849/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17849/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> LogAggregationTFileController deletes/renames while file is open
>
>
>
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199569#comment-16199569
]
Shane Kumpf commented on YARN-7286:
---
Makes sense, thanks for taking another look. On the none vs NONE, I'm torn.
NONE isn't a capability, but the docs do even mention it's is best to use
uppercase for the value. However, Eric clearly documents to use "none".
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7198) Add jsvc support for RegistryDNS
[ https://issues.apache.org/jira/browse/YARN-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199566#comment-16199566 ] Jian He commented on YARN-7198: --- [~aw], it's been a month since Billie first posted the first version of the patch. We appreciate your reviews, but this is going too slow. we are going to commit this by Thursday. That said, you can always raise issues when you got time to it. > Add jsvc support for RegistryDNS > > > Key: YARN-7198 > URL: https://issues.apache.org/jira/browse/YARN-7198 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7198-yarn-native-services.01.patch, > YARN-7198-yarn-native-services.02.patch, > YARN-7198-yarn-native-services.03.patch, > YARN-7198-yarn-native-services.04.patch, > YARN-7198-yarn-native-services.05.patch > > > RegistryDNS should have jsvc support and be managed through the shell > scripts, rather than being started manually. See original comments on > YARN-7191. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) Add UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7202: -- Summary: Add UT for api-server (was: End-to-end UT for api-server) > Add UT for api-server > - > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7013) merge related work for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7013: --- Attachment: YARN-7013.branch-3.0.000.patch branch-3.0 patch for only resource types, without resource profiles. > merge related work for YARN-3926 branch > --- > > Key: YARN-7013 > URL: https://issues.apache.org/jira/browse/YARN-7013 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7013.001.patch, YARN-7013.002.patch, > YARN-7013.003.patch, YARN-7013.004.patch, YARN-7013.005.patch, > YARN-7013.006.patch, YARN-7013.008.patch, YARN-7013.branch-3.0.000.patch > > > To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-7013) merge related work for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reopened YARN-7013: Assignee: Daniel Templeton (was: Sunil G) Reopening to test the branch-3.0 patch > merge related work for YARN-3926 branch > --- > > Key: YARN-7013 > URL: https://issues.apache.org/jira/browse/YARN-7013 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Daniel Templeton > Attachments: YARN-7013.001.patch, YARN-7013.002.patch, > YARN-7013.003.patch, YARN-7013.004.patch, YARN-7013.005.patch, > YARN-7013.006.patch, YARN-7013.008.patch, YARN-7013.branch-3.0.000.patch > > > To run jenkins for whole branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-6930: -- Attachment: YARN-6930.branch-2.8.002.patch > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.002.patch, YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199519#comment-16199519 ] Shane Kumpf commented on YARN-6930: --- Thanks for the review [~jlowe], makes sense. Attaching a new patch with those changes. > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.002.patch, YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7124) LogAggregationTFileController deletes/renames while file is open
[ https://issues.apache.org/jira/browse/YARN-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-7124: - Attachment: YARN-7124.001.patch This isn't pretty, but it's small and I think it will do the trick. It does imply that calling postWrite means it is not OK to call write afterwards, and only close() can be called once postWrite is called. That's what the code is doing now anyway, but this would codify that. [~djp] [~xgong] please take a look. > LogAggregationTFileController deletes/renames while file is open > > > Key: YARN-7124 > URL: https://issues.apache.org/jira/browse/YARN-7124 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Daryn Sharp >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-7124.001.patch > > > YARN-6288 changes the log aggregation writer to be an autoclosable. > Unfortunately the try-with-resources block for the writer will either rename > or delete the log while open. > Assuming the NM's behavior is correct, deleting open files only results in > ominous WARNs in the nodemanager log and increases the rate of logging in the > NN when the implicit try-with-resource close fails. These red herrings > complicate debugging efforts. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7127) Merge yarn-native-service branch into trunk
[ https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199509#comment-16199509 ] Jian He commented on YARN-7127: --- bq. I think the development team was concerned that command structure overload between batch applications and long running services. [~eyang], Not sure how did you get such impression. This was not the reason. The CLI subcommand was earlier called yarn "slider", then later we rename it to "service", we just can't reuse the existing "application" subCommand without rewriting entire CLI code. > Merge yarn-native-service branch into trunk > --- > > Key: YARN-7127 > URL: https://issues.apache.org/jira/browse/YARN-7127 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7127.01.patch, YARN-7127.02.patch, > YARN-7127.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7124) LogAggregationTFileController deletes/renames while file is open
[ https://issues.apache.org/jira/browse/YARN-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned YARN-7124: Assignee: Jason Lowe Affects Version/s: (was: 2.8.2) 2.9.0 3.0.0-beta1 Summary: LogAggregationTFileController deletes/renames while file is open (was: CLONE - Log aggregation deletes/renames while file is open) > LogAggregationTFileController deletes/renames while file is open > > > Key: YARN-7124 > URL: https://issues.apache.org/jira/browse/YARN-7124 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Daryn Sharp >Assignee: Jason Lowe >Priority: Critical > > YARN-6288 changes the log aggregation writer to be an autoclosable. > Unfortunately the try-with-resources block for the writer will either rename > or delete the log while open. > Assuming the NM's behavior is correct, deleting open files only results in > ominous WARNs in the nodemanager log and increases the rate of logging in the > NN when the implicit try-with-resource close fails. These red herrings > complicate debugging efforts. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7127) Merge yarn-native-service branch into trunk
[
https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199486#comment-16199486
]
Jian He commented on YARN-7127:
---
{code}
yarn application -deploy –f spec.json
yarn application -stop
yarn application -restart
yarn application -remove
{code}
[~eyang]
The CLI code was from slider which uses JCommander and won't be easily
integrated with existing hadoop CLI syntax - the syntax is different, if you
look at the code, you will know what I'm talking about. Making it work with
the existing 'application' subcommand is not a simple 'rename' task. That
implies rewrite the whole thing.
cc [~billie.rinaldi], [~gsaha]
> Merge yarn-native-service branch into trunk
> ---
>
> Key: YARN-7127
> URL: https://issues.apache.org/jira/browse/YARN-7127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7127.01.patch, YARN-7127.02.patch,
> YARN-7127.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199472#comment-16199472 ] Jian He commented on YARN-7202: --- bq. After discussion with Jian, we agree to revert back to Patch 03, Ok, I missed this comment. In that case, my above comment doesn't apply. I think Billie's comment make sense, with that, the whole bunch of if/else condition are not needed. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199443#comment-16199443 ] Jian He commented on YARN-7202: --- bq. This is the reason that MiniHDFSCluster and MiniYARNCluster are both required dependencies for the test to run. MiniHDFSCluster is not even used in code at all. Can you remove the TestApiService from this patch and all the changes to followup patch ? because that's what needed by followup patches. Let's focus on one thing at a time. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199438#comment-16199438 ] Billie Rinaldi commented on YARN-7202: -- Instead of the patch 08 changes to updateService, I suggest changing stopService to return BAD_REQUEST instead of NOT_FOUND. This has almost the same effect on return codes as the patch 08 changes, but it does not remap OK response codes to NO_CONTENT. I would also suggest that when stopService succeeds, we should return OK instead of NO_CONTENT. We can delay addressing the issue of multiple types of operations being performed in PUT until YARN-7217. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199409#comment-16199409 ] Subru Krishnan commented on YARN-3661: -- [~elgoiri], thanks for backporting the patch and [~giovanni.fumarola] for the guidance. Is this fix part of YARN-7276 or is it only for branch-2? If it's the latter, fine to include it here. > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Attachments: YARN-3661-000.patch, YARN-3661-001.patch, > YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, > YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, > YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, > YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, > YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch, > YARN-3661-branch-2.002.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7294) TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199378#comment-16199378
]
Yufei Gu commented on YARN-7294:
Wait. It still failed with the patch.
{code}
java.lang.AssertionError: Attempt state is not correct (timeout).
Expected :ALLOCATED
Actual :SCHEDULED
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:358)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:298)
at
org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:955)
at
org.apache.hadoop.yarn.server.resourcemanager.TestSignalContainer.testSignalRequestDeliveryToNM(TestSignalContainer.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
> TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with
> Fair scheduler
> --
>
> Key: YARN-7294
> URL: https://issues.apache.org/jira/browse/YARN-7294
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-7294.000.patch
>
>
> This issue exists due to the fact that FS needs an update after allocation
> and more node updates for all the requests to be fulfilled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3661) Basic Federation UI
[
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199336#comment-16199336
]
Íñigo Goiri commented on YARN-3661:
---
Thank you [~giovanni.fumarola], your guidance showed me the way for successful
unit tests.
My guess is that the backport to branch-2 had some issues and this was
introduced there.
I add the fix in 002 in {{TestRouterWebServicesREST}}.
[~curino], [~subru], is it fine to include this fix here?
> Basic Federation UI
>
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch,
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch,
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch,
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch,
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch,
> YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch,
> YARN-3661-branch-2.002.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is
> running in a sub-cluster. In the context of federation we need new
> UIs that can track load, jobs, users across sub-clusters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7294) TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199335#comment-16199335 ] Yufei Gu commented on YARN-7294: The patch looks good to me. Should we test both FS and CS instead of either one depending on yarn configuration? File a new JIRA for that? > TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with > Fair scheduler > -- > > Key: YARN-7294 > URL: https://issues.apache.org/jira/browse/YARN-7294 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-7294.000.patch > > > This issue exists due to the fact that FS needs an update after allocation > and more node updates for all the requests to be fulfilled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-3661: -- Attachment: YARN-3661-branch-2.002.patch > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Attachments: YARN-3661-000.patch, YARN-3661-001.patch, > YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, > YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, > YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, > YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, > YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch, > YARN-3661-branch-2.002.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7315) Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-7315:
---
Description:
The original intention should be to make {{SchedulingPolicy}} Public is to let
users write their own policies. However, it shouldn't work for its all methods.
For example, {{isChildPolicyAllowed}} and {{checkIfUsageOverFairShare}}
shouldn't be Public API. We need to reconsider if we still want
{{SchedulingPolicy}} to be a public API or make it private then create
interface with less methods.
An incompatible change is made in YARN-5077, which removed the method
{{SchedulingPolicy#checkIfAMResourceUsageOverLimit}}. It may be OK since
{{checkIfAMResourceUsageOverLimit}} shouldn't work anyway for subclasses of
{{SchedulingPolicy}}.
was:
The original intention should be to make {{SchedulingPolicy}} Public is to let
users write their own policies. However, it shouldn't work for its all methods.
For example, {{isChildPolicyAllowed}} and {{checkIfUsageOverFairShare}}
shouldn't be Public API. We need to reconsider if we still want
{{SchedulingPolicy}} to be a public API or make it private then create
interface with less methods.
An incompatible change is made in YARN-5077, which removed the method
{{SchedulingPolicy#checkIfAMResourceUsageOverLimit}}
> Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
> --
>
> Key: YARN-7315
> URL: https://issues.apache.org/jira/browse/YARN-7315
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>
> The original intention should be to make {{SchedulingPolicy}} Public is to
> let users write their own policies. However, it shouldn't work for its all
> methods. For example, {{isChildPolicyAllowed}} and
> {{checkIfUsageOverFairShare}} shouldn't be Public API. We need to reconsider
> if we still want {{SchedulingPolicy}} to be a public API or make it private
> then create interface with less methods.
> An incompatible change is made in YARN-5077, which removed the method
> {{SchedulingPolicy#checkIfAMResourceUsageOverLimit}}. It may be OK since
> {{checkIfAMResourceUsageOverLimit}} shouldn't work anyway for subclasses of
> {{SchedulingPolicy}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7315) Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-7315:
---
Issue Type: Task (was: Bug)
> Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
> --
>
> Key: YARN-7315
> URL: https://issues.apache.org/jira/browse/YARN-7315
> Project: Hadoop YARN
> Issue Type: Task
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>
> The original intention should be to make {{SchedulingPolicy}} Public is to
> let users write their own policies. However, it shouldn't work for its all
> methods. For example, {{isChildPolicyAllowed}} and
> {{checkIfUsageOverFairShare}} shouldn't be Public API. We need to reconsider
> if we still want {{SchedulingPolicy}} to be a public API or make it private
> then create interface with less methods.
> An incompatible change is made in YARN-5077, which removed the method
> {{SchedulingPolicy#checkIfAMResourceUsageOverLimit}}. It may be OK since
> {{checkIfAMResourceUsageOverLimit}} shouldn't work anyway for subclasses of
> {{SchedulingPolicy}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7315) Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
[
https://issues.apache.org/jira/browse/YARN-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-7315:
---
Description:
The original intention should be to make {{SchedulingPolicy}} Public is to let
users write their own policies. However, it shouldn't work for its all methods.
For example, {{isChildPolicyAllowed}} and {{checkIfUsageOverFairShare}}
shouldn't be Public API. We need to reconsider if we still want
{{SchedulingPolicy}} to be a public API or make it private then create
interface with less methods.
An incompatible change is made in YARN-5077, which removed the method
{{SchedulingPolicy#checkIfAMResourceUsageOverLimit}}
was:The original intention should be to make {{SchedulingPolicy}} Public is
to let users write their own policies. However, it shouldn't work for its all
methods. For example, {{isChildPolicyAllowed}} and
{{checkIfUsageOverFairShare}} shouldn't be Public API. We need to reconsider if
we still want {{SchedulingPolicy}} to be a public API or make it private then
create interface with less methods.
> Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
> --
>
> Key: YARN-7315
> URL: https://issues.apache.org/jira/browse/YARN-7315
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>
> The original intention should be to make {{SchedulingPolicy}} Public is to
> let users write their own policies. However, it shouldn't work for its all
> methods. For example, {{isChildPolicyAllowed}} and
> {{checkIfUsageOverFairShare}} shouldn't be Public API. We need to reconsider
> if we still want {{SchedulingPolicy}} to be a public API or make it private
> then create interface with less methods.
> An incompatible change is made in YARN-5077, which removed the method
> {{SchedulingPolicy#checkIfAMResourceUsageOverLimit}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7315) Reconsider "Public" API in SchedulingPolicy of Fair Scheduler
Yufei Gu created YARN-7315:
--
Summary: Reconsider "Public" API in SchedulingPolicy of Fair
Scheduler
Key: YARN-7315
URL: https://issues.apache.org/jira/browse/YARN-7315
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 3.1.0
Reporter: Yufei Gu
The original intention should be to make {{SchedulingPolicy}} Public is to let
users write their own policies. However, it shouldn't work for its all methods.
For example, {{isChildPolicyAllowed}} and {{checkIfUsageOverFairShare}}
shouldn't be Public API. We need to reconsider if we still want
{{SchedulingPolicy}} to be a public API or make it private then create
interface with less methods.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7270) Resource#getVirtualCores() does unsafe casting from long to int.
[
https://issues.apache.org/jira/browse/YARN-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199290#comment-16199290
]
Yufei Gu commented on YARN-7270:
The javac warning is fine since it is the test code calling the deprecated
method.
> Resource#getVirtualCores() does unsafe casting from long to int.
>
>
> Key: YARN-7270
> URL: https://issues.apache.org/jira/browse/YARN-7270
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-7270.001.patch, YARN-7270.002.patch
>
>
> Class {{Resource}} has three sub classes(FixedValueResource,
> LightWeightResource, and ResourcePBImpl). Only FixedValueResource handle
> long-to-int casting nicely. The other two didn't. This bug is introduced by
> resource type feature and causes several unit test failures. For example:
> {code}
> Error Message
> expected:<> but was:<>
> Stacktrace
> java.lang.AssertionError: expected:<> but
> was:<>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt.testHeadroomWithBlackListedNodes(TestFSAppAttempt.java:325)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7270) Resource#getVirtualCores() does unsafe casting from long to int.
[
https://issues.apache.org/jira/browse/YARN-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199226#comment-16199226
]
Hadoop QA commented on YARN-7270:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 25s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 79 unchanged -
0 fixed = 80 total (was 79) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
49s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7270 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891303/YARN-7270.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux cd3e39fe2540 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d6602b5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/17847/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
[jira] [Commented] (YARN-3661) Basic Federation UI
[
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199220#comment-16199220
]
Giovanni Matteo Fumarola commented on YARN-3661:
Thanks [~elgoiri] for the patch.
Please fix the {{STATUS_BADREQUEST}} with {{NO_CONTENT}}. In trunk there are
more check fro Scheduler logs call. These checks are not in branch-2 and Yarn
RM returns {{NO_CONTENT}}.
> Basic Federation UI
>
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch,
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch,
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch,
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch,
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch,
> YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is
> running in a sub-cluster. In the context of federation we need new
> UIs that can track load, jobs, users across sub-clusters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6033) Add support for sections in container-executor configuration file
[
https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199215#comment-16199215
]
Hadoop QA commented on YARN-6033:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m
21s{color} | {color:green} root generated 0 new + 14 unchanged - 1 fixed = 14
total (was 15) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-maven-plugins in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 55s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | YARN-6033 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891304/YARN-6033-branch-2.014.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml cc |
| uname | Linux a3323cc47bdf 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 759d7d5 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5221
[
https://issues.apache.org/jira/browse/YARN-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199176#comment-16199176
]
Ray Chiang commented on YARN-7138:
--
Sorry to come late into this. Should we add a Release Note to YARN-5221 to
document the incompatibility and mark the JIRA as incompatible?
> Fix incompatible API change for YarnScheduler involved by YARN-5221
> ---
>
> Key: YARN-7138
> URL: https://issues.apache.org/jira/browse/YARN-7138
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Junping Du
>Priority: Critical
>
> From JACC report for 2.8.2 against 2.7.4, it indicates that we have
> incompatible changes happen in YarnScheduler:
> {noformat}
> hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
> package org.apache.hadoop.yarn.server.resourcemanager.scheduler
> YarnScheduler.allocate ( ApplicationAttemptId p1, List p2,
> List p3, List p4, List p5 ) [abstract] :
> Allocation
> {noformat}
> The root cause is YARN-5221. We should change it back or workaround this by
> adding back original API (mark as deprecated if not used any more).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4006) YARN AltKerberos HTTP Authentication doesn't work
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated YARN-4006:
---
Summary: YARN AltKerberos HTTP Authentication doesn't work (was: YARN
Alternate Kerberos HTTP Authentication Changes)
> YARN AltKerberos HTTP Authentication doesn't work
> -
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199174#comment-16199174
]
Allen Wittenauer commented on YARN-4006:
It's still very much broken for all YARN UIs (not just timeline server). We
have opted not to use the YARN UI on secure clusters.
At one point, we were investigating HADOOP-12082 as a replacement for
AltKerberos, but the lack of documentation is a major hindrance.
> YARN Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4006) YARN Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated YARN-4006:
---
Summary: YARN Alternate Kerberos HTTP Authentication Changes (was: YARN
ATSv1 Alternate Kerberos HTTP Authentication Changes)
> YARN Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4006) YARN Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated YARN-4006:
---
Target Version/s: (was: 2.9.0)
> YARN Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198941#comment-16198941 ] Jason Lowe edited comment on YARN-6930 at 10/10/17 6:26 PM: Thanks for the 2.8 patch! For consistency with the trunk/branch-2 code updateEnvForWhitelistVars should not throw ContainerExecutionException. See DelegatingLinuxContainerRuntime#updateEnvForWhitelistVars in branch-2 for reference. Once that exception declaration is removed then the TestContainerLaunch changes should also drop out of the patch. was (Author: jlowe): Thanks for the 2.8 patch! I don't understand why updateEnvForWhitelistVars has been updated to throw ContainerExecutionException when it isn't that way in trunk or branch-2. For consistency with the trunk/branch-2 code it should be consistent. See DelegatingLinuxContainerRuntime#updateEnvForWhitelistVars in branch-2 for reference. Once that exception declaration is removed then the TestContainerLaunch changes should also drop out of the patch. > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6384) Add configuratin to set max cpu usage when strict-resource-usage is false with cgroups
[
https://issues.apache.org/jira/browse/YARN-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199131#comment-16199131
]
Hadoop QA commented on YARN-6384:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 4 new + 211 unchanged - 0 fixed = 215 total (was 211) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
13s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 59s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestCGroupsCpuResourceHandlerImpl
|
| |
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
|
| | hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6384 |
| JIRA Patch URL |
[jira] [Commented] (YARN-7309) TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by
[
https://issues.apache.org/jira/browse/YARN-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199098#comment-16199098
]
Hudson commented on YARN-7309:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13061 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13061/])
YARN-7309. TestClientRMService#testUpdateApplicationPriorityRequest and
(rkanter: rev ec8bf9e48ac4dc675516a57c06a29890b5138548)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
> TestClientRMService#testUpdateApplicationPriorityRequest and
> TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test
> functionality not supported by FairScheduler
> --
>
> Key: YARN-7309
> URL: https://issues.apache.org/jira/browse/YARN-7309
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7309.001.patch
>
>
> {{TestClientRMService#testUpdateApplicationPriorityRequest}} and
> {{TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource}}
> test functionality (i.e. Application Priorities) not supported by
> FairScheduler. We should skip these two tests when using FairScheduler or
> they'll fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7179) The log data is corrupted due to the corrupted inputstream
[
https://issues.apache.org/jira/browse/YARN-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ting Dai updated YARN-7179:
---
Summary: The log data is corrupted due to the corrupted inputstream (was:
The log data is corrupted )
> The log data is corrupted due to the corrupted inputstream
> --
>
> Key: YARN-7179
> URL: https://issues.apache.org/jira/browse/YARN-7179
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Reporter: Ting Dai
>
> In hadoop-0.23.0, inside readAcontainerLogs function, when valueStream is
> corrupted, then writer will be corrupted.
> {code:java}
> public static void readAcontainerLogs(DataInputStream valueStream, Writer
> writer) throws IOException {
>
> while (true) {
> try {
>fileType = valueStream.readUTF();
> } catch (EOFException e) {
>return;
> }
> fileLengthStr = valueStream.readUTF(); //corrupted
> fileLength = Long.parseLong(fileLengthStr); //0
> writer.write("\n\nLogType:");
> writer.write(fileType);
> writer.write("\nLogLength:");
> writer.write(fileLengthStr);
> writer.write("\nLog Contents:\n");
> BoundedInputStream bis = new BoundedInputStream(valueStream,
> fileLength); //empty stream
> InputStreamReader reader = new InputStreamReader(bis);
> int currentRead = 0;
> int totalRead = 0;
> while ((currentRead = reader.read(cbuf, 0, bufferSize)) != -1) {
> //always return -1
> writer.write(cbuf);
> totalRead += currentRead;
> }
> }
> }
> {code}
>
> When the fileLengthStr is corrupted, especially when it is "0", but the
> valueStream is actually not empty. This will cause bis, reader be empty due
> to fileLength. The empty reader causes currentRead be -1 immediately, making
> writer.write inside the while loop never execute.
> During next iteration, the fileType, fileLength and bis, reader will be
> corrupted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5734) OrgQueue for easy CapacityScheduler queue configuration management
[ https://issues.apache.org/jira/browse/YARN-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung resolved YARN-5734. - Resolution: Fixed Resolving this ticket, since all of the required tasks have been merged to 2.9.0/3.0.0/3.1.0. The consolidated patches can be found at YARN-7241. Thanks everyone! > OrgQueue for easy CapacityScheduler queue configuration management > -- > > Key: YARN-5734 > URL: https://issues.apache.org/jira/browse/YARN-5734 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Min Shen >Assignee: Min Shen > Attachments: > OrgQueueAPI-BasedSchedulerConfigurationManagement_v2.pdf, > OrgQueue_API-Based_Config_Management_v1.pdf, OrgQueue_Design_v0.pdf, > YARN-5734-YARN-5734.001.patch > > > The current xml based configuration mechanism in CapacityScheduler makes it > very inconvenient to apply any changes to the queue configurations. We saw 2 > main drawbacks in the file based configuration mechanism: > # This makes it very inconvenient to automate queue configuration updates. > For example, in our cluster setup, we leverage the queue mapping feature from > YARN-2411 to route users to their dedicated organization queues. It could be > extremely cumbersome to keep updating the config file to manage the very > dynamic mapping between users to organizations. > # Even a user has the admin permission on one specific queue, that user is > unable to make any queue configuration changes to resize the subqueues, > changing queue ACLs, or creating new queues. All these operations need to be > performed in a centralized manner by the cluster administrators. > With these current limitations, we realized the need of a more flexible > configuration mechanism that allows queue configurations to be stored and > managed more dynamically. We developed the feature internally at LinkedIn > which introduces the concept of MutableConfigurationProvider. What it > essentially does is to provide a set of configuration mutation APIs that > allows queue configurations to be updated externally with a set of REST APIs. > When performing the queue configuration changes, the queue ACLs will be > honored, which means only queue administrators can make configuration changes > to a given queue. MutableConfigurationProvider is implemented as a pluggable > interface, and we have one implementation of this interface which is based on > Derby embedded database. > This feature has been deployed at LinkedIn's Hadoop cluster for a year now, > and have gone through several iterations of gathering feedbacks from users > and improving accordingly. With this feature, cluster administrators are able > to automate lots of thequeue configuration management tasks, such as setting > the queue capacities to adjust cluster resources between queues based on > established resource consumption patterns, or managing updating the user to > queue mappings. We have attached our design documentation with this ticket > and would like to receive feedbacks from the community regarding how to best > integrate it with the latest version of YARN. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[
https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199055#comment-16199055
]
Hadoop QA commented on YARN-6927:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 21s{color} | {color:orange} root: The patch generated 12 new + 879 unchanged
- 5 fixed = 891 total (was 884) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
31s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}104m
33s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891255/YARN-6927.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 856879afc608 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven
[jira] [Updated] (YARN-7252) Removing queue then failing over results in exception
[
https://issues.apache.org/jira/browse/YARN-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-7252:
Fix Version/s: (was: YARN-5734)
3.1.0
3.0.0
2.9.0
> Removing queue then failing over results in exception
> -
>
> Key: YARN-7252
> URL: https://issues.apache.org/jira/browse/YARN-7252
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Critical
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7252-YARN-5734.001.patch,
> YARN-7252-YARN-5734.002.patch
>
>
> Scenario: rm1 and rm2, starting configuration with root.default, root.a. rm1
> is active. First, put root.a into STOPPED state, then remove it. Then put rm1
> in standby and rm2 in active. Here's the exception: {noformat}Operation
> failed: Error on refreshAll during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
> at
> org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:747)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 10 more
> Caused by: java.io.IOException: Failed to re-init queues : root.a is deleted
> from the new capacity scheduler configuration, but the queue is not yet in
> stopped state. Current State : RUNNING
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:436)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:405)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:736)
> ... 11 more
> Caused by: java.io.IOException: root.a is deleted from the new capacity
> scheduler configuration, but the queue is not yet in stopped state. Current
> State : RUNNING
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.validateQueueHierarchy(CapacitySchedulerQueueManager.java:312)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:174)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:648)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:432)
> ... 13 more{noformat}
> Seems rm2 does not think root.a was STOPPED, so when it can't find root.a and
> sees it is deleted, it throws exception.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7251) Misc changes to YARN-5734
[ https://issues.apache.org/jira/browse/YARN-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7251: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Misc changes to YARN-5734 > - > > Key: YARN-7251 > URL: https://issues.apache.org/jira/browse/YARN-7251 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-7251-YARN-5734.001.patch, > YARN-7251-YARN-5734.002.patch, YARN-7251-YARN-5734.003-v2.patch, > YARN-7251-YARN-5734.004-v2.patch, YARN-7251-YARN-5734.005-v2.patch, > YARN-7251-YARN-5734.006-final.patch > > > Documentation/style changes to YARN-5734 before merge. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7238) Documentation for API based scheduler configuration management
[ https://issues.apache.org/jira/browse/YARN-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7238: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Documentation for API based scheduler configuration management > -- > > Key: YARN-7238 > URL: https://issues.apache.org/jira/browse/YARN-7238 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-7238-YARN-5734.001.patch, > YARN-7238-YARN-5734.002.patch > > > Documentation for configurations to set / how to use scheduler configuration > mutation API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7309) TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by
[
https://issues.apache.org/jira/browse/YARN-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199045#comment-16199045
]
Sunil G commented on YARN-7309:
---
Thanks [~rkanter]. Committing later today if no other objections.
> TestClientRMService#testUpdateApplicationPriorityRequest and
> TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test
> functionality not supported by FairScheduler
> --
>
> Key: YARN-7309
> URL: https://issues.apache.org/jira/browse/YARN-7309
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7309.001.patch
>
>
> {{TestClientRMService#testUpdateApplicationPriorityRequest}} and
> {{TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource}}
> test functionality (i.e. Application Priorities) not supported by
> FairScheduler. We should skip these two tests when using FairScheduler or
> they'll fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7024) Fix issues on recovery in LevelDB store
[ https://issues.apache.org/jira/browse/YARN-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7024: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Fix issues on recovery in LevelDB store > --- > > Key: YARN-7024 > URL: https://issues.apache.org/jira/browse/YARN-7024 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-7024-YARN-5734.001.patch > > > On recovery, we get the pending mutations not yet committed to store. Upon > confirming them, we remove them in memory while iterating, which may cause > issues in the future. > Also, when getting the pending mutations which were logged but not committed > on recovery, we don't update the txnId in memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7046) Add closing logic to configuration store
[ https://issues.apache.org/jira/browse/YARN-7046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7046: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Add closing logic to configuration store > > > Key: YARN-7046 > URL: https://issues.apache.org/jira/browse/YARN-7046 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-7046-YARN-5734.001.patch, > YARN-7046-YARN-5734.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6840) Implement zookeeper based store for scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-6840: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Implement zookeeper based store for scheduler configuration updates > --- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-6840-YARN-5734.001.patch, > YARN-6840-YARN-5734.002.patch, YARN-6840-YARN-5734.003.patch, > YARN-6840-YARN-5734.004.patch, YARN-6840-YARN-5734.005.patch, > YARN-6840-YARN-5734.006.patch, YARN-6840-YARN-5734.007.patch > > > Right now there is only in-memory and leveldb based configuration store > supported. Need one which supports RM HA. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6322) Disable queue refresh when configuration mutation is enabled
[
https://issues.apache.org/jira/browse/YARN-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-6322:
Fix Version/s: (was: YARN-5734)
3.1.0
3.0.0
2.9.0
> Disable queue refresh when configuration mutation is enabled
>
>
> Key: YARN-6322
> URL: https://issues.apache.org/jira/browse/YARN-6322
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-6322-YARN-5734.001.patch,
> YARN-6322-YARN-5734.002.patch, YARN-6322-YARN-5734.003.patch,
> YARN-6322-YARN-5734.004.patch
>
>
> When configuration mutation is enabled, the configuration store is the source
> of truth. Calling {{-refreshQueues}} won't work as intended, so we should
> just disable this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5947) Create LeveldbConfigurationStore class using Leveldb as backing store
[ https://issues.apache.org/jira/browse/YARN-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5947: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Create LeveldbConfigurationStore class using Leveldb as backing store > - > > Key: YARN-5947 > URL: https://issues.apache.org/jira/browse/YARN-5947 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5947-YARN-5734.001.patch, > YARN-5947-YARN-5734.002.patch, YARN-5947-YARN-5734.003.patch, > YARN-5947-YARN-5734.004.patch, YARN-5947.001.patch > > > LeveldbConfigurationStore will extend YarnConfigurationStore for storing > scheduler configuration in a LevelDB. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6575) Support global configuration mutation in MutableConfProvider
[ https://issues.apache.org/jira/browse/YARN-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-6575: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Support global configuration mutation in MutableConfProvider > > > Key: YARN-6575 > URL: https://issues.apache.org/jira/browse/YARN-6575 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-6575-YARN-5734.001.patch, > YARN-6575-YARN-5734.002.patch, YARN-6575-YARN-5734.003.patch > > > Right now mutating configs assumes they are only queue configs. Support > should be added to mutate global scheduler configs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5953) Create CLI for changing YARN configurations
[ https://issues.apache.org/jira/browse/YARN-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5953: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Create CLI for changing YARN configurations > --- > > Key: YARN-5953 > URL: https://issues.apache.org/jira/browse/YARN-5953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5953-YARN-5734.001.patch, > YARN-5953-YARN-5734.002.patch, YARN-5953-YARN-5734.003.patch, > YARN-5953-addendum-YARN-5734.001.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5949) Add pluggable configuration ACL policy interface and implementation
[ https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5949: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Add pluggable configuration ACL policy interface and implementation > --- > > Key: YARN-5949 > URL: https://issues.apache.org/jira/browse/YARN-5949 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5949-YARN-5734.001.patch, > YARN-5949-YARN-5734.002.patch, YARN-5949-YARN-5734.003.patch, > YARN-5949-YARN-5734.004.patch, YARN-5949-YARN-5734.005.patch > > > This will allow different policies to customize how/if configuration changes > should be applied (for example, a policy might restrict whether a > configuration change by a certain user is allowed). This will be enforced by > the MutableCSConfigurationProvider. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5951) Changes to allow CapacityScheduler to use configuration store
[ https://issues.apache.org/jira/browse/YARN-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5951: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Changes to allow CapacityScheduler to use configuration store > - > > Key: YARN-5951 > URL: https://issues.apache.org/jira/browse/YARN-5951 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5951-YARN-5734.001.patch, > YARN-5951-YARN-5734.002.patch, YARN-5951-YARN-5734.003.patch, > YARN-5951-YARN-5734.004.patch > > > EDIT: changing this ticket. Found that the CapacityStoreConfigurationProvider > is not necessary, since we can just grab a Configuration object from > StoreConfigurationProvider with type "SCHEDULER" and create a > CapacitySchedulerConfiguration from it. > This ticket will track changes needed for integrating other components to be > used by the capacity scheduler. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5948) Implement MutableConfigurationManager for handling storage into configuration store
[ https://issues.apache.org/jira/browse/YARN-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5948: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Implement MutableConfigurationManager for handling storage into configuration > store > --- > > Key: YARN-5948 > URL: https://issues.apache.org/jira/browse/YARN-5948 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5948-YARN-5734.002.patch, > YARN-5948-YARN-5734.003.patch, YARN-5948-YARN-5734.004.patch, > YARN-5948-YARN-5734.005.patch, YARN-5948-YARN-5734.006.patch, > YARN-5948-YARN-5734.007.patch, YARN-5948-YARN-5734.008.patch, > YARN-5948.001.patch > > > The MutableConfigurationManager will take REST calls with desired client > configuration changes and call YarnConfigurationStore methods to store these > changes in the backing store. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5952) Create REST API for changing YARN scheduler configurations
[ https://issues.apache.org/jira/browse/YARN-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5952: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Create REST API for changing YARN scheduler configurations > -- > > Key: YARN-5952 > URL: https://issues.apache.org/jira/browse/YARN-5952 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5952-YARN-5734.003.patch, > YARN-5952-YARN-5734.004.patch, YARN-5952-YARN-5734.005.patch, > YARN-5952-YARN-5734.006.patch, YARN-5952-YARN-5734.007.patch, > YARN-5952-YARN-5734.008.patch, YARN-5952.001.patch, YARN-5952.002.patch > > > Based on the design in YARN-5734. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5946) Create YarnConfigurationStore interface and InMemoryConfigurationStore class
[ https://issues.apache.org/jira/browse/YARN-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-5946: Fix Version/s: (was: YARN-5734) 3.1.0 3.0.0 2.9.0 > Create YarnConfigurationStore interface and InMemoryConfigurationStore class > > > Key: YARN-5946 > URL: https://issues.apache.org/jira/browse/YARN-5946 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-5946-YARN-5734.002.patch, > YARN-5946-YARN-5734.003.patch, YARN-5946-YARN-5734.004.patch, > YARN-5946.001.patch > > > This class provides the interface to persist YARN configurations in a backing > store. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199024#comment-16199024 ] Yufei Gu commented on YARN-2162: [~wangda], posted the new patch in YARN-7270. Wound you mind review it? > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-2162.001.patch, YARN-2162.002.patch, > YARN-2162.003.patch, YARN-2162.004.patch, YARN-2162.005.patch, > YARN-2162.006.patch, YARN-2162.007.patch, YARN-2162.008.patch, > YARN-2162.branch-2.010.patch, YARN-2162.branch-3.0.009.patch, > test-400nm-200app-2k_NODE_UPDATE.timecost.svg > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7270) Resource#getVirtualCores() does unsafe casting from long to int.
[
https://issues.apache.org/jira/browse/YARN-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-7270:
---
Attachment: YARN-7270.002.patch
Thanks for the review, [~templedf]. Uploaded patch v2 for your comments.
For the name of test class, I assume it is a convention to use "Test" + [class
to be tested] as the test class name. In that case, "TestResource" is the name
we should use. Am I wrong?
> Resource#getVirtualCores() does unsafe casting from long to int.
>
>
> Key: YARN-7270
> URL: https://issues.apache.org/jira/browse/YARN-7270
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-7270.001.patch, YARN-7270.002.patch
>
>
> Class {{Resource}} has three sub classes(FixedValueResource,
> LightWeightResource, and ResourcePBImpl). Only FixedValueResource handle
> long-to-int casting nicely. The other two didn't. This bug is introduced by
> resource type feature and causes several unit test failures. For example:
> {code}
> Error Message
> expected:<> but was:<>
> Stacktrace
> java.lang.AssertionError: expected:<> but
> was:<>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt.testHeadroomWithBlackListedNodes(TestFSAppAttempt.java:325)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6033: Attachment: YARN-6033-branch-2.014.patch Patch for branch-2. It also takes the modifications for hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/cmakebuilder/TestMojo.java from HADOOP-12714 to make sure tests run correctly. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0-beta1 > > Attachments: YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch, YARN-6033-branch-2.014.patch, > YARN-6033.003.patch, YARN-6033.004.patch, YARN-6033.005.patch, > YARN-6033.006.patch, YARN-6033.007.patch, YARN-6033.008.patch, > YARN-6033.009.patch, YARN-6033.010.patch, YARN-6033.011.patch, > YARN-6033.012.patch, YARN-6033.013.patch, YARN-6033.014.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7202) End-to-end UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198976#comment-16198976
]
Eric Yang edited comment on YARN-7202 at 10/10/17 5:20 PM:
---
[~jianhe] wrote:
{quote}
Confused. I'm seeing the opposite.
Before the patch, it directly returns the status code based on whatever failure
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST. e.g. the
NOT_FOUND status code for stopService method is instead converted to the
BAD_REQUEST status code. Why is this converstion needed?
{quote}
In ServiceClient, YarnException is used for invalid request, i.e. Service Name
not found, Application already exist, lost contact to AM, or Service definition
is not found in HDFS. There are many other exceptions overloaded using
YarnException and all of them are general error handling. We end up with web
response that are inconsistent between Web response code and the root cause,
such as 404 NOT_FOUND, and application already exist. This is the reason the
response code is generalized into BAD_REQUEST to avoid confusion.
was (Author: eyang):
[~jianhe] wrote:
{code}
Confused. I'm seeing the opposite.
Before the patch, it directly returns the status code based on whatever failure
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST. e.g. the
NOT_FOUND status code for stopService method is instead converted to the
BAD_REQUEST status code. Why is this converstion needed?
{code}
In ServiceClient, YarnException is used for invalid request, i.e. Service Name
not found, Application already exist, lost contact to AM, or Service definition
is not found in HDFS. There are many other exceptions overloaded using
YarnException and all of them are general error handling. We end up with web
response that are inconsistent between Web response code and the root cause,
such as 404 NOT_FOUND, and application already exist. This is the reason the
response code is generalized into BAD_REQUEST to avoid confusion.
> End-to-end UT for api-server
>
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch,
> YARN-7202.yarn-native-services.002.patch,
> YARN-7202.yarn-native-services.003.patch,
> YARN-7202.yarn-native-services.004.patch,
> YARN-7202.yarn-native-services.005.patch,
> YARN-7202.yarn-native-services.006.patch,
> YARN-7202.yarn-native-services.007.patch,
> YARN-7202.yarn-native-services.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-6033) Add support for sections in container-executor configuration file
[ https://issues.apache.org/jira/browse/YARN-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev reopened YARN-6033: - Re-opening for branch-2 version of the patch. > Add support for sections in container-executor configuration file > - > > Key: YARN-6033 > URL: https://issues.apache.org/jira/browse/YARN-6033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0-beta1 > > Attachments: YARN-6033-YARN-5673.001.patch, > YARN-6033-YARN-5673.002.patch, YARN-6033.003.patch, YARN-6033.004.patch, > YARN-6033.005.patch, YARN-6033.006.patch, YARN-6033.007.patch, > YARN-6033.008.patch, YARN-6033.009.patch, YARN-6033.010.patch, > YARN-6033.011.patch, YARN-6033.012.patch, YARN-6033.013.patch, > YARN-6033.014.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7309) TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by
[
https://issues.apache.org/jira/browse/YARN-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198989#comment-16198989
]
Robert Kanter commented on YARN-7309:
-
Thanks [~sunilg] for the review. Those failures are definitely not related -
the patch only adds an {{Assume}} statement to two specific tests and doesn't
actually change anything that could affect anything else. I also took a look
at the previous run on Jenkins, and the same tests were failing then (without
the patch).
> TestClientRMService#testUpdateApplicationPriorityRequest and
> TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test
> functionality not supported by FairScheduler
> --
>
> Key: YARN-7309
> URL: https://issues.apache.org/jira/browse/YARN-7309
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7309.001.patch
>
>
> {{TestClientRMService#testUpdateApplicationPriorityRequest}} and
> {{TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource}}
> test functionality (i.e. Application Priorities) not supported by
> FairScheduler. We should skip these two tests when using FairScheduler or
> they'll fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198977#comment-16198977
]
Eric Badger commented on YARN-7286:
---
bq. If we decide to keep with the "none" approach, given other properties can
be specified case-insensitive, I think it would be consistent to allow either
"none" or "NONE" as a value.
I'll wait for [~sidharta-s] and [~vvasudev] to respond before I upload a new
patch.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198976#comment-16198976
]
Eric Yang commented on YARN-7202:
-
[~jianhe] wrote:
{code}
Confused. I'm seeing the opposite.
Before the patch, it directly returns the status code based on whatever failure
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST. e.g. the
NOT_FOUND status code for stopService method is instead converted to the
BAD_REQUEST status code. Why is this converstion needed?
{code}
In ServiceClient, YarnException is used for invalid request, i.e. Service Name
not found, Application already exist, lost contact to AM, or Service definition
is not found in HDFS. There are many other exceptions overloaded using
YarnException and all of them are general error handling. We end up with web
response that are inconsistent between Web response code and the root cause,
such as 404 NOT_FOUND, and application already exist. This is the reason the
response code is generalized into BAD_REQUEST to avoid confusion.
> End-to-end UT for api-server
>
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch,
> YARN-7202.yarn-native-services.002.patch,
> YARN-7202.yarn-native-services.003.patch,
> YARN-7202.yarn-native-services.004.patch,
> YARN-7202.yarn-native-services.005.patch,
> YARN-7202.yarn-native-services.006.patch,
> YARN-7202.yarn-native-services.007.patch,
> YARN-7202.yarn-native-services.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198968#comment-16198968 ] Sunil G commented on YARN-6620: --- Cool. Makes sense. I will wait for hear from others if they have some more comments. Otherwise I will commit this patch tomorrow. Thank You. > [YARN-6223] NM Java side code changes to support isolate GPU devices by using > CGroups > - > > Key: YARN-6620 > URL: https://issues.apache.org/jira/browse/YARN-6620 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6620.001.patch, YARN-6620.002.patch, > YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, > YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, > YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, > YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, > YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch > > > This JIRA plan to add support of: > 1) GPU configuration for NodeManagers > 2) Isolation in CGroups. (Java side). > 3) NM restart and recovery allocated GPU devices -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198968#comment-16198968 ] Sunil G edited comment on YARN-6620 at 10/10/17 5:01 PM: - Cool. Makes sense. I will wait to hear from others if they have some more comments. Otherwise I will commit this patch tomorrow. Thank You. was (Author: sunilg): Cool. Makes sense. I will wait for hear from others if they have some more comments. Otherwise I will commit this patch tomorrow. Thank You. > [YARN-6223] NM Java side code changes to support isolate GPU devices by using > CGroups > - > > Key: YARN-6620 > URL: https://issues.apache.org/jira/browse/YARN-6620 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6620.001.patch, YARN-6620.002.patch, > YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, > YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, > YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, > YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, > YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch > > > This JIRA plan to add support of: > 1) GPU configuration for NodeManagers > 2) Isolation in CGroups. (Java side). > 3) NM restart and recovery allocated GPU devices -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7307) Revisit resource-types.xml loading behaviors
[ https://issues.apache.org/jira/browse/YARN-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7307: -- Attachment: YARN-7307.001.patch Uploading an initial version of patch to add more control for clients to load or not to load *resource-types.xml*. [~leftnoteasy] [~templedf] [~vvasudev] please help to share your thoughts. > Revisit resource-types.xml loading behaviors > > > Key: YARN-7307 > URL: https://issues.apache.org/jira/browse/YARN-7307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Blocker > Attachments: YARN-7307.001.patch > > > Existing feature requires every client has a resource-types.xml in order to > use multiple resource types, should we allow client/AM update supported > resource types via Yarn APIs? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198947#comment-16198947 ] Wangda Tan commented on YARN-6620: -- Thanks for the review, [~sunilg]. For #1, The JIRA is YARN-7159. I'm not sure if it is a valid use case for client to define different unit / type for resource-types.xml, and in YARN-7307 we're considering to remove client-side resource-type.xml. Let's discuss more on the related JIRAs. For #2, It might be better to rename "mandatory" to "first-class", so once FPGA support added to YARN, it is a first-class resource instead of mandatory resources. I think it might be valuable to have a list of first class resources supported by YARN, and we don't allow client to change core properties (includes unit and type, not include min/max value). With this we can have better out-of-box user experiences because most use cases should be addressed by first-class resources. To me NUMA is a little different here, today we don't support NUMA as a separate resource, instead it is additional affinity for known resource types (cpu/memory). So before we want to support centralized NUMA allocation in RM/scheduler, we don't need to model NUMA as a separate resource type. And once we need to do that, some refactoring need to be done to existing resource api since it cannot describe hierarchy and relations between resource units. > [YARN-6223] NM Java side code changes to support isolate GPU devices by using > CGroups > - > > Key: YARN-6620 > URL: https://issues.apache.org/jira/browse/YARN-6620 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6620.001.patch, YARN-6620.002.patch, > YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, > YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, > YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, > YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, > YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch > > > This JIRA plan to add support of: > 1) GPU configuration for NodeManagers > 2) Isolation in CGroups. (Java side). > 3) NM restart and recovery allocated GPU devices -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198941#comment-16198941 ] Jason Lowe commented on YARN-6930: -- Thanks for the 2.8 patch! I don't understand why updateEnvForWhitelistVars has been updated to throw ContainerExecutionException when it isn't that way in trunk or branch-2. For consistency with the trunk/branch-2 code it should be consistent. See DelegatingLinuxContainerRuntime#updateEnvForWhitelistVars in branch-2 for reference. Once that exception declaration is removed then the TestContainerLaunch changes should also drop out of the patch. > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6384) Add configuratin to set max cpu usage when strict-resource-usage is false with cgroups
[ https://issues.apache.org/jira/browse/YARN-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dengkai updated YARN-6384: -- Attachment: YARN-6384-3.patch > Add configuratin to set max cpu usage when strict-resource-usage is false > with cgroups > -- > > Key: YARN-6384 > URL: https://issues.apache.org/jira/browse/YARN-6384 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: dengkai > Attachments: YARN-6384-0.patch, YARN-6384-1.patch, YARN-6384-2.patch, > YARN-6384-3.patch > > > When using cgroups on yarn, if > yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage is > false, user may get very more cpu time than expected based on the vcores. > There should be a upper limit even resource-usage is not strict, just like a > percentage which user can get more than promised by vcores. I think it's > important in a shared cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations
[ https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198915#comment-16198915 ] Haibo Chen commented on YARN-4511: -- Thanks a lot for the review, [~asuresh]! bq. in the context of container updates, generally, a container stays on the same node - so was surprised that you added a newNode and oldNode arguments to the method I was not sure about whether container update is always node-local, hence I added a newNode and another oldNode argument. Will remove one in the follow up patch. The change to swapContainer(), IIRC, is necessary to fix some of the unit tests that the rest of the patch causes. The revelant change in the patch is that SchedulerNode is now modified to distinguish between Opportunistic and Guaranteed containers. Since swapContainer only swaps the internal container objects rather than RMContainers which SchedulerNod gets notified of, we need to update the resource accounting explicitly on the node every time there is a container swap. Otherwise, the incorrect resource accounting would cause existing unit tests to fail. I hope it makes sense to you. > Common scheduler changes supporting scheduler-specific implementations > -- > > Key: YARN-4511 > URL: https://issues.apache.org/jira/browse/YARN-4511 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Haibo Chen > Attachments: YARN-4511-YARN-1011.00.patch, > YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, > YARN-4511-YARN-1011.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198910#comment-16198910 ] Wangda Tan commented on YARN-2162: -- Thanks [~yufeigu], it looks like Sunil and Daniel have posted some comments, please let me know if you need any help from my side. > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Fix For: 2.9.0, 3.0.0, 3.1.0 > > Attachments: YARN-2162.001.patch, YARN-2162.002.patch, > YARN-2162.003.patch, YARN-2162.004.patch, YARN-2162.005.patch, > YARN-2162.006.patch, YARN-2162.007.patch, YARN-2162.008.patch, > YARN-2162.branch-2.010.patch, YARN-2162.branch-3.0.009.patch, > test-400nm-200app-2k_NODE_UPDATE.timecost.svg > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[
https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198897#comment-16198897
]
Hadoop QA commented on YARN-6930:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} branch-2.8 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in branch-2.8 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
51s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:c2d96dd |
| JIRA Issue | YARN-6930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891283/YARN-6930.branch-2.8.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 83291e37f2cc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2.8 / b41a702 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| findbugs |
[jira] [Commented] (YARN-6744) Recover component information on YARN native services AM restart
[
https://issues.apache.org/jira/browse/YARN-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198894#comment-16198894
]
Hadoop QA commented on YARN-6744:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
22s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 2 new + 54 unchanged - 11 fixed = 56 total (was 65)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
7s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-6744 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12891286/YARN-6744-yarn-native-services.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5b44caeb106b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 4993b8a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17844/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| Test Results |
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: (was: YARN-7202.yarn-native-services.010.patch) > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: (was: YARN-7202.yarn-native-services.009.patch) > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198881#comment-16198881
]
Eric Yang commented on YARN-7202:
-
[~jianhe] I took another look at the code from patch 08, and I have answers for
this question:
{quote}
confused, isn't this code not part of this patch ? I did clean before build.
Can you try compile only this patch with the dependency removed?
{quote}
I refactored TestYarnNativeServices to move the configuration initializer code
to ServiceTestUtils java class and move ServiceTestUtils java class to
org/apache/hadoop/yarn/service/client/ServiceTestUtils. This was done to
enhance the ability to reuse the utility class for YARN service setup for test
cases. TestApiService extends ServiceTestUtils for mini yarn cluster setup.
This is the reason that MiniHDFSCluster and MiniYARNCluster are both required
dependencies for the test to run. The scope of the dependencies is set to
"test". This is the reason that you were able to compile without the
dependencies expressed, they are used during test phase. Hence, both exception
throwing, and dependency expression are non-issues. Could you take another
look at patch 08? I deleted patch 09 and 10 because the decision that we made
are not correct in yesterday's meeting.
> End-to-end UT for api-server
>
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch,
> YARN-7202.yarn-native-services.002.patch,
> YARN-7202.yarn-native-services.003.patch,
> YARN-7202.yarn-native-services.004.patch,
> YARN-7202.yarn-native-services.005.patch,
> YARN-7202.yarn-native-services.006.patch,
> YARN-7202.yarn-native-services.007.patch,
> YARN-7202.yarn-native-services.008.patch,
> YARN-7202.yarn-native-services.009.patch,
> YARN-7202.yarn-native-services.010.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198880#comment-16198880
]
Jason Lowe commented on YARN-7286:
--
Yeah, the core problem with trying to use empty as a value was YARN-4258 set a
precedence for the default value to be
DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES, a significant list of capabilites by
default. The closest we could get is to leave the default in yarn-default.xml
to be the current list in DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES but treat an
empty/unset property as no capability. That would technically be incompatible
with the behavior in YARN-4258, and users would still have to explicitly unset
the property or set it to an empty string to get nothing rather than getting
nothing by default. However I think it's probably rare for someone to be
explicitly unsetting this property today or setting it to an empty string and
expecting the large list of capabilities in
DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES to be picked up. Pinging
[~sidharta-s] and [~vvasudev] since they worked on YARN-4258 and may have
opinions on how best to allow no capabilities to be set.
If we decide to keep with the "none" approach, given other properties can be
specified case-insensitive, I think it would be consistent to allow either
"none" or "NONE" as a value.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
[ https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198875#comment-16198875 ] lovekesh bansal commented on YARN-7312: --- Ok thanks for guiding me. And I will wait for your comments. > [Resource Profiles] getMaximumAllocationPerQueue to account for all resources > - > > Key: YARN-7312 > URL: https://issues.apache.org/jira/browse/YARN-7312 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: lovekesh bansal >Assignee: lovekesh bansal > Fix For: 3.1.0 > > Attachments: YARN-7312_trunk.001.patch > > > getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be > made generic for all resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6523) RM requires large memory in sending out security tokens as part of Node Heartbeat in large cluster
[ https://issues.apache.org/jira/browse/YARN-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198847#comment-16198847 ] Jason Lowe commented on YARN-6523: -- [~Naganarasimha] is this issue still Critical after the revelations from [this comment|https://issues.apache.org/jira/browse/YARN-6523?focusedCommentId=16000998=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16000998]? I got the impression this is not a problem for most deployments in practice because it is rare for a job to be submitted without the HDFS token needed for logging. > RM requires large memory in sending out security tokens as part of Node > Heartbeat in large cluster > -- > > Key: YARN-6523 > URL: https://issues.apache.org/jira/browse/YARN-6523 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Affects Versions: 2.8.0, 2.7.3 >Reporter: Naganarasimha G R >Assignee: Manikandan R >Priority: Critical > > Currently as part of heartbeat response RM sets all application's tokens > though all applications might not be active on the node. On top of it > NodeHeartbeatResponsePBImpl converts tokens for each app into > SystemCredentialsForAppsProto. Hence for each node and each heartbeat too > many SystemCredentialsForAppsProto objects were getting created. > We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with > 8GB RAM configured for RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
[ https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198828#comment-16198828 ] Sunil G commented on YARN-7312: --- Yes. As these test cases are not because of this patch, we can go ahead. I will take a close look at this patch and will share my comments if any. > [Resource Profiles] getMaximumAllocationPerQueue to account for all resources > - > > Key: YARN-7312 > URL: https://issues.apache.org/jira/browse/YARN-7312 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: lovekesh bansal >Assignee: lovekesh bansal > Fix For: 3.1.0 > > Attachments: YARN-7312_trunk.001.patch > > > getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be > made generic for all resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6744) Recover component information on YARN native services AM restart
[ https://issues.apache.org/jira/browse/YARN-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-6744: - Attachment: YARN-6744-yarn-native-services.002.patch Fix checkstyle issues and ensure that components can reach STABLE state after recovering containers. > Recover component information on YARN native services AM restart > > > Key: YARN-6744 > URL: https://issues.apache.org/jira/browse/YARN-6744 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-6744-yarn-native-services.001.patch, > YARN-6744-yarn-native-services.002.patch > > > The new RoleInstance#Container constructor does not populate all the > information needed for a RoleInstance. This is the constructor used when > recovering running containers in AppState#addRestartedContainer. We will have > to figure out a way to determine this information for a running container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198823#comment-16198823
]
Eric Badger commented on YARN-7286:
---
[~shaneku...@gmail.com], I don't see a way of doing it since the {{get()}}
method returns null both when the value is null and when the key doesn't exist.
So there's no way to differentiate between the two. If there's some method that
differentiates between those two cases, then we could make this work, but I'm
not aware of such a method.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
[ https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198825#comment-16198825 ] lovekesh bansal commented on YARN-7312: --- [~sunilg] Ok sure, I could see some of issues very specifically related to Fair scheduler but not covering these test cases. And I am new to apache community. Just for my understanding, in this case, we'll continue with committing the patches or fix those test cases first? > [Resource Profiles] getMaximumAllocationPerQueue to account for all resources > - > > Key: YARN-7312 > URL: https://issues.apache.org/jira/browse/YARN-7312 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: lovekesh bansal >Assignee: lovekesh bansal > Fix For: 3.1.0 > > Attachments: YARN-7312_trunk.001.patch > > > getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be > made generic for all resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-6930: -- Attachment: YARN-6930.branch-2.8.001.patch > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch, YARN-6930.004.patch, YARN-6930.005.patch, > YARN-6930.006.patch, YARN-6930.branch-2.001.patch, > YARN-6930.branch-2.002.patch, YARN-6930.branch-2.8.001.patch, > YARN-6930.branch-2.8.2.001.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
[ https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198794#comment-16198794 ] Sunil G commented on YARN-7312: --- Thanks. I guess there were some other known issues raised already. I ll help to identify them and update YARN-7314 as needed > [Resource Profiles] getMaximumAllocationPerQueue to account for all resources > - > > Key: YARN-7312 > URL: https://issues.apache.org/jira/browse/YARN-7312 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: lovekesh bansal >Assignee: lovekesh bansal > Fix For: 3.1.0 > > Attachments: YARN-7312_trunk.001.patch > > > getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be > made generic for all resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7312) [Resource Profiles] getMaximumAllocationPerQueue to account for all resources
[ https://issues.apache.org/jira/browse/YARN-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198789#comment-16198789 ] lovekesh bansal commented on YARN-7312: --- [~sunilg] Thanks for quick reply. Have created the bug YARN-7314 and linked to the parent umbrella. > [Resource Profiles] getMaximumAllocationPerQueue to account for all resources > - > > Key: YARN-7312 > URL: https://issues.apache.org/jira/browse/YARN-7312 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: lovekesh bansal >Assignee: lovekesh bansal > Fix For: 3.1.0 > > Attachments: YARN-7312_trunk.001.patch > > > getMaximumAllocationPerQueue has memory and vcores hardcoded. So it should be > made generic for all resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7314) Multiple Resource manager test cases failing
[ https://issues.apache.org/jira/browse/YARN-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198775#comment-16198775 ] lovekesh bansal commented on YARN-7314: --- Linking to the parent umbrella for resource profiles. > Multiple Resource manager test cases failing > > > Key: YARN-7314 > URL: https://issues.apache.org/jira/browse/YARN-7314 > Project: Hadoop YARN > Issue Type: Bug >Reporter: lovekesh bansal >Priority: Critical > > Following 14 unit tests are failing in the trunk: > > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService.testResourceTypes > > > org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testSchedulerRecovery[FAIR] > > > org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testDynamicQueueRecovery[FAIR] > > > org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForFairSche > > > org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt.testHeadroomWithBlackListedNodes > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShare > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testMultipleCompletedEvent > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testRefreshQueuesWhenRMHA > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShareWithContainerReservation > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testComputeMaxAMResource > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShareDefault > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testReservationMetrics > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testRequestAMResourceInZeroFairShareQueue > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7314) Multiple Resource manager test cases failing
lovekesh bansal created YARN-7314: - Summary: Multiple Resource manager test cases failing Key: YARN-7314 URL: https://issues.apache.org/jira/browse/YARN-7314 Project: Hadoop YARN Issue Type: Bug Reporter: lovekesh bansal Priority: Critical Following 14 unit tests are failing in the trunk: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService.testResourceTypes org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testSchedulerRecovery[FAIR] org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testDynamicQueueRecovery[FAIR] org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForFairSche org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt.testHeadroomWithBlackListedNodes org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShare org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testMultipleCompletedEvent org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testRefreshQueuesWhenRMHA org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShareWithContainerReservation org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testComputeMaxAMResource org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueMaxAMShareDefault org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testReservationMetrics org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testRequestAMResourceInZeroFairShareQueue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
