[jira] [Commented] (YARN-9701) Yarn service cli commands do not connect to ssl enabled RM using ssl-client.xml configs
[ https://issues.apache.org/jira/browse/YARN-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904321#comment-16904321 ] Tarun Parimi commented on YARN-9701: Thanks [~Prabhu Joseph] for reviewing the patch. Attaching a new patch which, # Removes the redundant SSLSocketFactory. # Reuses a single instance of Client and SSLFactory created for the ApiServiceClient class and destroys them on stop. This is safe as jersey docs https://jersey.github.io/documentation/1.19.1/client-api.html#d4e621 recommend reuse of Client instance. # Added unit tests in TestSecureApiServiceClient to test the simple auth, ssl, spnego and spnegoWithSsl combinations. Manually tested the change and it seems to work for both ssl enabled/disabled as well as spnego enabled/disabled in RM. > Yarn service cli commands do not connect to ssl enabled RM using > ssl-client.xml configs > --- > > Key: YARN-9701 > URL: https://issues.apache.org/jira/browse/YARN-9701 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: YARN-9701.001.patch, YARN-9701.002.patch > > > Yarn service commands use the yarn service rest api. When ssl is enabled for > RM, the yarn service commands fail as they don't read the ssl-client.xml > configs to create ssl connection to the rest api. > This becomes a problem especially for self signed certificates as the > truststore location specified at ssl.client.truststore.location is not > considered by commands. > As workaround, we need to import the certificates to the java default cacert > for the yarn service commands to work via ssl. It would be more proper if the > yarn service commands makes use of the configs at ssl-client.xml instead to > configure and create an ssl client connection. This workaround may not even > work if there are additional properties configured in ssl-client.xml that are > necessary apart from the truststore related properties. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9701) Yarn service cli commands do not connect to ssl enabled RM using ssl-client.xml configs
[ https://issues.apache.org/jira/browse/YARN-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tarun Parimi updated YARN-9701: --- Attachment: YARN-9701.002.patch > Yarn service cli commands do not connect to ssl enabled RM using > ssl-client.xml configs > --- > > Key: YARN-9701 > URL: https://issues.apache.org/jira/browse/YARN-9701 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services >Affects Versions: 3.1.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: YARN-9701.001.patch, YARN-9701.002.patch > > > Yarn service commands use the yarn service rest api. When ssl is enabled for > RM, the yarn service commands fail as they don't read the ssl-client.xml > configs to create ssl connection to the rest api. > This becomes a problem especially for self signed certificates as the > truststore location specified at ssl.client.truststore.location is not > considered by commands. > As workaround, we need to import the certificates to the java default cacert > for the yarn service commands to work via ssl. It would be more proper if the > yarn service commands makes use of the configs at ssl-client.xml instead to > configure and create an ssl client connection. This workaround may not even > work if there are additional properties configured in ssl-client.xml that are > necessary apart from the truststore related properties. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904311#comment-16904311
]
Hadoop QA commented on YARN-9719:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
30s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9719 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977211/YARN-9719.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 052af2d15870 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8a77a224 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24517/testReport/ |
| Max. process+thread count | 755 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24517/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Failed to restart
[jira] [Commented] (YARN-9730) Support forcing configured partitions to be exclusive based on app node label
[
https://issues.apache.org/jira/browse/YARN-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904308#comment-16904308
]
Hadoop QA commented on YARN-9730:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 71 new + 1094 unchanged - 3 fixed = 1165 total (was 1097) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 49s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m
4s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
57s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9730 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977207/YARN-9730.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c6e732025c02 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e02ffed |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
|
[jira] [Updated] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kyungwan nam updated YARN-9719:
---
Attachment: YARN-9719.007.patch
> Failed to restart yarn-service if it doesn’t exist in RM
>
>
> Key: YARN-9719
> URL: https://issues.apache.org/jira/browse/YARN-9719
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9719.001.patch, YARN-9719.002.patch,
> YARN-9719.003.patch, YARN-9719.004.patch, YARN-9719.005.patch,
> YARN-9719.006.patch, YARN-9719.007.patch
>
>
> Sometimes, restarting a yarn-service is failed as follows.
> {code}
> {"diagnostics":"Application with id 'application_1562735362534_10461' doesn't
> exist in RM. Please check that the job submission was successful.\n\tat
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)\n\tat
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)\n\tat
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)\n\tat
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)\n\tat
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)\n\tat
> java.security.AccessController.doPrivileged(Native Method)\n\tat
> javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)\n\tat
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)\n"}
> {code}
> It seems like that it occurs when restarting a yarn-service who was stopped
> long ago.
> by default, RM keeps up to 1000 completed applications
> (yarn.resourcemanager.max-completed-applications)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9698) [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-9698: - Target Version/s: 3.3.0 > [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler > > > Key: YARN-9698 > URL: https://issues.apache.org/jira/browse/YARN-9698 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Weiwei Yang >Priority: Major > Labels: fs2cs > > We see some users want to migrate from Fair Scheduler to Capacity Scheduler, > this Jira is created as an umbrella to track all related efforts for the > migration, the scope contains > * Bug fixes > * Add missing features > * Migration tools that help to generate CS configs based on FS, validate > configs etc > * Documents > this is part of CS component, the purpose is to make the migration process > smooth. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9730) Support forcing configured partitions to be exclusive based on app node label
[
https://issues.apache.org/jira/browse/YARN-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904280#comment-16904280
]
Jonathan Hung commented on YARN-9730:
-
Attached 001 patch. 1a and 1b are handled in
{{SchedulerUtils#enforcePartitionExclusivity}}. 2 is handled in
{{FifoOrderingPolicyWithExclusivePartitions}}.
Configurations to enable:
* {{yarn-site.xml}}
{noformat}
yarn.node-labels.exclusive-enforced-partitions
P
{noformat}
* {{capacity-scheduler.xml}}
{noformat}
yarn.scheduler.capacity..ordering-policy
fifo-with-partitions
yarn.scheduler.capacity..ordering-policy.exclusive-enforced-partitions
P
{noformat}
> Support forcing configured partitions to be exclusive based on app node label
> -
>
> Key: YARN-9730
> URL: https://issues.apache.org/jira/browse/YARN-9730
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9730.001.patch
>
>
> Use case: queue X has all of its workload in non-default (exclusive)
> partition P (by setting app submission context's node label set to P). Node
> in partition Q != P heartbeats to RM. Capacity scheduler loops through every
> application in X, and every scheduler key in this application, and fails to
> allocate each time since the app's requested label and the node's label don't
> match. This causes huge performance degradation when number of apps in X is
> large.
> To fix the issue, allow RM to configure partitions as "forced-exclusive". If
> partition P is "forced-exclusive", then:
> * 1a. If app sets its submission context's node label to P, all its resource
> requests will be overridden to P
> * 1b. If app sets its submission context's node label Q, any of its resource
> requests whose labels are P will be overridden to Q
> * 2. In the scheduler, we add apps with node label expression P to a
> separate data structure. When a node in partition P heartbeats to scheduler,
> we only try to schedule apps in this data structure. When a node in partition
> Q heartbeats to scheduler, we schedule the rest of the apps as normal.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9730) Support forcing configured partitions to be exclusive based on app node label
[ https://issues.apache.org/jira/browse/YARN-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9730: Description: Use case: queue X has all of its workload in non-default (exclusive) partition P (by setting app submission context's node label set to P). Node in partition Q != P heartbeats to RM. Capacity scheduler loops through every application in X, and every scheduler key in this application, and fails to allocate each time since the app's requested label and the node's label don't match. This causes huge performance degradation when number of apps in X is large. To fix the issue, allow RM to configure partitions as "forced-exclusive". If partition P is "forced-exclusive", then: * 1a. If app sets its submission context's node label to P, all its resource requests will be overridden to P * 1b. If app sets its submission context's node label Q, any of its resource requests whose labels are P will be overridden to Q * 2. In the scheduler, we add apps with node label expression P to a separate data structure. When a node in partition P heartbeats to scheduler, we only try to schedule apps in this data structure. When a node in partition Q heartbeats to scheduler, we schedule the rest of the apps as normal. was: Use case: queue X has all of its workload in non-default (exclusive) partition P (by setting app submission context's node label set to P). Node in partition Q != P heartbeats to RM. Capacity scheduler loops through every application in X, and every scheduler key in this application, and fails to allocate each time since the app's requested label and the node's label don't match. This causes huge performance degradation when number of apps in X is large. To fix the issue, allow RM to configure partitions as "forced-exclusive". If partition P is "forced-exclusive", then: * If app sets its submission context's node label to P, all its resource requests will be overridden to P * If app sets its submission context's node label Q, any of its resource requests whose labels are P will be overridden to Q * In the scheduler, we add apps with node label expression P to a separate data structure. When a node in partition P heartbeats to scheduler, we only try to schedule apps in this data structure. When a node in partition Q heartbeats to scheduler, we schedule the rest of the apps as normal. > Support forcing configured partitions to be exclusive based on app node label > - > > Key: YARN-9730 > URL: https://issues.apache.org/jira/browse/YARN-9730 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9730.001.patch > > > Use case: queue X has all of its workload in non-default (exclusive) > partition P (by setting app submission context's node label set to P). Node > in partition Q != P heartbeats to RM. Capacity scheduler loops through every > application in X, and every scheduler key in this application, and fails to > allocate each time since the app's requested label and the node's label don't > match. This causes huge performance degradation when number of apps in X is > large. > To fix the issue, allow RM to configure partitions as "forced-exclusive". If > partition P is "forced-exclusive", then: > * 1a. If app sets its submission context's node label to P, all its resource > requests will be overridden to P > * 1b. If app sets its submission context's node label Q, any of its resource > requests whose labels are P will be overridden to Q > * 2. In the scheduler, we add apps with node label expression P to a > separate data structure. When a node in partition P heartbeats to scheduler, > we only try to schedule apps in this data structure. When a node in partition > Q heartbeats to scheduler, we schedule the rest of the apps as normal. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9730) Support forcing configured partitions to be exclusive based on app node label
[ https://issues.apache.org/jira/browse/YARN-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9730: Attachment: YARN-9730.001.patch > Support forcing configured partitions to be exclusive based on app node label > - > > Key: YARN-9730 > URL: https://issues.apache.org/jira/browse/YARN-9730 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9730.001.patch > > > Use case: queue X has all of its workload in non-default (exclusive) > partition P (by setting app submission context's node label set to P). Node > in partition Q != P heartbeats to RM. Capacity scheduler loops through every > application in X, and every scheduler key in this application, and fails to > allocate each time since the app's requested label and the node's label don't > match. This causes huge performance degradation when number of apps in X is > large. > To fix the issue, allow RM to configure partitions as "forced-exclusive". If > partition P is "forced-exclusive", then: > * If app sets its submission context's node label to P, all its resource > requests will be overridden to P > * If app sets its submission context's node label Q, any of its resource > requests whose labels are P will be overridden to Q > * In the scheduler, we add apps with node label expression P to a separate > data structure. When a node in partition P heartbeats to scheduler, we only > try to schedule apps in this data structure. When a node in partition Q > heartbeats to scheduler, we schedule the rest of the apps as normal. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904256#comment-16904256 ] Eric Yang commented on YARN-9442: - [~ebadger] [~Jim_Brennan] Thank you for the confirmation. > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6539) Create SecureLogin inside Router
[ https://issues.apache.org/jira/browse/YARN-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904254#comment-16904254 ] Subru Krishnan commented on YARN-6539: -- [~yifan.stan], great to hear that you are running Federation in a secure cluster! I would love to hear more details about it. I thought I had mentioned it to [~shenyinjie] but guess not - I am not familiar with the security code. Hopefully [~bibinchundatt] or [~Prabhu Joseph] can help? Also, would it be possible to add a test? Thanks. > Create SecureLogin inside Router > > > Key: YARN-6539 > URL: https://issues.apache.org/jira/browse/YARN-6539 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Xie YiFan >Priority: Minor > Attachments: YARN-6359_1.patch, YARN-6359_2.patch, YARN-6539_3.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904219#comment-16904219
]
Eric Badger commented on YARN-9564:
---
The skopeo error looks to be because {{XDG_RUNTIME_DIR}} is set to /run/user/0,
which is root-owned. If you unset this environment variable or set it to
/run/user/$UID, I believe skopeo should work.
The permission failure in your 2nd comment is a result of running skopeo as
root and then the script not being able to remove the root-owned file as a
normal user.
As for running the script as root, yes, I would like to remove root altogether.
However, in the meantime, you should be able to get around the issue by running
as root with a kerberos credential.
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904218#comment-16904218 ] Jim Brennan commented on YARN-9442: --- [~eyang], [~ebadger] thanks for the discussion. I have tested on a test cluster with 0700 for the container working directory and that works just fine for running my test jobs. However, I did some poking around in the source base and found one case that I think will break if we remove execute permissions - ContainerImpl.ResourceLocalizedWhileRunningTransition() is attempting to check whether a symbolic link exists in the working directory (for a localized resource). I don't think that exists() check will work without execute permissions on the container working directory. To actually create the link, we will need to use a privileged operation, so I don't think that part would be affected. Given this case (and the potential for others like it), and the fact that DefaultContainerExecutor is using 0710, I think we should stick with 0710. > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Add Yarnclient#getDelegationToken API implementation and SecureLogin in router
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904212#comment-16904212
]
Hadoop QA commented on YARN-9708:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-9708 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9708 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975960/Add_getDelegationToken_and_SecureLogin_in_router.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24515/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add Yarnclient#getDelegationToken API implementation and SecureLogin in router
> --
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: Xie YiFan
>Priority: Minor
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9708) Add Yarnclient#getDelegationToken API implementation and SecureLogin in router
[ https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned YARN-9708: - Assignee: Xie YiFan > Add Yarnclient#getDelegationToken API implementation and SecureLogin in router > -- > > Key: YARN-9708 > URL: https://issues.apache.org/jira/browse/YARN-9708 > Project: Hadoop YARN > Issue Type: New Feature > Components: router >Affects Versions: 3.1.1 >Reporter: Xie YiFan >Assignee: Xie YiFan >Priority: Minor > Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch > > > 1.we use router as proxy to manage multiple cluster which be independent of > each other in order to apply unified client. Thus, we implement our > customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other > cluster. > 2.Our production environment need kerberos. But router doesn't support > SecureLogin for now. > https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we > improvement it. > 3.Some framework like oozie would get Token via yarnclient#getDelegationToken > which router doesn't support. Our solution is that adding homeCluster to > ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would > be submitted with specified clusterid so that router knows which cluster to > submit this job. Router would get Token from one RM according to specified > clusterid when client call getDelegation meanwhile apply some mechanism to > save this token in memory. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9736) Recursively configure app ordering policies
Jonathan Hung created YARN-9736:
---
Summary: Recursively configure app ordering policies
Key: YARN-9736
URL: https://issues.apache.org/jira/browse/YARN-9736
Project: Hadoop YARN
Issue Type: Task
Reporter: Jonathan Hung
Currently app ordering policy will find confs with prefix
{{.ordering-policy}}. For queues with same ordering policy
configurations it's easier to have a queue inherit confs from its parent.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9718) Yarn REST API, services endpoint remote command ejection
[
https://issues.apache.org/jira/browse/YARN-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904201#comment-16904201
]
Hadoop QA commented on YARN-9718:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
53s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9718 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977200/YARN-9718.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ea76db31dee7 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 98dd7c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24514/testReport/ |
| Max. process+thread count | 753 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24514/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Yarn REST API, services
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904188#comment-16904188
]
Eric Yang commented on YARN-9564:
-
If I added sudo to skopeo, the script only fail on removing temp directory. I
think this is toward the end of conversion process. I think this script is
suffering from mixed identity that some operations must be done as root, and
some operation must be done as hdfs admin. If running this script as root user
with a kerberos credential to hdfs, the script is likely pass. However, most
admin will require to run this script has themselves to avoid getting root user
mixed up with normal user kerberos ticket cache. In the later case, the
current script would encounter permission denied errors.
{code}
DEBUG: command: ['sudo', 'mksquashfs',
u'/tmp/sq/expand_archive_8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df',
u'/tmp/sq/centos:latest/8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df.sqsh']
DEBUG: command: ['sudo', 'rm', '-rf', '/tmp/sq']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 612, in pull_build_push_update
docker_to_squash(skopeo_dir, layer, working_dir)
File "./docker_to_squash.py", line 512, in docker_to_squash
os.remove(layer_path)
OSError: [Errno 13] Permission denied:
'/tmp/sq/centos:latest/8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df'
{code}
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9564) Create docker-to-squash tool for image conversion
[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904182#comment-16904182
]
Eric Yang commented on YARN-9564:
-
[~ebadger] Thank for the patch 003. The script seems to progress further with
some errors:
{code:bash}
$ ./docker_to_squash.py --hadoop-prefix=/usr/local/hadoop-3.3.0-SNAPSHOT
--working-dir=/tmp/sq --log=DEBUG pull-build-push-update
centos:latest,centos:latest
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'version']
DEBUG: command: ['skopeo', '-v']
DEBUG: command: ['mksquashfs', '-version']
DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False,
func=,
hadoop_prefix='/usr/local/hadoop-3.3.0-SNAPSHOT', hdfs_root='/runc-root',
image_tag_to_hash='image-tag-to-hash',
images_and_tags=['centos:latest,centos:latest'],
magic_file='etc/dockerfile-version', pull_format='docker', replication=1,
skopeo_format='dir', sub_command='pull-build-push-update',
working_dir='/tmp/sq')
DEBUG: extra: []
DEBUG: image-tag-to-hash: image-tag-to-hash
DEBUG: LOG_LEVEL: DEBUG
DEBUG: HADOOP_BIN_DIR: /usr/local/hadoop-3.3.0-SNAPSHOT/bin
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
ls: `/runc-root': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root']
DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
ls: `/runc-root/layers': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
ls: `/runc-root/config': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
ls: `/runc-root/manifests': No such file or directory
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-mkdir',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/image-tag-to-hash']
ls: `/runc-root/image-tag-to-hash': No such file or directory
INFO: Working on image centos:latest with tags ['centos:latest']
DEBUG: command: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
FATA[] error getting username and password: error reading JSON file
"/run/user/0/containers/auth.json": open /run/user/0/containers/auth.json:
permission denied
DEBUG: command: ['sudo', 'rm', '-rf', '/tmp/sq']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 560, in pull_build_push_update
manifest, manifest_hash = get_manifest_from_docker_image(pull_format, image)
File "./docker_to_squash.py", line 237, in get_manifest_from_docker_image
False, True, True)
File "./docker_to_squash.py", line 67, in shell_command
+ str(out) + "\nstderr: " + str(err))
Exception: Commmand: ['skopeo', 'inspect', '--raw', 'docker://centos:latest']
failed with returncode: 1
stdout:
stderr: None
{code}
It appears some commands are not expected to prompt password during execution.
Any hint?
> Create docker-to-squash tool for image conversion
> -
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own
[jira] [Commented] (YARN-9728) ResourceManager REST API can produce an illegal xml response
[
https://issues.apache.org/jira/browse/YARN-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904179#comment-16904179
]
Thomas commented on YARN-9728:
--
Hi Prabhu Joseph,
I didn't plan to submit a patch, so sure you can work on this.
Thomas
> ResourceManager REST API can produce an illegal xml response
> -
>
> Key: YARN-9728
> URL: https://issues.apache.org/jira/browse/YARN-9728
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api, resourcemanager
>Affects Versions: 2.7.3
>Reporter: Thomas
>Priority: Major
> Attachments: IllegalResponseChrome.png
>
>
> When a spark job throws an exception with a message containing a character
> out of the range supported by xml 1.0, then
> the application fails and the stack trace will be stored into the
> {{diagnostics}} field. So far, so good.
> But the issue occurred when we try to get application information with the
> ResourceManager REST API
> The xml response will contain the illegal xml 1.0 char and will be invalid.
> *+Examples of illegals characters in xml 1.0 :+*
> * \u
> * \u0001
> * \u0002
> * \u0003
> * \u0004
> _For more information about supported characters :_
> [https://www.w3.org/TR/xml/#charsets]
> *+Example of illegal response from the Ressource Manager API :+*
> {code:xml}
>
>
> application_1326821518301_0005
> user1
> job
> a1
> FINISHED
> FAILED
> 100.0
> History
>
> http://host.domain.com:8088/proxy/application_1326821518301_0005/jobhistory/job/job_1326821518301_5_5
> Exception in thread "main" java.lang.Exception: \u0001
> at com..main(JobWithSpecialCharMain.java:6)
> [...]
>
> {code}
>
> *+Example of job to reproduce :+*
> {code:java}
> public class JobWithSpecialCharMain {
> public static void main(String[] args) throws Exception {
> throw new Exception("\u0001");
> }
> }
> {code}
> !IllegalResponseChrome.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904172#comment-16904172 ] Eric Badger commented on YARN-9442: --- YARN-88 gives some relevant discussion based on the permissions in DefaultContainerExecutor. It seems like back in 2012 they thought 700 might be ok, but committed the patch before figuring it out > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9728) ResourceManager REST API can produce an illegal xml response
[
https://issues.apache.org/jira/browse/YARN-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904170#comment-16904170
]
Prabhu Joseph commented on YARN-9728:
-
Hi [~tde], Thanks for the detailed description. Do you want to submit a patch
for this issue, if not, i will start work on this.
> ResourceManager REST API can produce an illegal xml response
> -
>
> Key: YARN-9728
> URL: https://issues.apache.org/jira/browse/YARN-9728
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api, resourcemanager
>Affects Versions: 2.7.3
>Reporter: Thomas
>Priority: Major
> Attachments: IllegalResponseChrome.png
>
>
> When a spark job throws an exception with a message containing a character
> out of the range supported by xml 1.0, then
> the application fails and the stack trace will be stored into the
> {{diagnostics}} field. So far, so good.
> But the issue occurred when we try to get application information with the
> ResourceManager REST API
> The xml response will contain the illegal xml 1.0 char and will be invalid.
> *+Examples of illegals characters in xml 1.0 :+*
> * \u
> * \u0001
> * \u0002
> * \u0003
> * \u0004
> _For more information about supported characters :_
> [https://www.w3.org/TR/xml/#charsets]
> *+Example of illegal response from the Ressource Manager API :+*
> {code:xml}
>
>
> application_1326821518301_0005
> user1
> job
> a1
> FINISHED
> FAILED
> 100.0
> History
>
> http://host.domain.com:8088/proxy/application_1326821518301_0005/jobhistory/job/job_1326821518301_5_5
> Exception in thread "main" java.lang.Exception: \u0001
> at com..main(JobWithSpecialCharMain.java:6)
> [...]
>
> {code}
>
> *+Example of job to reproduce :+*
> {code:java}
> public class JobWithSpecialCharMain {
> public static void main(String[] args) throws Exception {
> throw new Exception("\u0001");
> }
> }
> {code}
> !IllegalResponseChrome.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9735) Allow User Keytab to submit YARN Native Service
Prabhu Joseph created YARN-9735:
---
Summary: Allow User Keytab to submit YARN Native Service
Key: YARN-9735
URL: https://issues.apache.org/jira/browse/YARN-9735
Project: Hadoop YARN
Issue Type: Improvement
Components: yarn-native-services
Affects Versions: 3.2.0
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
Yarn Native Service launch fails on a secure cluster with user keytab. It
allows only service keytab. Have seen most of the users test their jobs with
user keytab.
{code}
[ambari-qa@pjosephdocker-3 ~]$ yarn app -launch sleeper-service
/usr/hdp/3.0.1.0-187/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
19/08/03 17:17:04 ERROR client.ApiServiceClient: Kerberos principal
(ambari-qa-pjosephdoc...@docker.com) does not contain a hostname.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9734) LogAggregationIndexedFileController fails to upload logs in rolling fashion
Prabhu Joseph created YARN-9734:
---
Summary: LogAggregationIndexedFileController fails to upload logs
in rolling fashion
Key: YARN-9734
URL: https://issues.apache.org/jira/browse/YARN-9734
Project: Hadoop YARN
Issue Type: Bug
Components: log-aggregation
Affects Versions: 3.3.0
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
LogAggregationIndexedFileController fails to upload logs in rolling fashion.
*Configs:*
{code}
yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds = 60
yarn.nodemanager.log-aggregation.debug-enabled = true
yarn.log-aggregation.file-formats=IFile
yarn.log-aggregation.file-controller.IFile.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
{code}
*Initialize writer fails with below error:*
{code}
2019-08-09 07:46:12,411 ERROR
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
Cannot create writer for app application_1565102314214_0007. Skip log upload
this time.
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.RecoveryInProgressException):
Failed to APPEND_FILE
/app-logs/ambari-qa/bucket-logs-ifile/0007/application_1565102314214_0007/yarnDocker-1_45454_1565335809907
for DFSClient_NONMAPREDUCE_-1185242013_202 on 172.26.86.24 because lease
recovery is in progress. Try again later.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2697)
at
org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.appendFile(FSDirAppendOp.java:125)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2745)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:823)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:500)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:529)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1001)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:929)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2920)
at
org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.initializeWriter(LogAggregationIndexedFileController.java:227)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.uploadLogsForContainers(AppLogAggregatorImpl.java:312)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.doAppLogAggregation(AppLogAggregatorImpl.java:482)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.run(AppLogAggregatorImpl.java:449)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$1.run(LogAggregationService.java:295)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.RecoveryInProgressException):
Failed to APPEND_FILE
/app-logs/ambari-qa/bucket-logs-ifile/0007/application_1565102314214_0007/yarnDocker-1_45454_1565335809907
for DFSClient_NONMAPREDUCE_-1185242013_202 on 172.26.86.24 because lease
recovery is in progress. Try again later.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2697)
at
org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.appendFile(FSDirAppendOp.java:125)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2745)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:823)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:500)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
[jira] [Commented] (YARN-9718) Yarn REST API, services endpoint remote command ejection
[ https://issues.apache.org/jira/browse/YARN-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904167#comment-16904167 ] Eric Yang commented on YARN-9718: - Patch 004 fixed checkstyle issues from patch 003. > Yarn REST API, services endpoint remote command ejection > > > Key: YARN-9718 > URL: https://issues.apache.org/jira/browse/YARN-9718 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.1.2 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9718.001.patch, YARN-9718.002.patch, > YARN-9718.003.patch, YARN-9718.004.patch > > > Email from Oskars Vegeris: > > During internal infrastructure testing it was discovered that the Hadoop Yarn > REST endpoint /app/v1/services contains a command injection vulnerability. > > The services endpoint's normal use-case is for launching containers (e.g. > Docker images/apps), however by providing an argument with special shell > characters it is possible to execute arbitrary commands on the Host server - > this would allow to escalate privileges and access. > > The command injection is possible in the parameter for JVM options - > "yarn.service.am.java.opts". It's possible to enter arbitrary shell commands > by using sub-shell syntax `cmd` or $(cmd). No shell character filtering is > performed. > > The "launch_command" which needs to be provided is meant for the container > and if it's not being run in privileged mode or with special options, host OS > should not be accessible. > > I've attached a minimal request sample with an injected 'ping' command. The > endpoint can also be found via UI @ > [http://yarn-resource-manager:8088/ui2/#/yarn-services] > > If no auth, or "simple auth" (username) is enabled, commands can be executed > on the host OS. I know commands can also be ran by the "new-application" > feature, however this is clearly not meant to be a way to touch the host OS. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9718) Yarn REST API, services endpoint remote command ejection
[ https://issues.apache.org/jira/browse/YARN-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-9718: Attachment: YARN-9718.004.patch > Yarn REST API, services endpoint remote command ejection > > > Key: YARN-9718 > URL: https://issues.apache.org/jira/browse/YARN-9718 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.1.2 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: YARN-9718.001.patch, YARN-9718.002.patch, > YARN-9718.003.patch, YARN-9718.004.patch > > > Email from Oskars Vegeris: > > During internal infrastructure testing it was discovered that the Hadoop Yarn > REST endpoint /app/v1/services contains a command injection vulnerability. > > The services endpoint's normal use-case is for launching containers (e.g. > Docker images/apps), however by providing an argument with special shell > characters it is possible to execute arbitrary commands on the Host server - > this would allow to escalate privileges and access. > > The command injection is possible in the parameter for JVM options - > "yarn.service.am.java.opts". It's possible to enter arbitrary shell commands > by using sub-shell syntax `cmd` or $(cmd). No shell character filtering is > performed. > > The "launch_command" which needs to be provided is meant for the container > and if it's not being run in privileged mode or with special options, host OS > should not be accessible. > > I've attached a minimal request sample with an injected 'ping' command. The > endpoint can also be found via UI @ > [http://yarn-resource-manager:8088/ui2/#/yarn-services] > > If no auth, or "simple auth" (username) is enabled, commands can be executed > on the host OS. I know commands can also be ran by the "new-application" > feature, however this is clearly not meant to be a way to touch the host OS. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904159#comment-16904159 ] Eric Yang commented on YARN-9442: - [~Jim_Brennan] I agree with you regarding read permission. Sorry, I meant to say file stats operation for looking into the directory for files like container directory in application directory, or distributed cache symlink/timestamp in working directory. Those maybe required by mapreduce distrubted cache, but let us know how it turns out. Thanks > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904142#comment-16904142 ] Jim Brennan commented on YARN-9442: --- [~eyang] read permission is needed for directory listing. execute permissions would allow that group to access files in the directory where the files themselves have appropriate permissions. But I think all of the NM setup/access of the working directory is done as a privileged operation, so the group read permissions are not needed for that. > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904140#comment-16904140 ] Eric Yang commented on YARN-9442: - [~Jim_Brennan] I could be wrong, but I thought the group executable permission is required for node manager to perform file listing operations for various admin operations like check application id, container id, or distributed cache symlink etc. > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6492) Generate queue metrics for each partition
[
https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904139#comment-16904139
]
Hadoop QA commented on YARN-6492:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
57s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 2 new + 16 unchanged - 0 fixed = 18 total (was 16) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 152 new + 222 unchanged - 4 fixed = 374 total (was 226)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 40 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 1s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.TestPartitionQueueMetrics |
| |
hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetricsForCustomResources
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivitiesWithMultiNodesEnabled
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerMultiNodes
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-6492 |
| JIRA Patch URL |
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904132#comment-16904132 ] Jim Brennan commented on YARN-9442: --- I am actually testing out a change - [~ebadger] and I discussed off-line why we need 0710 vs 0700 permissions. I can't think of a reason why we need execute only group permissions. So I'm testing that out that change and will put up another patch shortly. > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904131#comment-16904131 ] Eric Yang commented on YARN-9442: - +1 for commit and back port. > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9442) container working directory has group read permissions
[ https://issues.apache.org/jira/browse/YARN-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904118#comment-16904118 ] Eric Badger commented on YARN-9442: --- [~Jim_Brennan], [~eyang], [~shaneku...@gmail.com], [~jeagles], what are your thoughts on committing this all the way back to 2.8? Any objection? > container working directory has group read permissions > -- > > Key: YARN-9442 > URL: https://issues.apache.org/jira/browse/YARN-9442 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.2.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: YARN-9442.001.patch, YARN-9442.002.patch, > YARN-9442.003.patch > > > Container working directories are currently created with permissions 0750, > owned by the user and with the group set to the node manager group. > Is there any reason why these directories need group read permissions? > I have been testing with group read permissions removed and so far I haven't > encountered any problems. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9527) Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file
[ https://issues.apache.org/jira/browse/YARN-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904112#comment-16904112 ] Hudson commented on YARN-9527: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17078 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17078/]) YARN-9527. Prevent rogue Localizer Runner from downloading same file (eyang: rev 6ff0453edeeb0ed7bc9a7d3fb6dfa7048104238b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file > - > > Key: YARN-9527 > URL: https://issues.apache.org/jira/browse/YARN-9527 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.8.5, 3.1.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9527.001.patch, YARN-9527.002.patch, > YARN-9527.003.patch, YARN-9527.004.patch > > > A rogue ContainerLocalizer can get stuck in a loop continuously downloading > the same file while generating an "Invalid event: LOCALIZED at LOCALIZED" > exception on each iteration. Sometimes this continues long enough that it > fills up a disk or depletes available inodes for the filesystem. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9527) Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file
[ https://issues.apache.org/jira/browse/YARN-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904109#comment-16904109 ] Jim Brennan commented on YARN-9527: --- Thanks [~eyang] and [~ebadger]! > Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file > - > > Key: YARN-9527 > URL: https://issues.apache.org/jira/browse/YARN-9527 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.8.5, 3.1.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9527.001.patch, YARN-9527.002.patch, > YARN-9527.003.patch, YARN-9527.004.patch > > > A rogue ContainerLocalizer can get stuck in a loop continuously downloading > the same file while generating an "Invalid event: LOCALIZED at LOCALIZED" > exception on each iteration. Sometimes this continues long enough that it > fills up a disk or depletes available inodes for the filesystem. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9527) Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file
[ https://issues.apache.org/jira/browse/YARN-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904104#comment-16904104 ] Eric Badger commented on YARN-9527: --- Thanks, [~eyang]! > Rogue LocalizerRunner/ContainerLocalizer repeatedly downloading same file > - > > Key: YARN-9527 > URL: https://issues.apache.org/jira/browse/YARN-9527 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.8.5, 3.1.2 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9527.001.patch, YARN-9527.002.patch, > YARN-9527.003.patch, YARN-9527.004.patch > > > A rogue ContainerLocalizer can get stuck in a loop continuously downloading > the same file while generating an "Invalid event: LOCALIZED at LOCALIZED" > exception on each iteration. Sometimes this continues long enough that it > fills up a disk or depletes available inodes for the filesystem. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9732) yarn.system-metrics-publisher.enabled=false is not honored by RM
[ https://issues.apache.org/jira/browse/YARN-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904045#comment-16904045 ] Hudson commented on YARN-9732: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17077 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17077/]) YARN-9732. yarn.system-metrics-publisher.enabled=false is not honored by (abmodi: rev a79564fed0b641f7207f5066500128326a3ac296) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMTimelineService.java > yarn.system-metrics-publisher.enabled=false is not honored by RM > > > Key: YARN-9732 > URL: https://issues.apache.org/jira/browse/YARN-9732 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineclient >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Assignee: KWON BYUNGCHANG >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9732.0001.patch > > > RM does not use yarn.system-metrics-publisher.enabled=false, > so if configure only yarn.timeline-service.enabled=true, > YARN system metrics are always published on the timeline server by RM > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9732) yarn.system-metrics-publisher.enabled=false is not honored by RM
[ https://issues.apache.org/jira/browse/YARN-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904037#comment-16904037 ] Abhishek Modi commented on YARN-9732: - Thanks [~magnum] for patch and [~Prabhu Joseph] for review. Committed to trunk. > yarn.system-metrics-publisher.enabled=false is not honored by RM > > > Key: YARN-9732 > URL: https://issues.apache.org/jira/browse/YARN-9732 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineclient >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Assignee: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9732.0001.patch > > > RM does not use yarn.system-metrics-publisher.enabled=false, > so if configure only yarn.timeline-service.enabled=true, > YARN system metrics are always published on the timeline server by RM > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9732) yarn.system-metrics-publisher.enabled=false is not honored by RM
[ https://issues.apache.org/jira/browse/YARN-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-9732: Summary: yarn.system-metrics-publisher.enabled=false is not honored by RM (was: yarn.system-metrics-publisher.enabled=false does not work) > yarn.system-metrics-publisher.enabled=false is not honored by RM > > > Key: YARN-9732 > URL: https://issues.apache.org/jira/browse/YARN-9732 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineclient >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Assignee: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9732.0001.patch > > > RM does not use yarn.system-metrics-publisher.enabled=false, > so if configure only yarn.timeline-service.enabled=true, > YARN system metrics are always published on the timeline server by RM > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6492) Generate queue metrics for each partition
[ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-6492: --- Attachment: YARN-6492.005.WIP.patch > Generate queue metrics for each partition > - > > Key: YARN-6492 > URL: https://issues.apache.org/jira/browse/YARN-6492 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > Attachments: PartitionQueueMetrics_default_partition.txt, > PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt, > YARN-6492.001.patch, YARN-6492.002.patch, YARN-6492.003.patch, > YARN-6492.004.patch, YARN-6492.005.WIP.patch, partition_metrics.txt > > > We are interested in having queue metrics for all partitions. Right now each > queue has one QueueMetrics object which captures metrics either in default > partition or across all partitions. (After YARN-6467 it will be in default > partition) > But having the partition metrics would be very useful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6492) Generate queue metrics for each partition
[
https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904027#comment-16904027
]
Manikandan R commented on YARN-6492:
Ok, [~eepayne]. Will look into this.
Some observations on .004.patch are
1. Since partition info are being extracted from request and node, there is a
problem. For example,
Node N has been mapped to Label X (Non exclusive). Queue A has been configured
with ANY Node label. App A requested resources from Queue A and its containers
ran on Node N for some reasons. During AbstractCSQueue#allocateResource call,
Node partition (using SchedulerNode ) would get used for calculation. Lets say
allocate call has been fired for 3 containers of 1 GB each, then
a. PartitionDefault * queue A -> pending mb is 3 GB
b. PartitionX * queue A -> pending mb is -3 GB
is the outcome. Because app request has been fired without any label
specification and #a metrics has been derived. After allocation is over,
pending resources usually gets decreased. When this happens, it use node
partition info. hence #b metrics has derived.
Given this kind of situation, We will need to put some thoughts on achieving
the metrics correctly.
2. Though the intent of this jira is to do Partition Queue Metrics, we would
like to retain the existing Queue Metrics for backward compatibility (as you
can see from jira's discussion).
With this patch and YARN-9596 patch, queuemetrics (for queue's) would be
overridden either with some specific partition values or default partition
values. It could be vice - versa as well. For example, after the queues (say
queue A) has been initialised with some min and max cap and also with node
label's min and max cap, Queuemetrics (availableMB) for queue A return values
based on node label's cap config.
I've been working on these observations to provide a fix and attached
.005.WIP.patch. Focus of .005.WIP.patch is to ensure availableMB,
availableVcores is correct (Please refer above #2 observation). Added more
asserts in {{testQueueMetricsWithLabelsOnDefaultLabelNode}} to ensure fix for
#2 is working properly.
Also one more thing to note is, user metrics for availableMB, availableVcores
at root queue was not there even before. Retained the same behaviour. User
metrics for availableMB, availableVcores is available only at child queue's
level and also with partitions.
Will focus on #1 in next patch.
> Generate queue metrics for each partition
> -
>
> Key: YARN-6492
> URL: https://issues.apache.org/jira/browse/YARN-6492
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Jonathan Hung
>Assignee: Manikandan R
>Priority: Major
> Attachments: PartitionQueueMetrics_default_partition.txt,
> PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt,
> YARN-6492.001.patch, YARN-6492.002.patch, YARN-6492.003.patch,
> YARN-6492.004.patch, partition_metrics.txt
>
>
> We are interested in having queue metrics for all partitions. Right now each
> queue has one QueueMetrics object which captures metrics either in default
> partition or across all partitions. (After YARN-6467 it will be in default
> partition)
> But having the partition metrics would be very useful.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904018#comment-16904018
]
Hadoop QA commented on YARN-9719:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 6s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9719 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977183/YARN-9719.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ac69bec57048 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e25a5c2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24512/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24512/testReport/ |
| Max. process+thread count | 731 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
[jira] [Commented] (YARN-8586) Extract log aggregation related fields and methods from RMAppImpl
[
https://issues.apache.org/jira/browse/YARN-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903989#comment-16903989
]
Hadoop QA commented on YARN-8586:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 105 unchanged - 10 fixed = 107 total (was 115)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 17s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-8586 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977161/YARN-8586.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5ce487f6b806 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 43a91f8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24511/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
|
[jira] [Updated] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kyungwan nam updated YARN-9719:
---
Attachment: YARN-9719.006.patch
> Failed to restart yarn-service if it doesn’t exist in RM
>
>
> Key: YARN-9719
> URL: https://issues.apache.org/jira/browse/YARN-9719
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9719.001.patch, YARN-9719.002.patch,
> YARN-9719.003.patch, YARN-9719.004.patch, YARN-9719.005.patch,
> YARN-9719.006.patch
>
>
> Sometimes, restarting a yarn-service is failed as follows.
> {code}
> {"diagnostics":"Application with id 'application_1562735362534_10461' doesn't
> exist in RM. Please check that the job submission was successful.\n\tat
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)\n\tat
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)\n\tat
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)\n\tat
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)\n\tat
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)\n\tat
> java.security.AccessController.doPrivileged(Native Method)\n\tat
> javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)\n\tat
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)\n"}
> {code}
> It seems like that it occurs when restarting a yarn-service who was stopped
> long ago.
> by default, RM keeps up to 1000 completed applications
> (yarn.resourcemanager.max-completed-applications)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kyungwan nam updated YARN-9719:
---
Attachment: (was: YARN-9719.006.patch)
> Failed to restart yarn-service if it doesn’t exist in RM
>
>
> Key: YARN-9719
> URL: https://issues.apache.org/jira/browse/YARN-9719
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9719.001.patch, YARN-9719.002.patch,
> YARN-9719.003.patch, YARN-9719.004.patch, YARN-9719.005.patch
>
>
> Sometimes, restarting a yarn-service is failed as follows.
> {code}
> {"diagnostics":"Application with id 'application_1562735362534_10461' doesn't
> exist in RM. Please check that the job submission was successful.\n\tat
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)\n\tat
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)\n\tat
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)\n\tat
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)\n\tat
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)\n\tat
> java.security.AccessController.doPrivileged(Native Method)\n\tat
> javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)\n\tat
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)\n"}
> {code}
> It seems like that it occurs when restarting a yarn-service who was stopped
> long ago.
> by default, RM keeps up to 1000 completed applications
> (yarn.resourcemanager.max-completed-applications)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kyungwan nam updated YARN-9719:
---
Attachment: YARN-9719.006.patch
> Failed to restart yarn-service if it doesn’t exist in RM
>
>
> Key: YARN-9719
> URL: https://issues.apache.org/jira/browse/YARN-9719
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9719.001.patch, YARN-9719.002.patch,
> YARN-9719.003.patch, YARN-9719.004.patch, YARN-9719.005.patch,
> YARN-9719.006.patch
>
>
> Sometimes, restarting a yarn-service is failed as follows.
> {code}
> {"diagnostics":"Application with id 'application_1562735362534_10461' doesn't
> exist in RM. Please check that the job submission was successful.\n\tat
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)\n\tat
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)\n\tat
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)\n\tat
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)\n\tat
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)\n\tat
> java.security.AccessController.doPrivileged(Native Method)\n\tat
> javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)\n\tat
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)\n"}
> {code}
> It seems like that it occurs when restarting a yarn-service who was stopped
> long ago.
> by default, RM keeps up to 1000 completed applications
> (yarn.resourcemanager.max-completed-applications)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903963#comment-16903963
]
Abhishek Modi commented on YARN-9731:
-
Thanks [~magnum] for the patch. Thanks [~Prabhu Joseph] for review. Patch looks
good to me.
[~magnum] could you please take care of check-style warnings.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9732) yarn.system-metrics-publisher.enabled=false does not work
[ https://issues.apache.org/jira/browse/YARN-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903961#comment-16903961 ] Abhishek Modi commented on YARN-9732: - Thanks [~magnum] for the patch. I will commit it in couple of hours. > yarn.system-metrics-publisher.enabled=false does not work > - > > Key: YARN-9732 > URL: https://issues.apache.org/jira/browse/YARN-9732 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, timelineclient >Affects Versions: 3.1.2 >Reporter: KWON BYUNGCHANG >Assignee: KWON BYUNGCHANG >Priority: Major > Attachments: YARN-9732.0001.patch > > > RM does not use yarn.system-metrics-publisher.enabled=false, > so if configure only yarn.timeline-service.enabled=true, > YARN system metrics are always published on the timeline server by RM > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8586) Extract log aggregation related fields and methods from RMAppImpl
[ https://issues.apache.org/jira/browse/YARN-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-8586: --- Attachment: YARN-8586.003.patch > Extract log aggregation related fields and methods from RMAppImpl > - > > Key: YARN-8586 > URL: https://issues.apache.org/jira/browse/YARN-8586 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-8586.001.patch, YARN-8586.002.patch, > YARN-8586.002.patch, YARN-8586.003.patch > > > Given that RMAppImpl is already above 2000 lines and it is very complex, as a > very simple > and straightforward step, all Log aggregation related fields and methods > could be extracted to a new class. > The clients of RMAppImpl may access the same methods and RMAppImpl would > delegate all those calls to the newly introduced class. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7721) TestContinuousScheduling fails sporadically with NPE
[
https://issues.apache.org/jira/browse/YARN-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903848#comment-16903848
]
Hadoop QA commented on YARN-7721:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m
58s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-7721 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945331/YARN-7721.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d470aef07b80 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f6fa865 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24510/testReport/ |
| Max. process+thread count | 926 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24510/console |
| Powered by | Apache
[jira] [Commented] (YARN-8199) Logging fileSize of log files under NM Local Dir
[ https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903827#comment-16903827 ] Prabhu Joseph commented on YARN-8199: - Thanks [~snemeth]. > Logging fileSize of log files under NM Local Dir > > > Key: YARN-8199 > URL: https://issues.apache.org/jira/browse/YARN-8199 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: supportability > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch, > YARN-8199-003.patch, YARN-8199-004.patch, YARN-8199-branch-3.1.001.patch, > YARN-8199-branch-3.2.001.patch > > > Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir > by NodeManager before the cleanup will help to find the application which has > written too verbose. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8199) Logging fileSize of log files under NM Local Dir
[ https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8199: - Hadoop Flags: Reviewed > Logging fileSize of log files under NM Local Dir > > > Key: YARN-8199 > URL: https://issues.apache.org/jira/browse/YARN-8199 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: supportability > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch, > YARN-8199-003.patch, YARN-8199-004.patch, YARN-8199-branch-3.1.001.patch, > YARN-8199-branch-3.2.001.patch > > > Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir > by NodeManager before the cleanup will help to find the application which has > written too verbose. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8199) Logging fileSize of log files under NM Local Dir
[ https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903820#comment-16903820 ] Szilard Nemeth commented on YARN-8199: -- Hi [~Prabhu Joseph]]! Thanks for the other patches, commited to branch-3.2 and branch-3.1! Thanks [~adam.antal]] for the reviews! > Logging fileSize of log files under NM Local Dir > > > Key: YARN-8199 > URL: https://issues.apache.org/jira/browse/YARN-8199 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: supportability > Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch, > YARN-8199-003.patch, YARN-8199-004.patch, YARN-8199-branch-3.1.001.patch, > YARN-8199-branch-3.2.001.patch > > > Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir > by NodeManager before the cleanup will help to find the application which has > written too verbose. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[
https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903816#comment-16903816
]
Hadoop QA commented on YARN-9217:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 15 unchanged - 2 fixed = 15 total (was 17) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
1s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
54s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| | Possible null pointer dereference in
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer.lookUpAutoDiscoveryBinary(Configuration)
due to return value of called method Dereferenced at
GpuDiscoverer.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer.lookUpAutoDiscoveryBinary(Configuration)
due to return value of called method Dereferenced at GpuDiscoverer.java:[line
288] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977135/YARN-9217.008.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2b5688a20ea4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018
[jira] [Commented] (YARN-9134) No test coverage for redefining FPGA / GPU resource types in TestResourceUtils
[
https://issues.apache.org/jira/browse/YARN-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903807#comment-16903807
]
Hadoop QA commented on YARN-9134:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
The patch generated 0 new + 3 unchanged - 6 fixed = 3 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
59s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9134 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977121/YARN-9134.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 646857d04e9d 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f6fa865 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24507/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24507/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> No test coverage for redefining FPGA /
[jira] [Commented] (YARN-9133) Make tests more easy to comprehend in TestGpuResourceHandler
[ https://issues.apache.org/jira/browse/YARN-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903805#comment-16903805 ] Peter Bacsko commented on YARN-9133: [~snemeth] similarly to YARN-9140, conflict resolution might not be trivial, also this file has been renamed in the meantime and it wasn't treated as a simple rename (see commits [https://github.com/apache/hadoop/commit/b8bf09ba3d2514ccfa3c6beb4a7530cd2f3555c7] and [https://github.com/apache/hadoop/commit/7c2042a44d1cd7e60b911cb40642cdd9c443b076]). Not sure if this is worth it. > Make tests more easy to comprehend in TestGpuResourceHandler > > > Key: YARN-9133 > URL: https://issues.apache.org/jira/browse/YARN-9133 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9133.001.patch, YARN-9133.001.patch, > YARN-9133.002.patch, YARN-9133.003.patch, YARN-9133.004.patch, > YARN-9133.005.patch, YARN-9133.006.patch, YARN-9133.006.patch > > > Tests are not quite easy to read: > - Some more helper methods would improve readability. > - Eliminating the boolean flag that controls if docker is used would also > improve readability and clarity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903802#comment-16903802 ] Peter Bacsko commented on YARN-9140: [~snemeth] there are already 4 commits difference between trunk vs branch-3.2 in TestResourcePluginManager. I'm not sure that this change is worth the backport to these earlier branches. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9715) [UI2] yarn-container-log URI need to be encoded to avoid potential misuses
[ https://issues.apache.org/jira/browse/YARN-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903800#comment-16903800 ] Hudson commented on YARN-9715: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17073 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17073/]) YARN-9715. [UI2] yarn-container-log URI need to be encoded to avoid (sunilg: rev acffec7a92be540aa8531dbe06a3ea7bb813ab93) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-node-container.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-node-app.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-container-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-node-apps.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-node-containers.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-node-container.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-node-containers.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-node-apps.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-container-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-node-app.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-node-app.js > [UI2] yarn-container-log URI need to be encoded to avoid potential misuses > -- > > Key: YARN-9715 > URL: https://issues.apache.org/jira/browse/YARN-9715 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Prabhu Joseph >Assignee: Akhil PB >Priority: Major > Attachments: Screen Shot 2019-08-08 at 12.54.40 PM.png, Screen Shot > 2019-08-08 at 12.55.03 PM.png, Screen Shot 2019-08-08 at 2.51.46 PM.png, > Screen Shot 2019-08-08 at 3.03.16 PM.png, YARN-9715.001.patch, > YARN-9715.002.patch > > > Currently yarn-container-log (UI2 - Nodes - List of Containers - log file) > creates url with node scheme (http) and nodeHttpAddress. This does not work > with Knox Gateway https url. The logic to construct url can be improved to > accept both normal and knox case. The similar way is used in Applications -> > Logs Section. > And also UI2 - Nodes - List of Containers - log file does not have pagination > support for log file. > > *Screenshot of Problematic Page *: Knox Url - UI2 - Nodes - List of > Containers - log file > !Screen Shot 2019-08-08 at 3.03.16 PM.png|height=200|width=350! > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903792#comment-16903792
]
Hadoop QA commented on YARN-9731:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
The patch generated 4 new + 105 unchanged - 10 fixed = 109 total (was 115)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
49s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9731 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977117/YARN-9731.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3c59e33d1bc4 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f6fa865 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24504/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24504/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (YARN-9729) [UI2] Fix error message for logs without ATSv2
[
https://issues.apache.org/jira/browse/YARN-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903787#comment-16903787
]
Hadoop QA commented on YARN-9729:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-9729 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9729 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24508/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [UI2] Fix error message for logs without ATSv2
> --
>
> Key: YARN-9729
> URL: https://issues.apache.org/jira/browse/YARN-9729
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-ui-v2
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Zoltan Siegl
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: ATS_NOT_UP.png, ATS_UP_WITH_NO_LOGS.png, Screenshot
> 2019-08-08 at 13.23.11.png, Screenshot 2019-08-08 at 13.23.21.png, Screenshot
> 2019-08-09 at 3.22.19 PM.png, YARN-9729.001.patch, after_patch.png
>
>
> On UI2 applications page logs are not available unless ATSv2 is running. The
> reason for logs not to appear is unclarified on the UI.
> When ATS is reported to be unhealthy, a descriptive error message should
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[ https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9217: --- Attachment: YARN-9217.008.patch > Nodemanager will fail to start if GPU is misconfigured on the node or GPU > drivers missing > - > > Key: YARN-9217 > URL: https://issues.apache.org/jira/browse/YARN-9217 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9217.001.patch, YARN-9217.002.patch, > YARN-9217.003.patch, YARN-9217.004.patch, YARN-9217.005.patch, > YARN-9217.006.patch, YARN-9217.007.patch, YARN-9217.008.patch > > > Nodemanager will not start > 1. If Autodiscovery is enabled: > * If nvidia-smi path is misconfigured or the file does not exist. > * There is 0 GPU found > * If the file exists but it is not pointing to an nvidia-smi > * if the binary is ok but there is an IOException > 2. If the manually configured GPU devices are misconfigured > * Any index:minor number format failure will cause a problem > * 0 configured device will cause a problem > * NumberFormatException is not handled > It would be a better option to add warnings about the configuration, set 0 > available GPUs and let the node work and run non-gpu jobs. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9692) ContainerAllocationExpirer is missspelled
[ https://issues.apache.org/jira/browse/YARN-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903782#comment-16903782 ] Adam Antal commented on YARN-9692: -- Hi [~runzhouwu], Thanks for filing this issue. To be honest, it seems to me that the class is named properly. Could you tell me some arguments why it has to be changed? IMO this is the class responsible for expiring ContainerAllocation, that is why it was named ContainerAllocationExpirer: the class which expires the ContainerAllocation related objects. > ContainerAllocationExpirer is missspelled > - > > Key: YARN-9692 > URL: https://issues.apache.org/jira/browse/YARN-9692 > Project: Hadoop YARN > Issue Type: Bug >Reporter: runzhou wu >Assignee: runzhou wu >Priority: Trivial > Labels: newbie > Attachments: YARN-9692.001.patch > > > The class ContainerAllocationExpirer is missspelled. > I think it should be changed to ContainerAllocationExpired -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[
https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903776#comment-16903776
]
Hadoop QA commented on YARN-9217:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} YARN-9217 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977115/YARN-9217.007.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24506/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Nodemanager will fail to start if GPU is misconfigured on the node or GPU
> drivers missing
> -
>
> Key: YARN-9217
> URL: https://issues.apache.org/jira/browse/YARN-9217
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9217.001.patch, YARN-9217.002.patch,
> YARN-9217.003.patch, YARN-9217.004.patch, YARN-9217.005.patch,
> YARN-9217.006.patch, YARN-9217.007.patch
>
>
> Nodemanager will not start
> 1. If Autodiscovery is enabled:
> * If nvidia-smi path is misconfigured or the file does not exist.
> * There is 0 GPU found
> * If the file exists but it is not pointing to an nvidia-smi
> * if the binary is ok but there is an IOException
> 2. If the manually configured GPU devices are misconfigured
> * Any index:minor number format failure will cause a problem
> * 0 configured device will cause a problem
> * NumberFormatException is not handled
> It would be a better option to add warnings about the configuration, set 0
> available GPUs and let the node work and run non-gpu jobs.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7721) TestContinuousScheduling fails sporadically with NPE
[
https://issues.apache.org/jira/browse/YARN-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903773#comment-16903773
]
Szilard Nemeth commented on YARN-7721:
--
Hi [~sunilg]!
Would you reupload the latest patch?
If we have a green jenkins result, I'd happily commit this.
Thanks!
> TestContinuousScheduling fails sporadically with NPE
>
>
> Key: YARN-7721
> URL: https://issues.apache.org/jira/browse/YARN-7721
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Jason Lowe
>Assignee: Sunil Govindan
>Priority: Major
> Attachments: YARN-7721.001.patch
>
>
> TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime is
> failing sporadically with an NPE in precommit builds, and I can usually
> reproduce it locally after a few tries:
> {noformat}
> [ERROR]
> testFairSchedulerContinuousSchedulingInitTime(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling)
> Time elapsed: 0.085 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling.testFairSchedulerContinuousSchedulingInitTime(TestContinuousScheduling.java:383)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [...]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9729) [UI2] Fix error message for logs without ATSv2
[
https://issues.apache.org/jira/browse/YARN-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903768#comment-16903768
]
Akhil PB commented on YARN-9729:
[~zsiegl] Could you please use follow code snippet in {{isTimelineUnHealthy}}
if block in logs.hbs file.
{noformat}
Logs are unavailable because Application Timeline Service seems
unhealthy.
{noformat}
The above code will produce !Screenshot 2019-08-09 at 3.22.19 PM.png!
cc: [~sunilg]
> [UI2] Fix error message for logs without ATSv2
> --
>
> Key: YARN-9729
> URL: https://issues.apache.org/jira/browse/YARN-9729
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-ui-v2
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Zoltan Siegl
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: ATS_NOT_UP.png, ATS_UP_WITH_NO_LOGS.png, Screenshot
> 2019-08-08 at 13.23.11.png, Screenshot 2019-08-08 at 13.23.21.png, Screenshot
> 2019-08-09 at 3.22.19 PM.png, YARN-9729.001.patch, after_patch.png
>
>
> On UI2 applications page logs are not available unless ATSv2 is running. The
> reason for logs not to appear is unclarified on the UI.
> When ATS is reported to be unhealthy, a descriptive error message should
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9729) [UI2] Fix error message for logs without ATSv2
[ https://issues.apache.org/jira/browse/YARN-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-9729: --- Attachment: Screenshot 2019-08-09 at 3.22.19 PM.png > [UI2] Fix error message for logs without ATSv2 > -- > > Key: YARN-9729 > URL: https://issues.apache.org/jira/browse/YARN-9729 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Major > Attachments: ATS_NOT_UP.png, ATS_UP_WITH_NO_LOGS.png, Screenshot > 2019-08-08 at 13.23.11.png, Screenshot 2019-08-08 at 13.23.21.png, Screenshot > 2019-08-09 at 3.22.19 PM.png, YARN-9729.001.patch, after_patch.png > > > On UI2 applications page logs are not available unless ATSv2 is running. The > reason for logs not to appear is unclarified on the UI. > When ATS is reported to be unhealthy, a descriptive error message should > appear. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9124) Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently
[
https://issues.apache.org/jira/browse/YARN-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903761#comment-16903761
]
Szilard Nemeth commented on YARN-9124:
--
Hi [~adam.antal]!
Thanks for the latest patches for other branches, committed to branch-3.2 and
branch-3.1!
> Resolve contradiction in ResourceUtils: addMandatoryResources /
> checkMandatoryResources work differently
>
>
> Key: YARN-9124
> URL: https://issues.apache.org/jira/browse/YARN-9124
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9124.branch-3.1.001.patch,
> YARN-9124.branch-3.2.001.patch, YARN-9124.branch-3.2.001.patch
>
>
> {{ResourceUtils#addMandatoryResources}}: Adds only memory and vcores as
> mandatory resources.
> {{ResourceUtils#checkMandatoryResources}}: YARN-6620 added some code to this.
> This method not only checks memory and vcores, but all the resources referred
> in ResourceInformation#MANDATORY_RESOURCES.
> I think it would be good to call {{MANDATORY_RESOURCES}} as
> {{PREDEFINED_RESOURCES}} or something like that and use a similar name for
> {{checkMandatoryResources}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9134) No test coverage for redefining FPGA / GPU resource types in TestResourceUtils
[ https://issues.apache.org/jira/browse/YARN-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903741#comment-16903741 ] Peter Bacsko commented on YARN-9134: Uploaded patch v4. _"Also, setupResourceTypes has been deprecated as it is dangerous to use, see the javadoc for details."_ - I decided to fix this issue instead of deprecating the method. > No test coverage for redefining FPGA / GPU resource types in TestResourceUtils > -- > > Key: YARN-9134 > URL: https://issues.apache.org/jira/browse/YARN-9134 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9134.001.patch, YARN-9134.002.patch, > YARN-9134.003.patch, YARN-9134.004.patch > > > The patch also includes some trivial code cleanup. > Also, setupResourceTypes has been deprecated as it is dangerous to use, see > the javadoc for details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9134) No test coverage for redefining FPGA / GPU resource types in TestResourceUtils
[ https://issues.apache.org/jira/browse/YARN-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9134: --- Attachment: YARN-9134.004.patch > No test coverage for redefining FPGA / GPU resource types in TestResourceUtils > -- > > Key: YARN-9134 > URL: https://issues.apache.org/jira/browse/YARN-9134 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9134.001.patch, YARN-9134.002.patch, > YARN-9134.003.patch, YARN-9134.004.patch > > > The patch also includes some trivial code cleanup. > Also, setupResourceTypes has been deprecated as it is dangerous to use, see > the javadoc for details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903739#comment-16903739
]
Hadoop QA commented on YARN-9719:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core:
The patch generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 52s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9719 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977107/YARN-9719.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ca2c4765f199 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e0c21c6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24502/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
|
| unit |
[jira] [Commented] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[
https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903737#comment-16903737
]
Hadoop QA commented on YARN-9217:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-9217 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977115/YARN-9217.007.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24505/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Nodemanager will fail to start if GPU is misconfigured on the node or GPU
> drivers missing
> -
>
> Key: YARN-9217
> URL: https://issues.apache.org/jira/browse/YARN-9217
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9217.001.patch, YARN-9217.002.patch,
> YARN-9217.003.patch, YARN-9217.004.patch, YARN-9217.005.patch,
> YARN-9217.006.patch, YARN-9217.007.patch
>
>
> Nodemanager will not start
> 1. If Autodiscovery is enabled:
> * If nvidia-smi path is misconfigured or the file does not exist.
> * There is 0 GPU found
> * If the file exists but it is not pointing to an nvidia-smi
> * if the binary is ok but there is an IOException
> 2. If the manually configured GPU devices are misconfigured
> * Any index:minor number format failure will cause a problem
> * 0 configured device will cause a problem
> * NumberFormatException is not handled
> It would be a better option to add warnings about the configuration, set 0
> available GPUs and let the node work and run non-gpu jobs.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903706#comment-16903706
]
KWON BYUNGCHANG edited comment on YARN-9731 at 8/9/19 8:38 AM:
---
[~Prabhu Joseph] I've attached patch fixed testcases.
was (Author: magnum):
[~Prabhu Joseph] I've attaced patch fixed testcases.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KWON BYUNGCHANG updated YARN-9731:
--
Attachment: YARN-9731.002.patch
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903706#comment-16903706
]
KWON BYUNGCHANG commented on YARN-9731:
---
[~Prabhu Joseph] I've attaced patch fixed testcases.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, YARN-9731.002.patch,
> ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9092) Create an object for cgroups mount enable and cgroups mount path as they belong together
[ https://issues.apache.org/jira/browse/YARN-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903703#comment-16903703 ] Hudson commented on YARN-9092: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17071 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17071/]) YARN-9092. Create an object for cgroups mount enable and cgroups mount (snemeth: rev e0c21c6da91776caf661661a19c368939c81fcc4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMountConfig.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java > Create an object for cgroups mount enable and cgroups mount path as they > belong together > > > Key: YARN-9092 > URL: https://issues.apache.org/jira/browse/YARN-9092 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Minor > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9092.001.patch, YARN-9092.002.patch, > YARN-9092.003.patch > > > YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT and > YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH are used in > conjunction many places in the code, so for the sake of readabilty and > simplicity, it is better to wrap the values of these configs to an object and > use it instead of having 2 fields in > CGroupsHandlerImpl and in CgroupsLCEResourcesHandler as well. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9727) Allowed Origin pattern is discouraged if regex contains *
[ https://issues.apache.org/jira/browse/YARN-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903700#comment-16903700 ] Hudson commented on YARN-9727: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17071 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17071/]) YARN-9727: Allowed Origin pattern is discouraged if regex contains *. (snemeth: rev df30d8ea092cddc037482bc60fc790b26b577963) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/http/CrossOriginFilter.java > Allowed Origin pattern is discouraged if regex contains * > - > > Key: YARN-9727 > URL: https://issues.apache.org/jira/browse/YARN-9727 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Minor > Attachments: YARN-9727.001.patch > > > HADOOP-14908 if allowed-origins regex contains any * characters an > incorrectwarning log is triggered: "Allowed Origin pattern > 'regex:.*[.]example[.]com' is discouraged, use the 'regex:' prefix and use a > Java regular expression instead." > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9094) Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM
[ https://issues.apache.org/jira/browse/YARN-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903701#comment-16903701 ] Hudson commented on YARN-9094: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17071 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17071/]) YARN-9094: Remove unused interface method: (snemeth: rev 72d7e570a73989aa18b737c0e642d570a55c6781) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/NodeResourceUpdaterPlugin.java > Remove unused interface method: > NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM > - > > Key: YARN-9094 > URL: https://issues.apache.org/jira/browse/YARN-9094 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Trivial > Labels: newbie, newbie++ > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9094.001.patch, YARN-9094.001.patch > > > Additionally, there's a typo can be fixed in the javadoc of > NodeResourceUpdaterPlugin#updateConfiguredResource: look for "mododule" -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9096) Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily
[
https://issues.apache.org/jira/browse/YARN-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903702#comment-16903702
]
Hudson commented on YARN-9096:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17071 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17071/])
YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are
(snemeth: rev 742e30b47381ad63e2b2fe63738cd0fe6cbce106)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuResourcePlugin.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/ResourcePluginManager.java
> Some GpuResourcePlugin and ResourcePluginManager methods are synchronized
> unnecessarily
> ---
>
> Key: YARN-9096
> URL: https://issues.apache.org/jira/browse/YARN-9096
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Gergely Pollak
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9096.001.patch, YARN-9096.002.patch,
> YARN-9096.002.patch
>
>
> These methods are not used concurrently, they are part of the initialization
> code of NM that happens from one thread.
> This is the list of the call hierarchies:
> 1. GpuResourcePlugin.initialize + ResourcePluginManager.initialize
>
> {code:java}
> GpuResourcePlugin.initialize(Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> ResourcePluginManager.initialize(Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin)
> NodeManager.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager){code}
>
>
> 2. GpuResourcePlugin.createResourceHandler:
>
> {code:java}
> GpuResourcePlugin.createResourceHandler(Context, CGroupsHandler,
> PrivilegedOperationExecutor)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> ResourceHandlerModule.addHandlersFromConfiguredResourcePlugins(List,
> Configuration, Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ResourceHandlerModule.initializeConfiguredResourceHandlerChain(Configuration,
> Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ResourceHandlerModule.getConfiguredResourceHandlerChain(Configuration,
> Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ContainerScheduler.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler)
> LinuxContainerExecutor.init(Context)
> (org.apache.hadoop.yarn.server.nodemanager)
> {code}
>
> 3. GpuResourcePlugin.getNodeResourceHandlerInstance:
>
> {code:java}
> GpuResourcePlugin.getNodeResourceHandlerInstance()
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> NodeStatusUpdaterImpl.updateConfiguredResourcesViaPlugins(Resource)(2 usages)
> (org.apache.hadoop.yarn.server.nodemanager)
> NodeStatusUpdaterImpl.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9723) ApplicationPlacementContext is not required for terminated jobs during recovery
[
https://issues.apache.org/jira/browse/YARN-9723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903698#comment-16903698
]
Prabhu Joseph commented on YARN-9723:
-
Thanks [~adam.antal] for reviewing.
[~snemeth] Can you review this patch when you get time.
> ApplicationPlacementContext is not required for terminated jobs during
> recovery
> ---
>
> Key: YARN-9723
> URL: https://issues.apache.org/jira/browse/YARN-9723
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9723-001.patch
>
>
>Placement of application (RMAppManager.placeApplication) is called for all
> the jobs during recovery. This can be ignored for the terminated jobs.
> {code}
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.AppNameMappingPlacementRule.getPlacementForApp(AppNameMappingPlacementRule.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementManager.placeApplication(PlacementManager.java:66)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.placeApplication(RMAppManager.java:867)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:421)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:410)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:637)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1536)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9722) PlacementRule logs object ID in place of queue name.
[
https://issues.apache.org/jira/browse/YARN-9722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903699#comment-16903699
]
Prabhu Joseph commented on YARN-9722:
-
Thanks [~sunilg] for reviewing.
[~snemeth] Can you review and commit this patch when you get time. Thanks.
> PlacementRule logs object ID in place of queue name.
>
>
> Key: YARN-9722
> URL: https://issues.apache.org/jira/browse/YARN-9722
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Labels: supportability
> Attachments: YARN-9722-001.patch
>
>
> UserGroupMappingPlacementRule logs object ID in place of queue name.
> {code}
> 2019-08-05 09:28:52,664 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1564996871731_0003 user ambari-qa mapping [default]
> to
> [org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext@5aafe9b2]
> override false
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9723) ApplicationPlacementContext is not required for terminated jobs during recovery
[
https://issues.apache.org/jira/browse/YARN-9723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903696#comment-16903696
]
Adam Antal commented on YARN-9723:
--
This patch looks straightforward. +1 (non-binding)
> ApplicationPlacementContext is not required for terminated jobs during
> recovery
> ---
>
> Key: YARN-9723
> URL: https://issues.apache.org/jira/browse/YARN-9723
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9723-001.patch
>
>
>Placement of application (RMAppManager.placeApplication) is called for all
> the jobs during recovery. This can be ignored for the terminated jobs.
> {code}
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.AppNameMappingPlacementRule.getPlacementForApp(AppNameMappingPlacementRule.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementManager.placeApplication(PlacementManager.java:66)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.placeApplication(RMAppManager.java:867)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:421)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:410)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:637)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1536)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9134) No test coverage for redefining FPGA / GPU resource types in TestResourceUtils
[ https://issues.apache.org/jira/browse/YARN-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903692#comment-16903692 ] Szilard Nemeth commented on YARN-9134: -- Hi [~pbacsko]! Could you please check the javac issue above? Also, could you please update branch-3.2 and branch-3.1 patches, if applicable? Thanks! > No test coverage for redefining FPGA / GPU resource types in TestResourceUtils > -- > > Key: YARN-9134 > URL: https://issues.apache.org/jira/browse/YARN-9134 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9134.001.patch, YARN-9134.002.patch, > YARN-9134.003.patch > > > The patch also includes some trivial code cleanup. > Also, setupResourceTypes has been deprecated as it is dangerous to use, see > the javadoc for details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[ https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9217: --- Attachment: YARN-9217.007.patch > Nodemanager will fail to start if GPU is misconfigured on the node or GPU > drivers missing > - > > Key: YARN-9217 > URL: https://issues.apache.org/jira/browse/YARN-9217 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9217.001.patch, YARN-9217.002.patch, > YARN-9217.003.patch, YARN-9217.004.patch, YARN-9217.005.patch, > YARN-9217.006.patch, YARN-9217.007.patch > > > Nodemanager will not start > 1. If Autodiscovery is enabled: > * If nvidia-smi path is misconfigured or the file does not exist. > * There is 0 GPU found > * If the file exists but it is not pointing to an nvidia-smi > * if the binary is ok but there is an IOException > 2. If the manually configured GPU devices are misconfigured > * Any index:minor number format failure will cause a problem > * 0 configured device will cause a problem > * NumberFormatException is not handled > It would be a better option to add warnings about the configuration, set 0 > available GPUs and let the node work and run non-gpu jobs. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9096) Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily
[
https://issues.apache.org/jira/browse/YARN-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-9096:
-
Fix Version/s: 3.1.3
3.2.1
3.3.0
> Some GpuResourcePlugin and ResourcePluginManager methods are synchronized
> unnecessarily
> ---
>
> Key: YARN-9096
> URL: https://issues.apache.org/jira/browse/YARN-9096
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Gergely Pollak
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9096.001.patch, YARN-9096.002.patch,
> YARN-9096.002.patch
>
>
> These methods are not used concurrently, they are part of the initialization
> code of NM that happens from one thread.
> This is the list of the call hierarchies:
> 1. GpuResourcePlugin.initialize + ResourcePluginManager.initialize
>
> {code:java}
> GpuResourcePlugin.initialize(Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> ResourcePluginManager.initialize(Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin)
> NodeManager.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager){code}
>
>
> 2. GpuResourcePlugin.createResourceHandler:
>
> {code:java}
> GpuResourcePlugin.createResourceHandler(Context, CGroupsHandler,
> PrivilegedOperationExecutor)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> ResourceHandlerModule.addHandlersFromConfiguredResourcePlugins(List,
> Configuration, Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ResourceHandlerModule.initializeConfiguredResourceHandlerChain(Configuration,
> Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ResourceHandlerModule.getConfiguredResourceHandlerChain(Configuration,
> Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ContainerScheduler.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler)
> LinuxContainerExecutor.init(Context)
> (org.apache.hadoop.yarn.server.nodemanager)
> {code}
>
> 3. GpuResourcePlugin.getNodeResourceHandlerInstance:
>
> {code:java}
> GpuResourcePlugin.getNodeResourceHandlerInstance()
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> NodeStatusUpdaterImpl.updateConfiguredResourcesViaPlugins(Resource)(2 usages)
> (org.apache.hadoop.yarn.server.nodemanager)
> NodeStatusUpdaterImpl.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9094) Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM
[ https://issues.apache.org/jira/browse/YARN-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9094: - Fix Version/s: 3.1.3 3.2.1 3.3.0 > Remove unused interface method: > NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM > - > > Key: YARN-9094 > URL: https://issues.apache.org/jira/browse/YARN-9094 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Trivial > Labels: newbie, newbie++ > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9094.001.patch, YARN-9094.001.patch > > > Additionally, there's a typo can be fixed in the javadoc of > NodeResourceUpdaterPlugin#updateConfiguredResource: look for "mododule" -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9092) Create an object for cgroups mount enable and cgroups mount path as they belong together
[ https://issues.apache.org/jira/browse/YARN-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903688#comment-16903688 ] Szilard Nemeth commented on YARN-9092: -- Hi [~shuzirra]]! Thanks for this patch, committed to trunk, branch-3.2 and branch-3.1! Thanks [~pbacsko]] for the reviews! > Create an object for cgroups mount enable and cgroups mount path as they > belong together > > > Key: YARN-9092 > URL: https://issues.apache.org/jira/browse/YARN-9092 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Minor > Attachments: YARN-9092.001.patch, YARN-9092.002.patch, > YARN-9092.003.patch > > > YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT and > YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH are used in > conjunction many places in the code, so for the sake of readabilty and > simplicity, it is better to wrap the values of these configs to an object and > use it instead of having 2 fields in > CGroupsHandlerImpl and in CgroupsLCEResourcesHandler as well. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9729) [UI2] Fix error message for logs without ATSv2
[
https://issues.apache.org/jira/browse/YARN-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903687#comment-16903687
]
Hadoop QA commented on YARN-9729:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red} YARN-9729 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9729 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24503/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [UI2] Fix error message for logs without ATSv2
> --
>
> Key: YARN-9729
> URL: https://issues.apache.org/jira/browse/YARN-9729
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-ui-v2
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Zoltan Siegl
>Assignee: Zoltan Siegl
>Priority: Major
> Attachments: ATS_NOT_UP.png, ATS_UP_WITH_NO_LOGS.png, Screenshot
> 2019-08-08 at 13.23.11.png, Screenshot 2019-08-08 at 13.23.21.png,
> YARN-9729.001.patch, after_patch.png
>
>
> On UI2 applications page logs are not available unless ATSv2 is running. The
> reason for logs not to appear is unclarified on the UI.
> When ATS is reported to be unhealthy, a descriptive error message should
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9732) yarn.system-metrics-publisher.enabled=false does not work
[
https://issues.apache.org/jira/browse/YARN-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903685#comment-16903685
]
Hadoop QA commented on YARN-9732:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m
40s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9732 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977090/YARN-9732.0001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d3a4ef52ad7a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 88ed1e0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24500/testReport/ |
| Max. process+thread count | 882 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24500/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> yarn.system-metrics-publisher.enabled=false does not
[jira] [Commented] (YARN-9681) AM resource limit is incorrect for queue
[
https://issues.apache.org/jira/browse/YARN-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903679#comment-16903679
]
Hadoop QA commented on YARN-9681:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m
23s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9681 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977097/YARN-9681.0005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3f8d86dae91f 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 88ed1e0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24498/testReport/ |
| Max. process+thread count | 889 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24498/console |
| Powered by |
[jira] [Commented] (YARN-9096) Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily
[
https://issues.apache.org/jira/browse/YARN-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903678#comment-16903678
]
Szilard Nemeth commented on YARN-9096:
--
Hi [[~shuzirra]!
Thanks for the latest patch, committed to trunk, branch-3.2 and branch-3.1!
Thanks [~adam.antal]] for the reviews!
> Some GpuResourcePlugin and ResourcePluginManager methods are synchronized
> unnecessarily
> ---
>
> Key: YARN-9096
> URL: https://issues.apache.org/jira/browse/YARN-9096
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-9096.001.patch, YARN-9096.002.patch,
> YARN-9096.002.patch
>
>
> These methods are not used concurrently, they are part of the initialization
> code of NM that happens from one thread.
> This is the list of the call hierarchies:
> 1. GpuResourcePlugin.initialize + ResourcePluginManager.initialize
>
> {code:java}
> GpuResourcePlugin.initialize(Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> ResourcePluginManager.initialize(Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin)
> NodeManager.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager){code}
>
>
> 2. GpuResourcePlugin.createResourceHandler:
>
> {code:java}
> GpuResourcePlugin.createResourceHandler(Context, CGroupsHandler,
> PrivilegedOperationExecutor)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> ResourceHandlerModule.addHandlersFromConfiguredResourcePlugins(List,
> Configuration, Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ResourceHandlerModule.initializeConfiguredResourceHandlerChain(Configuration,
> Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ResourceHandlerModule.getConfiguredResourceHandlerChain(Configuration,
> Context)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources)
> ContainerScheduler.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler)
> LinuxContainerExecutor.init(Context)
> (org.apache.hadoop.yarn.server.nodemanager)
> {code}
>
> 3. GpuResourcePlugin.getNodeResourceHandlerInstance:
>
> {code:java}
> GpuResourcePlugin.getNodeResourceHandlerInstance()
> (org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu)
> NodeStatusUpdaterImpl.updateConfiguredResourcesViaPlugins(Resource)(2 usages)
> (org.apache.hadoop.yarn.server.nodemanager)
> NodeStatusUpdaterImpl.serviceInit(Configuration)
> (org.apache.hadoop.yarn.server.nodemanager)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9094) Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM
[ https://issues.apache.org/jira/browse/YARN-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903672#comment-16903672 ] Szilard Nemeth commented on YARN-9094: -- Hi [~shuzirra]! Thanks for this patch! +1 committed to trunk, branch-3.2 and branch-3.1. Thanks [~pbacsko] for the review! > Remove unused interface method: > NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM > - > > Key: YARN-9094 > URL: https://issues.apache.org/jira/browse/YARN-9094 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Trivial > Labels: newbie, newbie++ > Attachments: YARN-9094.001.patch, YARN-9094.001.patch > > > Additionally, there's a typo can be fixed in the javadoc of > NodeResourceUpdaterPlugin#updateConfiguredResource: look for "mododule" -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9217) Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
[ https://issues.apache.org/jira/browse/YARN-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903665#comment-16903665 ] Szilard Nemeth commented on YARN-9217: -- Hi [~pbacsko]! Please fix the checkstyle, findbugs and unit failures! > Nodemanager will fail to start if GPU is misconfigured on the node or GPU > drivers missing > - > > Key: YARN-9217 > URL: https://issues.apache.org/jira/browse/YARN-9217 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.0.0, 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9217.001.patch, YARN-9217.002.patch, > YARN-9217.003.patch, YARN-9217.004.patch, YARN-9217.005.patch, > YARN-9217.006.patch > > > Nodemanager will not start > 1. If Autodiscovery is enabled: > * If nvidia-smi path is misconfigured or the file does not exist. > * There is 0 GPU found > * If the file exists but it is not pointing to an nvidia-smi > * if the binary is ok but there is an IOException > 2. If the manually configured GPU devices are misconfigured > * Any index:minor number format failure will cause a problem > * 0 configured device will cause a problem > * NumberFormatException is not handled > It would be a better option to add warnings about the configuration, set 0 > available GPUs and let the node work and run non-gpu jobs. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9727) Allowed Origin pattern is discouraged if regex contains *
[ https://issues.apache.org/jira/browse/YARN-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903657#comment-16903657 ] Szilard Nemeth commented on YARN-9727: -- Hi [~zsiegl]! Committed to trunk! Please verify if we need this patch on branch-3.2 / branch-3.1 If you determine we need the patches, please justify and I can commit them! Thanks! > Allowed Origin pattern is discouraged if regex contains * > - > > Key: YARN-9727 > URL: https://issues.apache.org/jira/browse/YARN-9727 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Minor > Attachments: YARN-9727.001.patch > > > HADOOP-14908 if allowed-origins regex contains any * characters an > incorrectwarning log is triggered: "Allowed Origin pattern > 'regex:.*[.]example[.]com' is discouraged, use the 'regex:' prefix and use a > Java regular expression instead." > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9733) Method getCpuUsagePercent in Class ProcfsBasedProcessTree return 0 when subprocess of container dead
[ https://issues.apache.org/jira/browse/YARN-9733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903658#comment-16903658 ] qian han commented on YARN-9733: Thanks > Method getCpuUsagePercent in Class ProcfsBasedProcessTree return 0 when > subprocess of container dead > > > Key: YARN-9733 > URL: https://issues.apache.org/jira/browse/YARN-9733 > Project: Hadoop YARN > Issue Type: Bug >Reporter: qian han >Assignee: qian han >Priority: Major > > The method getTotalProcessJiffies only gets jiffies for running processes not > dead processes. > For example, process pid100 and its children pid200 and pid300. > We call getCpuUsagePercent the first time, assume that pid100 has a jiffies > 1000, pid200 2000 and pid300 3000. The totalProcessJiffies1 is 6000. > And We kill pid300. Then we call getCpuUsagePercent the second time, assume > that pid100 has a jiffies 1100, pid200 2200. The totalProcessJiffies2 is 3300. > So we got a cpu usage percent 0. > I would like to fix this bug. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9727) Allowed Origin pattern is discouraged if regex contains *
[ https://issues.apache.org/jira/browse/YARN-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903654#comment-16903654 ] Szilard Nemeth commented on YARN-9727: -- Hi [~zsiegl]! +1, committing this soon. > Allowed Origin pattern is discouraged if regex contains * > - > > Key: YARN-9727 > URL: https://issues.apache.org/jira/browse/YARN-9727 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Minor > Attachments: YARN-9727.001.patch > > > HADOOP-14908 if allowed-origins regex contains any * characters an > incorrectwarning log is triggered: "Allowed Origin pattern > 'regex:.*[.]example[.]com' is discouraged, use the 'regex:' prefix and use a > Java regular expression instead." > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9729) [UI2] Fix error message for logs without ATSv2
[ https://issues.apache.org/jira/browse/YARN-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903650#comment-16903650 ] Szilard Nemeth commented on YARN-9729: -- Hi [~zsiegl]! I will start to review this once we have a green jenkins result. In the meantime, could you please upload branch-3.2 / branch-3.1 patches? Thanks! > [UI2] Fix error message for logs without ATSv2 > -- > > Key: YARN-9729 > URL: https://issues.apache.org/jira/browse/YARN-9729 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Affects Versions: 3.2.0, 3.1.2 >Reporter: Zoltan Siegl >Assignee: Zoltan Siegl >Priority: Major > Attachments: ATS_NOT_UP.png, ATS_UP_WITH_NO_LOGS.png, Screenshot > 2019-08-08 at 13.23.11.png, Screenshot 2019-08-08 at 13.23.21.png, > YARN-9729.001.patch, after_patch.png > > > On UI2 applications page logs are not available unless ATSv2 is running. The > reason for logs not to appear is unclarified on the UI. > When ATS is reported to be unhealthy, a descriptive error message should > appear. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903648#comment-16903648
]
Prabhu Joseph commented on YARN-9731:
-
[~magnum] Failed testcases are related, can you fix the same.
> In ATS v1.5, all jobs are visible to all users without view-acl
> ---
>
> Key: YARN-9731
> URL: https://issues.apache.org/jira/browse/YARN-9731
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.1.2
>Reporter: KWON BYUNGCHANG
>Assignee: KWON BYUNGCHANG
>Priority: Major
> Attachments: YARN-9731.001.patch, ats_v1.5_screenshot.png
>
>
> In ATS v1.5 of secure mode,
> all jobs are visible to all users without view-acl.
> if user does not have view-acl, user should not be able to see jobs.
> I attatched ATS UI screenshot.
>
> ATS v1.5 log
> {code:java}
> 2019-08-09 10:21:13,679 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1954. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1954
> 2019-08-09 10:21:13,680 WARN
> applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
> (ApplicationHistoryManagerOnTimelineStore.java:generateApplicationReport(687))
> - Failed to authorize when generating application report for
> application_1565247558150_1951. Use a placeholder for its latest attempt id.
> org.apache.hadoop.security.authorize.AuthorizationException: User magnum does
> not have privilege to see this application application_1565247558150_1951
> {code}
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9731) In ATS v1.5, all jobs are visible to all users without view-acl
[
https://issues.apache.org/jira/browse/YARN-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903626#comment-16903626
]
Hadoop QA commented on YARN-9731:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
The patch generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 28s{color}
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryManagerOnTimelineStore
|
| | hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9731 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977087/YARN-9731.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c477a1600cb2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 88ed1e0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Updated] (YARN-9719) Failed to restart yarn-service if it doesn’t exist in RM
[
https://issues.apache.org/jira/browse/YARN-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kyungwan nam updated YARN-9719:
---
Attachment: YARN-9719.005.patch
> Failed to restart yarn-service if it doesn’t exist in RM
>
>
> Key: YARN-9719
> URL: https://issues.apache.org/jira/browse/YARN-9719
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9719.001.patch, YARN-9719.002.patch,
> YARN-9719.003.patch, YARN-9719.004.patch, YARN-9719.005.patch
>
>
> Sometimes, restarting a yarn-service is failed as follows.
> {code}
> {"diagnostics":"Application with id 'application_1562735362534_10461' doesn't
> exist in RM. Please check that the job submission was successful.\n\tat
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:382)\n\tat
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:234)\n\tat
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:561)\n\tat
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)\n\tat
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)\n\tat
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)\n\tat
> java.security.AccessController.doPrivileged(Native Method)\n\tat
> javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)\n\tat
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)\n"}
> {code}
> It seems like that it occurs when restarting a yarn-service who was stopped
> long ago.
> by default, RM keeps up to 1000 completed applications
> (yarn.resourcemanager.max-completed-applications)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9715) [UI2] yarn-container-log URI need to be encoded to avoid potential misuses
[ https://issues.apache.org/jira/browse/YARN-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-9715: - Summary: [UI2] yarn-container-log URI need to be encoded to avoid potential misuses (was: [YARN UI2] yarn-container-log support for https Knox Gateway url in nodes page) > [UI2] yarn-container-log URI need to be encoded to avoid potential misuses > -- > > Key: YARN-9715 > URL: https://issues.apache.org/jira/browse/YARN-9715 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Prabhu Joseph >Assignee: Akhil PB >Priority: Major > Attachments: Screen Shot 2019-08-08 at 12.54.40 PM.png, Screen Shot > 2019-08-08 at 12.55.03 PM.png, Screen Shot 2019-08-08 at 2.51.46 PM.png, > Screen Shot 2019-08-08 at 3.03.16 PM.png, YARN-9715.001.patch, > YARN-9715.002.patch > > > Currently yarn-container-log (UI2 - Nodes - List of Containers - log file) > creates url with node scheme (http) and nodeHttpAddress. This does not work > with Knox Gateway https url. The logic to construct url can be improved to > accept both normal and knox case. The similar way is used in Applications -> > Logs Section. > And also UI2 - Nodes - List of Containers - log file does not have pagination > support for log file. > > *Screenshot of Problematic Page *: Knox Url - UI2 - Nodes - List of > Containers - log file > !Screen Shot 2019-08-08 at 3.03.16 PM.png|height=200|width=350! > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
