[jira] [Commented] (YARN-9709) When we expanding queue list the scheduler page will not show any applications
[ https://issues.apache.org/jira/browse/YARN-9709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907837#comment-16907837 ] WEI-HSIAO-LEE commented on YARN-9709: - If nobody is working on it, could I assign it to me? > When we expanding queue list the scheduler page will not show any applications > -- > > Key: YARN-9709 > URL: https://issues.apache.org/jira/browse/YARN-9709 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > Attachments: list1.png, list3.png > > > When we expanding queue list the scheduler page will not show any > applications.But it works well in FairScheduler. > !list1.png! > !list3.png! -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9746) Rm should only rewrite partial jobConf passed by app when supporting multi-cluster token renew
[
https://issues.apache.org/jira/browse/YARN-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated YARN-9746:
---
Attachment: YARN-9746-01.patch
> Rm should only rewrite partial jobConf passed by app when supporting
> multi-cluster token renew
> --
>
> Key: YARN-9746
> URL: https://issues.apache.org/jira/browse/YARN-9746
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junfan Zhang
>Priority: Major
> Attachments: YARN-9746-01.patch
>
>
> This issue links to YARN-5910.
> When to support multi-cluster delegation token renew, the path of YARN-5910
> works in most scenarios.
> But when intergrating with Oozie, we encounter some problems. In Oozie having
> multi delegation tokens including HDFS_DELEGATION_TOKEN(another cluster HA
> token) and MR_DELEGATION_TOKEN(Oozie mr launcher token), to support renew
> another cluster's token, YARN-5910 was patched and related config was set.
> The config is as follows
> {code:xml}
>
> mapreduce.job.send-token-conf
>
> dfs.namenode.kerberos.principal|dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$
>
>
> dfs.nameservices
>
> hadoop-clusterA-ns01,hadoop-clusterA-ns02,hadoop-clusterA-ns03,hadoop-clusterA-ns04,hadoop-clusterB-ns01,hadoop-clusterB-ns02,hadoop-clusterB-ns03,hadoop-clusterB-ns04
>
>
> dfs.ha.namenodes.hadoop-clusterB-ns01
> nn1,nn2
>
>
>
> dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn1
> namenode01-clusterB.qiyi.hadoop:8020
>
>
>
> dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn2
> namenode02-clusterB.qiyi.hadoop:8020
>
>
>
> dfs.client.failover.proxy.provider.hadoop-clusterB-ns01
>
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
> {code}
> However, the MR_DELEGATION_TOKEN could‘t be renewed, because of lacking some
> config. Although we can set the required configurations through the app, this
> is not a good idea. So i think rm should only rewrite the jobConf passed by
> app to solve the above situation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9746) Rm should only rewrite partial jobConf passed by app when supporting multi-cluster token renew
[
https://issues.apache.org/jira/browse/YARN-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated YARN-9746:
---
Attachment: (was: YARN-9746-01.path)
> Rm should only rewrite partial jobConf passed by app when supporting
> multi-cluster token renew
> --
>
> Key: YARN-9746
> URL: https://issues.apache.org/jira/browse/YARN-9746
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junfan Zhang
>Priority: Major
> Attachments: YARN-9746-01.patch
>
>
> This issue links to YARN-5910.
> When to support multi-cluster delegation token renew, the path of YARN-5910
> works in most scenarios.
> But when intergrating with Oozie, we encounter some problems. In Oozie having
> multi delegation tokens including HDFS_DELEGATION_TOKEN(another cluster HA
> token) and MR_DELEGATION_TOKEN(Oozie mr launcher token), to support renew
> another cluster's token, YARN-5910 was patched and related config was set.
> The config is as follows
> {code:xml}
>
> mapreduce.job.send-token-conf
>
> dfs.namenode.kerberos.principal|dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$
>
>
> dfs.nameservices
>
> hadoop-clusterA-ns01,hadoop-clusterA-ns02,hadoop-clusterA-ns03,hadoop-clusterA-ns04,hadoop-clusterB-ns01,hadoop-clusterB-ns02,hadoop-clusterB-ns03,hadoop-clusterB-ns04
>
>
> dfs.ha.namenodes.hadoop-clusterB-ns01
> nn1,nn2
>
>
>
> dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn1
> namenode01-clusterB.qiyi.hadoop:8020
>
>
>
> dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn2
> namenode02-clusterB.qiyi.hadoop:8020
>
>
>
> dfs.client.failover.proxy.provider.hadoop-clusterB-ns01
>
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
> {code}
> However, the MR_DELEGATION_TOKEN could‘t be renewed, because of lacking some
> config. Although we can set the required configurations through the app, this
> is not a good idea. So i think rm should only rewrite the jobConf passed by
> app to solve the above situation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9746) Rm should only rewrite partial jobConf passed by app when supporting multi-cluster token renew
[
https://issues.apache.org/jira/browse/YARN-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junfan Zhang updated YARN-9746:
---
Issue Type: Bug (was: Improvement)
> Rm should only rewrite partial jobConf passed by app when supporting
> multi-cluster token renew
> --
>
> Key: YARN-9746
> URL: https://issues.apache.org/jira/browse/YARN-9746
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junfan Zhang
>Priority: Major
> Attachments: YARN-9746-01.path
>
>
> This issue links to YARN-5910.
> When to support multi-cluster delegation token renew, the path of YARN-5910
> works in most scenarios.
> But when intergrating with Oozie, we encounter some problems. In Oozie having
> multi delegation tokens including HDFS_DELEGATION_TOKEN(another cluster HA
> token) and MR_DELEGATION_TOKEN(Oozie mr launcher token), to support renew
> another cluster's token, YARN-5910 was patched and related config was set.
> The config is as follows
> {code:xml}
>
> mapreduce.job.send-token-conf
>
> dfs.namenode.kerberos.principal|dfs.nameservices|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$|^dfs.client.failover.proxy.provider.*$
>
>
> dfs.nameservices
>
> hadoop-clusterA-ns01,hadoop-clusterA-ns02,hadoop-clusterA-ns03,hadoop-clusterA-ns04,hadoop-clusterB-ns01,hadoop-clusterB-ns02,hadoop-clusterB-ns03,hadoop-clusterB-ns04
>
>
> dfs.ha.namenodes.hadoop-clusterB-ns01
> nn1,nn2
>
>
>
> dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn1
> namenode01-clusterB.qiyi.hadoop:8020
>
>
>
> dfs.namenode.rpc-address.hadoop-clusterB-ns01.nn2
> namenode02-clusterB.qiyi.hadoop:8020
>
>
>
> dfs.client.failover.proxy.provider.hadoop-clusterB-ns01
>
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
> {code}
> However, the MR_DELEGATION_TOKEN could‘t be renewed, because of lacking some
> config. Although we can set the required configurations through the app, this
> is not a good idea. So i think rm should only rewrite the jobConf passed by
> app to solve the above situation.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9748) Allow capacity-scheduler configuration on HDFS
[ https://issues.apache.org/jira/browse/YARN-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907757#comment-16907757 ] zhoukang commented on YARN-9748: Sure [~Prabhu Joseph] We have implemented an inner version in out production cluster > Allow capacity-scheduler configuration on HDFS > -- > > Key: YARN-9748 > URL: https://issues.apache.org/jira/browse/YARN-9748 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907697#comment-16907697 ] Eric Yang commented on YARN-9683: - Thank you [~jojochuang] for resolving this pull request properly. > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9562) Add Java changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907566#comment-16907566
]
Hadoop QA commented on YARN-9562:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 282 new + 689 unchanged - 1 fixed = 971 total (was 690) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 15 new + 0 unchanged - 0 fixed = 15 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 50s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 5s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| | Nullcheck of NodeManager.context at line 535 of value previously
dereferenced in
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop() At
NodeManager.java:535 of value previously dereferenced in
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop() At
NodeManager.java:[line 532] |
| | Unused field:NodeManager.java |
| | Dead store to refreshHdfsCacheThread in
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.ImageTagToManifestPlugin.serviceStart()
At
ImageTa
[jira] [Commented] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
[
https://issues.apache.org/jira/browse/YARN-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907542#comment-16907542
]
Hadoop QA commented on YARN-9488:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m
14s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9488 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977621/YARN-9488-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4bd03d3345fb 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 06d8ac9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24566/testReport/ |
| Max. process+thread count | 882 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24566/console |
| Powered by | Apache Ye
[jira] [Commented] (YARN-9735) Allow User Keytab to submit YARN Native Service
[
https://issues.apache.org/jira/browse/YARN-9735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907512#comment-16907512
]
Prabhu Joseph commented on YARN-9735:
-
Yes got it. Thanks [~eyang].
> Allow User Keytab to submit YARN Native Service
>
>
> Key: YARN-9735
> URL: https://issues.apache.org/jira/browse/YARN-9735
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-native-services
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> Yarn Native Service launch fails on a secure cluster with user keytab. It
> allows only service keytab. Have seen most of the users test their jobs with
> user keytab.
> {code}
> [ambari-qa@pjosephdocker-3 ~]$ yarn app -launch sleeper-service
> /usr/hdp/3.0.1.0-187/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
> 19/08/03 17:17:04 ERROR client.ApiServiceClient: Kerberos principal
> (ambari-qa-pjosephdoc...@docker.com) does not contain a hostname.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9290) Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler
[
https://issues.apache.org/jira/browse/YARN-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907508#comment-16907508
]
Prabhu Joseph commented on YARN-9290:
-
Thanks [~snemeth] for checking this.
*Overall design:*
When there is a invalid scheduling request made by an application, the allocate
will retry for configured retry attempts
({{yarn.resourcemanager.placement-constraints.retry-attempts)}} and then sets
the invalid SchedulingRequests in AllocateResponse so that ApplicationMaster is
aware.
*Changes:*
{{AppPlacementAllocator.java}} - Maintains the retry made so far. Added new
instance variable placementAttempt to track it.
{{SingleConstraintAppPlacementAllocator.java}} - Increments the retry attempt
when the scheduling request is invalid.
{{AppSchedulingInfo.java}} - Rejects the SchedulingRequest if the retry
attempts has exceeded the configured value.
{{Allocation.java}} - Maintains the list of rejected SchedulingRequest.
Constructors changed to accept the list.
{{FiCaSchedulerApp.java}} - Creates Allocation object with the list of rejected
scheduling requests. It fetches them from
AppSchedulingInfo#getSchedulingRequest.
{{DefaultAMSProcessor.java}} - Sets the rejected scheduling requests in
AllocateResponse which is passed to AM.
{{FairScheduler.java}} - Changed the Allocation constructor call.
*Test Cases:*
{{TestSchedulingRequestContainerAllocation.java}} - Actual TestCase which
verifies the rejection of invalid scheduling request.
All Below test classes are added with getYarnConfiguration in the mock
RMContext object as AppSchedulingInfo is changed to get the YarnConfigiration
from RMContext to read configured
{{yarn.resourcemanager.placement-constraints.retry-attempts}}.
{{TestAppSchedulingInfo.java}}
{{TestSchedulerApplicationAttempt.java}}
{{TestLeafQueue.java}}
{{TestUtils.java}}
{{TestFSAppAttempt.java}}
{{TestMaxRunningAppsEnforcer.java}}
{{TestQueueManager.java}}
{{TestFifoScheduler.java}}
> Invalid SchedulingRequest not rejected in Scheduler
> PlacementConstraintsHandler
>
>
> Key: YARN-9290
> URL: https://issues.apache.org/jira/browse/YARN-9290
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9290-001.patch, YARN-9290-002.patch,
> YARN-9290-003.patch, YARN-9290-004.patch, YARN-9290-005.patch,
> YARN-9290-006.patch
>
>
> SchedulingRequest with Invalid namespace is not rejected in Scheduler
> PlacementConstraintsHandler. RM keeps on trying to allocateOnNode with
> logging the exception. This is rejected in case of placement-processor
> handler.
> {code}
> 2019-02-08 16:51:27,548 WARN
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator:
> Failed to query node cardinality:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.InvalidAllocationTagsQueryException:
> Invalid namespace prefix: notselfi, valid values are:
> all,not-self,app-id,app-tag,self
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TargetApplicationsNamespace.fromString(TargetApplicationsNamespace.java:277)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TargetApplicationsNamespace.parse(TargetApplicationsNamespace.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.AllocationTags.createAllocationTags(AllocationTags.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfySingleConstraintExpression(PlacementConstraintsUtil.java:78)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfySingleConstraint(PlacementConstraintsUtil.java:240)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:321)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyAndConstraint(PlacementConstraintsUtil.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:324)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:365)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.checkCardinalityAndPending(SingleConstraintAppPlacementAllocator.java:355)
> at
> org.apache.hadoop.yarn.server
[jira] [Commented] (YARN-9562) Add Java changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907493#comment-16907493
]
Eric Badger commented on YARN-9562:
---
{noformat:title=Required Configs}
yarn.nodemanager.runtime.linux.allowed-runtimes
docker,default,runc
yarn.nodemanager.runtime.linux.runc.allowed-images
$Name of image tag
yarn.nodemanager.runtime.linux.runc.image-name
$Name of image tag
yarn.nodemanager.runtime.linux.runc.image-toplevel-dir
/runc-root
{noformat}
{noformat:title=Doesn't need to be nscd, but you need some strategy to make
sure that your username can be resolved in the container via its uid.}
yarn.nodemanager.runtime.linux.runc.default-rw-mounts
/var/run/nscd:/var/run/nscd
{noformat}
{noformat:title=At least 1 of the following 2 configs needs to be set. If you
use the docker-to-squash tool from YARN 9564 then you should only need to set
the hdfs hash file}
yarn.nodemanager.runtime.linux.runc.image-tag-to-manifest-plugin.local-hash-file
/home/ebadger/image-tag-to-hash
yarn.nodemanager.runtime.linux.runc.image-tag-to-manifest-plugin.hdfs-hash-file
/runc-root/image-tag-to-hash
{noformat}
[~eyang], I went ahead and moved the toplevel config into YarnConfiguration
since it is relevant to multiple pieces of the runc runtime. So the config key
is now {{yarn.nodemanager.runtime.linux.runc.image-toplevel-dir}}. I have
copied and edited the original list of configs that I gave. Additionally, I
have tested YARN-9562.004.patch with YARN-9561.003.patch and
YARN-9564.003.patch together.
{noformat}
export
vars="YARN_CONTAINER_RUNTIME_TYPE=runc,YARN_CONTAINER_RUNTIME_RUNC_IMAGE=hadoop/rhel6:current";
$HADOOP_HOME/bin/hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.*-tests.jar
sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
-Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
-Dyarn.app.mapreduce.am.env=$vars -Dmapreduce.map.env=$vars
-Dmapreduce.reduce.env=$vars -mt 1 -rt 1 -m 1 -r 1
{noformat}
I successfully ran the above sleep job (after running docker-to-squash.py on
hadoop/rhel6:current). Also note that the image that you use will need to
include java and any other Hadoop dependencies. We have created our own custom
image that is based off of rhel6 that add java, zstandard, lzo, and some other
packages.
> Add Java changes for the new RuncContainerRuntime
> -
>
> Key: YARN-9562
> URL: https://issues.apache.org/jira/browse/YARN-9562
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9562.001.patch, YARN-9562.002.patch,
> YARN-9562.003.patch, YARN-9562.004.patch
>
>
> This JIRA will be used to add the Java changes for the new
> RuncContainerRuntime. This will work off of YARN-9560 to use much of the
> existing DockerLinuxContainerRuntime code once it is moved up into an
> abstract class that can be extended.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907491#comment-16907491 ] Hudson commented on YARN-9683: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17126 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17126/]) YARN-9683. Remove reapDockerContainerNoPid left behind by YARN-9074 (weichiu: rev 167acd87daf61be9a27dfb5af6e7cbacb96e44de) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerCleanup.java > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9074) Docker container rm command should be executed after stop
[
https://issues.apache.org/jira/browse/YARN-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907492#comment-16907492
]
Hudson commented on YARN-9074:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17126 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17126/])
YARN-9683. Remove reapDockerContainerNoPid left behind by YARN-9074 (weichiu:
rev 167acd87daf61be9a27dfb5af6e7cbacb96e44de)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerCleanup.java
> Docker container rm command should be executed after stop
> -
>
> Key: YARN-9074
> URL: https://issues.apache.org/jira/browse/YARN-9074
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhaohui Xin
>Assignee: Zhaohui Xin
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9074.001.patch, YARN-9074.002.patch,
> YARN-9074.003.patch, image-2018-12-01-11-36-12-448.png,
> image-2018-12-01-11-38-18-191.png
>
>
> {code:java}
> @Override
> public void transition(ContainerImpl container, ContainerEvent event) {
> container.setIsReInitializing(false);
> // Set exit code to 0 on success
> container.exitCode = 0;
> // TODO: Add containerWorkDir to the deletion service.
> if (DockerLinuxContainerRuntime.isDockerContainerRequested(
> container.daemonConf,
> container.getLaunchContext().getEnvironment())) {
> removeDockerContainer(container);
> }
> if (clCleanupRequired) {
> container.dispatcher.getEventHandler().handle(
> new ContainersLauncherEvent(container,
> ContainersLauncherEventType.CLEANUP_CONTAINER));
> }
> container.cleanup();
> }{code}
> Now, when container is finished, NM firstly execute "_docker rm xxx"_ to
> remove it and this thread is placed in DeletionService. see more in YARN-5366
> .
> Next, NM will execute "_docker stop_" and "docker kill" command. these tow
> commands are wrapped up in ContainerCleanup thread and executed by
> ContainersLauncher. see more in YARN-7644.
> The above will cause the container's cleanup to be split into two threads. I
> think we should refactor these code to make all docker container killing
> process be place in ContainerCleanup thread and "_docker rm_" should be
> executed last.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9735) Allow User Keytab to submit YARN Native Service
[
https://issues.apache.org/jira/browse/YARN-9735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved YARN-9735.
-
Resolution: Invalid
[~Prabhu Joseph] User principal is not used as service principal because TGS
request authenticate client principal with service principal, and this
information is validated on the AM side to ensure that KDC pre-authentication
took place, and server can only reconfirm the end user credential based on
validation of Service principals granted to the end user. The service
principal must match the hostname of the running service. Without presence of
hostname in service principal, there is no security validation on service side
to determine that end user is allowed or not. Hence, allowing user principal
to run as service becomes a security hole. This reasoning makes the
implementation invalid. Thank you for trying.
> Allow User Keytab to submit YARN Native Service
>
>
> Key: YARN-9735
> URL: https://issues.apache.org/jira/browse/YARN-9735
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-native-services
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> Yarn Native Service launch fails on a secure cluster with user keytab. It
> allows only service keytab. Have seen most of the users test their jobs with
> user keytab.
> {code}
> [ambari-qa@pjosephdocker-3 ~]$ yarn app -launch sleeper-service
> /usr/hdp/3.0.1.0-187/hadoop-yarn/yarn-service-examples/sleeper/sleeper.json
> 19/08/03 17:17:04 ERROR client.ApiServiceClient: Kerberos principal
> (ambari-qa-pjosephdoc...@docker.com) does not contain a hostname.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9562) Add Java changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-9562: -- Attachment: YARN-9562.004.patch > Add Java changes for the new RuncContainerRuntime > - > > Key: YARN-9562 > URL: https://issues.apache.org/jira/browse/YARN-9562 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9562.001.patch, YARN-9562.002.patch, > YARN-9562.003.patch, YARN-9562.004.patch > > > This JIRA will be used to add the Java changes for the new > RuncContainerRuntime. This will work off of YARN-9560 to use much of the > existing DockerLinuxContainerRuntime code once it is moved up into an > abstract class that can be extended. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907484#comment-16907484 ] Wei-Chiu Chuang commented on YARN-9683: --- I merged the PR. Thanks all! > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907467#comment-16907467 ] kevin su edited comment on YARN-9683 at 8/14/19 5:20 PM: - [~eyang] [~adam.antal] Thanks for the review, but it looks like the patch has not been commit yet was (Author: pingsutw): [~eyang] [~adam.antal] Thanks for the review, but it looks like the patch didn't commit yet > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907467#comment-16907467 ] kevin su commented on YARN-9683: [~eyang] [~adam.antal] Thanks for the review, but it looks like the patch didn't commit yet > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9679) Regular code cleanup in TestResourcePluginManager
[
https://issues.apache.org/jira/browse/YARN-9679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907462#comment-16907462
]
Szilard Nemeth commented on YARN-9679:
--
Hi [~adam.antal]!
Your PR now has conflicts against trunk. Please update it and I will take a
look!
Thanks!
> Regular code cleanup in TestResourcePluginManager
> -
>
> Key: YARN-9679
> URL: https://issues.apache.org/jira/browse/YARN-9679
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Adam Antal
>Priority: Major
> Labels: newbie
>
> There are several things could be cleaned up in this class:
> 1. stubResourcePluginmanager should be private.
> 2. In tearDown, the result of dest.delete() should be checked
> 3. In class CustomizedResourceHandler, there are several methods where
> exceptions decalarations are unnecessary.
> 4. Class MyMockNM should be renamed to some more meaningful name.
> 5. There are some danling javadoc comments, for example:
> {code:java}
> /*
>* Make sure ResourcePluginManager is initialized during NM start up.
>*/
> {code}
> 6. There are some exceptions unnecessarily declared on test methods but they
> are never thrown, an example:
> testLinuxContainerExecutorWithResourcePluginsEnabled
> 7. Assert.assertTrue(false); expressions should be replaced with Assert.fail()
> 8. A handful of usages of Mockito's spy method. This method is not preferred
> so we should think about replacing it with mocks, somehow.
> The rest can be figured out by whoever takes this jira :)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8148) Update decimal values for queue capacities shown on queue status cli
[
https://issues.apache.org/jira/browse/YARN-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907457#comment-16907457
]
Prabhu Joseph commented on YARN-8148:
-
[~snemeth] Yes you are right. 3.2 and 3.1 should be fine.
> Update decimal values for queue capacities shown on queue status cli
>
>
> Key: YARN-8148
> URL: https://issues.apache.org/jira/browse/YARN-8148
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8148-002.patch, YARN-8148.1.patch
>
>
> Capacities are shown with two decimal values in RM UI as part of YARN-6182.
> The queue status cli are still showing one decimal value.
> {code}
> [root@bigdata3 yarn]# yarn queue -status default
> Queue Information :
> Queue Name : default
> State : RUNNING
> Capacity : 69.9%
> Current Capacity : .0%
> Maximum Capacity : 70.0%
> Default Node Label expression :
> Accessible Node Labels : *
> Preemption : enabled
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9140: - Fix Version/s: 3.1.3 > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907450#comment-16907450 ] Szilard Nemeth commented on YARN-9140: -- Hi Peter Bacsko! +1 on the latest patch for branch-3.1, just committed it. Thanks [~adam.antal] for the review! > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Fix For: 3.3.0, 3.2.1 > > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9140: - Fix Version/s: 3.2.1 3.3.0 > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Fix For: 3.3.0, 3.2.1 > > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9290) Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler
[
https://issues.apache.org/jira/browse/YARN-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907442#comment-16907442
]
Szilard Nemeth commented on YARN-9290:
--
Hi [~Prabhu Joseph]!
As the change is bigger, can you please elaborate in the description about why
you had to change as many files?
I mean, I don't have the full context and it'd make reviewing this more easily
if you'd describe the changes more detailed.
Thanks!
> Invalid SchedulingRequest not rejected in Scheduler
> PlacementConstraintsHandler
>
>
> Key: YARN-9290
> URL: https://issues.apache.org/jira/browse/YARN-9290
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9290-001.patch, YARN-9290-002.patch,
> YARN-9290-003.patch, YARN-9290-004.patch, YARN-9290-005.patch,
> YARN-9290-006.patch
>
>
> SchedulingRequest with Invalid namespace is not rejected in Scheduler
> PlacementConstraintsHandler. RM keeps on trying to allocateOnNode with
> logging the exception. This is rejected in case of placement-processor
> handler.
> {code}
> 2019-02-08 16:51:27,548 WARN
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator:
> Failed to query node cardinality:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.InvalidAllocationTagsQueryException:
> Invalid namespace prefix: notselfi, valid values are:
> all,not-self,app-id,app-tag,self
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TargetApplicationsNamespace.fromString(TargetApplicationsNamespace.java:277)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TargetApplicationsNamespace.parse(TargetApplicationsNamespace.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.AllocationTags.createAllocationTags(AllocationTags.java:93)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfySingleConstraintExpression(PlacementConstraintsUtil.java:78)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfySingleConstraint(PlacementConstraintsUtil.java:240)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:321)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyAndConstraint(PlacementConstraintsUtil.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:324)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.PlacementConstraintsUtil.canSatisfyConstraints(PlacementConstraintsUtil.java:365)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.checkCardinalityAndPending(SingleConstraintAppPlacementAllocator.java:355)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator.precheckNode(SingleConstraintAppPlacementAllocator.java:395)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.precheckNode(AppSchedulingInfo.java:779)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.preCheckForNodeCandidateSet(RegularContainerAllocator.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.allocate(RegularContainerAllocator.java:837)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.assignContainers(RegularContainerAllocator.java:890)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.ContainerAllocator.assignContainers(ContainerAllocator.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.assignContainers(FiCaSchedulerApp.java:977)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:1173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:795)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacit
[jira] [Commented] (YARN-8148) Update decimal values for queue capacities shown on queue status cli
[
https://issues.apache.org/jira/browse/YARN-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907439#comment-16907439
]
Szilard Nemeth commented on YARN-8148:
--
Hi [~Prabhu Joseph]!
As far as I can understand this change, it is trying to rationalize and use the
same decimal format for the web UI and the queue CLI. Am I right about this?
Please also tell me which other branches we need to target with the fix (3.2 /
3.1?) !
Thanks!
> Update decimal values for queue capacities shown on queue status cli
>
>
> Key: YARN-8148
> URL: https://issues.apache.org/jira/browse/YARN-8148
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-8148-002.patch, YARN-8148.1.patch
>
>
> Capacities are shown with two decimal values in RM UI as part of YARN-6182.
> The queue status cli are still showing one decimal value.
> {code}
> [root@bigdata3 yarn]# yarn queue -status default
> Queue Information :
> Queue Name : default
> State : RUNNING
> Capacity : 69.9%
> Current Capacity : .0%
> Maximum Capacity : 70.0%
> Default Node Label expression :
> Accessible Node Labels : *
> Preemption : enabled
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
[
https://issues.apache.org/jira/browse/YARN-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907427#comment-16907427
]
Szilard Nemeth commented on YARN-9488:
--
Hi [~Prabhu Joseph]!
Change looks good. Re-attaching latest patch to have a fresh jenkins result.
> Skip YARNFeatureNotEnabledException from ClientRMService
>
>
> Key: YARN-9488
> URL: https://issues.apache.org/jira/browse/YARN-9488
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9488-001.patch, YARN-9488-002.patch,
> YARN-9488-002.patch
>
>
> RM logs are accumulated with YARNFeatureNotEnabledException when running
> DIstributed Shell jobs while {{ClientRMService#getResourceProfiles}}
> {code}
> 2019-04-16 07:10:47,699 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 8050, call Call#5 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getResourceProfiles
> from 172.26.81.91:41198
> org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
> profile is not enabled, please enable resource profile feature before using
> its functions. (by setting yarn.resourcemanager.resource-profiles.enabled to
> true)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1833)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:670)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:665)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9488) Skip YARNFeatureNotEnabledException from ClientRMService
[
https://issues.apache.org/jira/browse/YARN-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-9488:
-
Attachment: YARN-9488-002.patch
> Skip YARNFeatureNotEnabledException from ClientRMService
>
>
> Key: YARN-9488
> URL: https://issues.apache.org/jira/browse/YARN-9488
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9488-001.patch, YARN-9488-002.patch,
> YARN-9488-002.patch
>
>
> RM logs are accumulated with YARNFeatureNotEnabledException when running
> DIstributed Shell jobs while {{ClientRMService#getResourceProfiles}}
> {code}
> 2019-04-16 07:10:47,699 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 8050, call Call#5 Retry#0
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getResourceProfiles
> from 172.26.81.91:41198
> org.apache.hadoop.yarn.exceptions.YARNFeatureNotEnabledException: Resource
> profile is not enabled, please enable resource profile feature before using
> its functions. (by setting yarn.resourcemanager.resource-profiles.enabled to
> true)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.checkAndThrowExceptionWhenFeatureDisabled(ResourceProfilesManagerImpl.java:191)
> at
> org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceProfilesManagerImpl.getResourceProfiles(ResourceProfilesManagerImpl.java:214)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getResourceProfiles(ClientRMService.java:1833)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getResourceProfiles(ApplicationClientProtocolPBServiceImpl.java:670)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:665)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7291) Better input parsing for resource in allocation file
[ https://issues.apache.org/jira/browse/YARN-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907423#comment-16907423 ] Szilard Nemeth commented on YARN-7291: -- Hi [~zsiegl]! Latest patch looks good, +1. [~wilfreds]: Before I commit this, could you please also take a look? Thanks! > Better input parsing for resource in allocation file > > > Key: YARN-7291 > URL: https://issues.apache.org/jira/browse/YARN-7291 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Zoltan Siegl >Priority: Minor > Labels: newbie > Attachments: YARN-7291.001.patch, YARN-7291.002.patch, > YARN-7291.003.patch, YARN-7291.004.patch, YARN-7291.005.patch, > YARN-7291.005.patch > > > When you set max/min share for queues in fair scheduler allocation file, > "1024 mb, 2 4 vcores" is parsed the same as "1024 mb, 4 vcores" without any > issue, the same to "50% memory, 50% 100%cpu" which is parsed the same as "50% > memory, 100%cpu". That causes confusing. We should fix it. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-9359) Avoid code duplication in Resources for calculation methods
[
https://issues.apache.org/jira/browse/YARN-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WEI-HSIAO-LEE updated YARN-9359:
Comment: was deleted
(was: Hi , [~snemeth] , I didn't find the code duplication like example ,
maybe someone has done.)
> Avoid code duplication in Resources for calculation methods
> ---
>
> Key: YARN-9359
> URL: https://issues.apache.org/jira/browse/YARN-9359
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: WEI-HSIAO-LEE
>Priority: Minor
> Labels: newbie, newbie++
>
> This is a follow-up for YARN-9318, dealing with code duplication issueas, as
> discussed with [~templedf] earlier.
> Resources has many very similar calculation methods like addTo, subtractFrom,
> multiply, etc.
> These are having extractable code as common, the only difference could be the
> calculation they perform on the passed Resource object(s).
> These methods either receive one or two Resource objects and make some
> calculations on these.
> One caveat that needs some attention is that some of them do clone the
> Resource and do the calculation on the cloned resource and return the result
> (leaving the passed Resource alone) and some of them perform the calculation
> on the passed Resource object itself.
> The common code could be extracted like this:
> {code:java}
> private static Resource applyFunctionOnValues(Resource lhs,
> Function valueFunction) {
> int numResources = ResourceUtils.getNumberOfCountableResourceTypes();
> for (int i = 0; i < numResources; i++) {
> try {
> ResourceInformation lhsValue = lhs.getResourceInformation(i);
> Long modifiedValue = valueFunction.apply(lhsValue.getValue());
> lhs.setResourceValue(i, modifiedValue);
> } catch (ResourceNotFoundException ye) {
> LOG.warn("Resource is missing:" + ye.getMessage());
> }
> }
> return lhs;
> }
> {code}
> And an example code could be like this:
> {code:java}
> public static Resource multiplyAndRoundUp(Resource lhs, double by) {
> return applyFunctionOnValues(clone(lhs),
> (value) -> (long) Math.ceil(value * by));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[
https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907421#comment-16907421
]
Hadoop QA commented on YARN-9140:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
42s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 7 unchanged - 5 fixed = 7 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
54s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-9140 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977610/YARN-9140.branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c4f4fce8c230 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / 224643a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24564/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-
[jira] [Assigned] (YARN-9430) Recovering containers does not check available resources on node
[
https://issues.apache.org/jira/browse/YARN-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth reassigned YARN-9430:
Assignee: Szilard Nemeth
> Recovering containers does not check available resources on node
>
>
> Key: YARN-9430
> URL: https://issues.apache.org/jira/browse/YARN-9430
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Critical
>
> I have a testcase that checks if some GPU devices gone offline and recovery
> happens, only the containers that fit into the node's resources will be
> recovered. Unfortunately, this is not the case: RM does not check available
> resources on node during recovery.
> *Detailed explanation:*
> *Testcase:*
> 1. There are 2 nodes running NodeManagers
> 2. nvidia-smi is replaced with a fake bash script that reports 2 GPU devices
> per node, initially. This means 4 GPU devices in the cluster altogether.
> 3. RM / NM recovery is enabled
> 4. The test starts off a sleep job, requesting 4 containers, 1 GPU device
> for each (AM does not request GPUs)
> 5. Before restart, the fake bash script is adjusted to report 1 GPU device
> per node (2 in the cluster) after restart.
> 6. Restart is initiated.
>
> *Expected behavior:*
> After restart, only the AM and 2 normal containers should have been started,
> as there are only 2 GPU devices in the cluster.
>
> *Actual behaviour:*
> AM + 4 containers are allocated, this is all containers started originally
> with step 4.
> App id was: 1553977186701_0001
> *Logs*:
>
> {code:java}
> 2019-03-30 13:22:30,299 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> Processing event for appattempt_1553977186701_0001_01 of type RECOVER
> 2019-03-30 13:22:30,366 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Added Application Attempt appattempt_1553977186701_0001_01 to scheduler
> from user: systest
> 2019-03-30 13:22:30,366 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> appattempt_1553977186701_0001_01 is recovering. Skipping notifying
> ATTEMPT_ADDED
> 2019-03-30 13:22:30,367 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1553977186701_0001_01 State change from NEW to LAUNCHED on
> event = RECOVER
> 2019-03-30 13:22:33,257 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Recovering container [container_e84_1553977186701_0001_01_01,
> CreateTime: 1553977260732, Version: 0, State: RUNNING, Capability:
> , Diagnostics: , ExitStatus: -1000,
> NodeLabelExpression: Priority: 0]
> 2019-03-30 13:22:33,275 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Recovering container [container_e84_1553977186701_0001_01_04,
> CreateTime: 1553977272802, Version: 0, State: RUNNING, Capability:
> , Diagnostics: , ExitStatus: -1000,
> NodeLabelExpression: Priority: 0]
> 2019-03-30 13:22:33,275 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode:
> Assigned container container_e84_1553977186701_0001_01_04 of capacity
> on host
> snemeth-gpu-2.vpc.cloudera.com:8041, which has 2 containers, vCores:2, yarn.io/gpu: 1> used and available after
> allocation
> 2019-03-30 13:22:33,276 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Recovering container [container_e84_1553977186701_0001_01_05,
> CreateTime: 1553977272803, Version: 0, State: RUNNING, Capability:
> , Diagnostics: , ExitStatus: -1000,
> NodeLabelExpression: Priority: 0]
> 2019-03-30 13:22:33,276 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> Processing container_e84_1553977186701_0001_01_05 of type RECOVER
> 2019-03-30 13:22:33,276 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e84_1553977186701_0001_01_05 Container Transitioned from NEW to
> RUNNING
> 2019-03-30 13:22:33,276 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode:
> Assigned container container_e84_1553977186701_0001_01_05 of capacity
> on host
> snemeth-gpu-2.vpc.cloudera.com:8041, which has 3 containers, vCores:3, yarn.io/gpu: 2> used and
> available after allocation
> 2019-03-30 13:22:33,279 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
> Recovering container [container_e84_1553977186701_0001_01_03,
> CreateTime: 1553977272166, Version: 0, State: RUNNING, Capability:
> , Diagnostics: , ExitStatus: -1000,
> NodeLabelExpression: Priority: 0]
> 2019-03-30 13:22:3
[jira] [Assigned] (YARN-9421) Implement SafeMode for ResourceManager by defining a resource threshold
[
https://issues.apache.org/jira/browse/YARN-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth reassigned YARN-9421:
Assignee: Szilard Nemeth
> Implement SafeMode for ResourceManager by defining a resource threshold
> ---
>
> Key: YARN-9421
> URL: https://issues.apache.org/jira/browse/YARN-9421
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: client-log.log, nodemanager.log, resourcemanager.log
>
>
> We have a hypothetical testcase in our test suite that tests Resource Types.
> The test does the following:
> 1. Sets up a resource named "gpu"
> 2. Out of 9 NodeManager nodes, 1 node has 100 of "gpu".
> 3. It executes a sleep job with resoure requests:
> "-Dmapreduce.reduce.resource.gpu=7" and
> "-Dyarn.app.mapreduce.am.resource.gpu=11"
> Sometimes, we encounter situations when the app submission fails with:
> {code:java}
> 2019-02-25 06:09:56,795 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: RM app submission
> failed in validating AM resource request for application
> application_1551103768202_0001
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request! Cannot allocate containers as requested resource is greater
> than maximum allowed allocation. Requested resource type=[gpu], Requested
> resource=, maximum allowed
> allocation=, please note that maximum allowed
> allocation is calculated by scheduler based on maximum resource of registered
> NodeManagers, which might be less than configured maximum
> allocation={code}
> It's clearly visible that the maximum allowed allocation does not have any
> "gpu" resources.
>
> Looking into the logs further, I realized that sometimes the node having the
> "gpu" resources are registered after the app is submitted.
> In a real world situation and even with this very special test exexution, we
> can't be sure which order NMs are registering with RM.
> With the advent of resource types, this issue was more likely surface.
> If we have a cluster with some "rare" resources like GPUs only on some nodes
> out of a 100, we can quickly run into a situation when the NMs with GPUs are
> registering later than the normal nodes. While the critical NMs are still
> registering, we will most likely experience the same
> InvalidResourceRequestException if we submit jobs requesting GPUs.
> There is a naive solution to this:
> 1. Give some time for RM to wait for NMs to be able to register themselves
> and put submitted applications on hold. This could work in some situations
> but it's not the most flexible solution as different clusters can have
> different requirements. Of course, we can make this more flexible by making
> the timeout value configurable.
> *A more flexible alternative would be:*
> 2. We define a threshold of Resource capability: While we haven't reached
> this threshold, we put submitted jobs on hold. Once we reached the threshold,
> we enable jobs to pass through.
> This is very similar to an already existing concept, the SafeMode in HDFS
> ([https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Safemode]).
> Back to my GPU example above, the threshold could be: 8 vcores, 16GB, 3
> GPUs.
> Defining a threshold like this, we can ensure most of the submitted jobs
> won't be lost, just "parked" until NMs are registered.
> The final solution could be the Resource threshold, or the combination of the
> threshold and timeout value. I'm open for any other suggestion as well.
> *Last but not least, a very easy way to reproduce the issue on a 3 node
> cluster:*
> 1. Configure a resource type, named 'testres'.
> 2. Node1 runs RM, Node 2/3 runs NMs
> 3. Node2 has 1 testres
> 4. Node3 has 0 testres
> 5. Stop all nodes
> 6. Start RM on Node1
> 7. Start NM on Node3 (the one without the resource)
> 8. Start a pi job, request 1 testres for the AM
> Here's the command to start the job:
> {code:java}
> MY_HADOOP_VERSION=3.3.0-SNAPSHOT;pushd /opt/hadoop;bin/yarn jar
> "./share/hadoop/mapreduce/hadoop-mapreduce-examples-$MY_HADOOP_VERSION.jar"
> pi -Dyarn.app.mapreduce.am.resource.testres=1 1 1000;popd{code}
>
> *Configurations*:
> node1: yarn-site.xml of ResourceManager:
> {code:java}
>
> yarn.resource-types
> testres
> {code}
> node2: yarn-site.xml of NodeManager:
> {code:java}
>
> yarn.resource-types
> testres
>
>
> yarn.nodemanager.resource-type.testres
> 1
> {code}
> node3: yarn-site.xml of NodeManager:
> {code:java}
>
> yarn.resource-types
> testres
> {code}
> Please see full process logs from RM, NM, YARN-client attached.
--
This message was sent by Atlassian JIRA
(v7.6.1
[jira] [Commented] (YARN-2599) Standby RM should also expose some jmx and metrics
[ https://issues.apache.org/jira/browse/YARN-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907389#comment-16907389 ] Sunil Govindan commented on YARN-2599: -- Thanks [~bibinchundatt] I do not think its a compatibility issue. Fundamentally current behaviour is broken. In one of old Jira prior to HA, I could see that each process was having its own jmx. So redirect on jmx was not correct. I am pulling this only to trunk and mark a note about same. I could not get the metrics servlet. I ll dig in a bit. This seems very old, and I am not sure whether its already moved out or not, > Standby RM should also expose some jmx and metrics > -- > > Key: YARN-2599 > URL: https://issues.apache.org/jira/browse/YARN-2599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1, 2.7.3, 3.0.0-alpha1 >Reporter: Karthik Kambatla >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-2599.patch > > > YARN-1898 redirects jmx and metrics to the Active. As discussed there, we > need to separate out metrics displayed so the Standby RM can also be > monitored. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907380#comment-16907380 ] Eric Yang commented on YARN-9683: - [~adam.antal] thank you for the commit. > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9676) Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes
[
https://issues.apache.org/jira/browse/YARN-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907375#comment-16907375
]
Hudson commented on YARN-9676:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17122 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17122/])
YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl…
(954799+szilard-nemeth: rev c89bdfacc8715fa6d72acd85437ab8cd257c8aad)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
> Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected
> classes
>
>
> Key: YARN-9676
> URL: https://issues.apache.org/jira/browse/YARN-9676
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> During the development of the last items of YARN-6875, it was typically
> difficult to extract information about the internal state of some log
> aggregation related classes (e.g. {{AppLogAggregatiorImpl}} and
> {{LogAggregationFileController}}).
> On my fork I added a few more messages to those classes like:
> - displaying the number of log aggregation cycles
> - displaying the names of the files currently considered for log aggregation
> by containers
> - immediately displaying any exception caught (and sent to the RM in the
> diagnostic messages) during the log aggregation process.
> Those messages were quite useful for debugging if any issue occurs, but
> otherwise it flooded the NM log file with these messages that are usually not
> needed. I suggest to add (some of) these messages in DEBUG or TRACE level.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9717) Add more logging to container-executor about issues with directory creation or permissions
[
https://issues.apache.org/jira/browse/YARN-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth reassigned YARN-9717:
Assignee: Peter Bacsko (was: Szilard Nemeth)
> Add more logging to container-executor about issues with directory creation
> or permissions
> --
>
> Key: YARN-9717
> URL: https://issues.apache.org/jira/browse/YARN-9717
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Peter Bacsko
>Priority: Major
>
> During some downstream testing we bumped into some problems with the
> container executor where an extra logging would be quite helpful when local
> files and directories could not be created (container-executor.c:1810).
> The most important log line could be the following:
> There's a function called create_container_directories in
> container-executor.c.
> We should place a log line like this:
> Before we're calling:
> We have:
> {code:java}
> if (mkdirs(container_dir, perms) == 0) {
> result = 0;
> }
> {code}
> We could add an else statement and add the following log, if creating the
> directory was not successful:
> {code:java}
> fprintf(LOGFILE, "Failed to create directory: %s, user: %s", container_dir,
> user);
> {code}
> This way, CE at least prints the directory itself if we have any permission
> issue while trying to create a subdirectory or file under it.
> If we want to be very precise, some logging into the mkdirs function could
> also be added as well.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9676) Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes
[
https://issues.apache.org/jira/browse/YARN-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907365#comment-16907365
]
Szilard Nemeth commented on YARN-9676:
--
Hi [~adam.antal]!
As discussed on github, I merged your PR to trunk.
Backport to branch-3.2 went fine so I also pushed to that branch.
However, cherry-picking the commit to branch-3.1 showed some conflicts.
Could you please update a patch targeting branch-3.1?
Thanks!
> Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected
> classes
>
>
> Key: YARN-9676
> URL: https://issues.apache.org/jira/browse/YARN-9676
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> During the development of the last items of YARN-6875, it was typically
> difficult to extract information about the internal state of some log
> aggregation related classes (e.g. {{AppLogAggregatiorImpl}} and
> {{LogAggregationFileController}}).
> On my fork I added a few more messages to those classes like:
> - displaying the number of log aggregation cycles
> - displaying the names of the files currently considered for log aggregation
> by containers
> - immediately displaying any exception caught (and sent to the RM in the
> diagnostic messages) during the log aggregation process.
> Those messages were quite useful for debugging if any issue occurs, but
> otherwise it flooded the NM log file with these messages that are usually not
> needed. I suggest to add (some of) these messages in DEBUG or TRACE level.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9098) Separate mtab file reader code and cgroups file system hierarchy parser code from CGroupsHandlerImpl and ResourceHandlerModule
[
https://issues.apache.org/jira/browse/YARN-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907364#comment-16907364
]
Hadoop QA commented on YARN-9098:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-9098 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9098 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977615/YARN-9098.008.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24565/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Separate mtab file reader code and cgroups file system hierarchy parser code
> from CGroupsHandlerImpl and ResourceHandlerModule
> --
>
> Key: YARN-9098
> URL: https://issues.apache.org/jira/browse/YARN-9098
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-9098.002.patch, YARN-9098.003.patch,
> YARN-9098.004.patch, YARN-9098.005.patch, YARN-9098.006.patch,
> YARN-9098.007.patch, YARN-9098.008.patch, YARN-9098.008.patch
>
>
> Separate mtab file reader code and cgroups file system hierarchy parser code
> from CGroupsHandlerImpl and ResourceHandlerModule
> CGroupsHandlerImpl has a method parseMtab that parses an mtab file and stores
> cgroups data.
> CGroupsLCEResourcesHandler also has a method with the same name, with
> identical code.
> The parser code should be extracted from these places and be added in a new
> class as this is a separate responsibility.
> As the output of the file parser is a Map>, it's better
> to encapsulate it in a domain object, named 'CGroupsMountConfig' for instance.
> ResourceHandlerModule has a method named parseConfiguredCGroupPath, that is
> responsible for producing the same results (Map>) to
> store cgroups data, it does not operate on mtab file, but looking at the
> filesystem for cgroup settings. As the output is the same, CGroupsMountConfig
> should be used here, too.
> Again, this could should not be part of ResourceHandlerModule as it is a
> different responsibility.
> One more thing which is strongly related to the methods above is
> CGroupsHandlerImpl.initializeFromMountConfig: This method processes the
> result of a parsed mtab file or a parsed cgroups filesystem data and stores
> file system paths for all available controllers. This method invokes
> findControllerPathInMountConfig, which is a duplicated in CGroupsHandlerImpl
> and CGroupsLCEResourcesHandler, so it should be moved to a single place. To
> store filesystem path and controller mappings, a new domain object could be
> introduced.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9676) Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes
[
https://issues.apache.org/jira/browse/YARN-9676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-9676:
-
Fix Version/s: 3.2.1
3.3.0
> Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected
> classes
>
>
> Key: YARN-9676
> URL: https://issues.apache.org/jira/browse/YARN-9676
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
>
> During the development of the last items of YARN-6875, it was typically
> difficult to extract information about the internal state of some log
> aggregation related classes (e.g. {{AppLogAggregatiorImpl}} and
> {{LogAggregationFileController}}).
> On my fork I added a few more messages to those classes like:
> - displaying the number of log aggregation cycles
> - displaying the names of the files currently considered for log aggregation
> by containers
> - immediately displaying any exception caught (and sent to the RM in the
> diagnostic messages) during the log aggregation process.
> Those messages were quite useful for debugging if any issue occurs, but
> otherwise it flooded the NM log file with these messages that are usually not
> needed. I suggest to add (some of) these messages in DEBUG or TRACE level.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9098) Separate mtab file reader code and cgroups file system hierarchy parser code from CGroupsHandlerImpl and ResourceHandlerModule
[ https://issues.apache.org/jira/browse/YARN-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9098: - Attachment: YARN-9098.008.patch > Separate mtab file reader code and cgroups file system hierarchy parser code > from CGroupsHandlerImpl and ResourceHandlerModule > -- > > Key: YARN-9098 > URL: https://issues.apache.org/jira/browse/YARN-9098 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-9098.002.patch, YARN-9098.003.patch, > YARN-9098.004.patch, YARN-9098.005.patch, YARN-9098.006.patch, > YARN-9098.007.patch, YARN-9098.008.patch, YARN-9098.008.patch > > > Separate mtab file reader code and cgroups file system hierarchy parser code > from CGroupsHandlerImpl and ResourceHandlerModule > CGroupsHandlerImpl has a method parseMtab that parses an mtab file and stores > cgroups data. > CGroupsLCEResourcesHandler also has a method with the same name, with > identical code. > The parser code should be extracted from these places and be added in a new > class as this is a separate responsibility. > As the output of the file parser is a Map>, it's better > to encapsulate it in a domain object, named 'CGroupsMountConfig' for instance. > ResourceHandlerModule has a method named parseConfiguredCGroupPath, that is > responsible for producing the same results (Map>) to > store cgroups data, it does not operate on mtab file, but looking at the > filesystem for cgroup settings. As the output is the same, CGroupsMountConfig > should be used here, too. > Again, this could should not be part of ResourceHandlerModule as it is a > different responsibility. > One more thing which is strongly related to the methods above is > CGroupsHandlerImpl.initializeFromMountConfig: This method processes the > result of a parsed mtab file or a parsed cgroups filesystem data and stores > file system paths for all available controllers. This method invokes > findControllerPathInMountConfig, which is a duplicated in CGroupsHandlerImpl > and CGroupsLCEResourcesHandler, so it should be moved to a single place. To > store filesystem path and controller mappings, a new domain object could be > introduced. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9133) Make tests more easy to comprehend in TestGpuResourceHandler
[ https://issues.apache.org/jira/browse/YARN-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907355#comment-16907355 ] Hudson commented on YARN-9133: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17121 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17121/]) YARN-9133. Make tests more easy to comprehend in TestGpuResourceHandler. (snemeth: rev 3e0410449fb36da098c897e874a18258253eac81) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandlerImpl.java > Make tests more easy to comprehend in TestGpuResourceHandler > > > Key: YARN-9133 > URL: https://issues.apache.org/jira/browse/YARN-9133 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0, 3.2.1 > > Attachments: YARN-9133.001.patch, YARN-9133.001.patch, > YARN-9133.002.patch, YARN-9133.003.patch, YARN-9133.004.patch, > YARN-9133.005.patch, YARN-9133.006.patch, YARN-9133.006.patch, > YARN-9133.007.patch, YARN-9133.branch-3.2.001.patch > > > Tests are not quite easy to read: > - Some more helper methods would improve readability. > - Eliminating the boolean flag that controls if docker is used would also > improve readability and clarity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9133) Make tests more easy to comprehend in TestGpuResourceHandler
[ https://issues.apache.org/jira/browse/YARN-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907350#comment-16907350 ] Szilard Nemeth edited comment on YARN-9133 at 8/14/19 3:18 PM: --- Hi [~pbacsko]! +1 on latest patch for trunk and branch-3.2, committed both of them! As you mentioned, as branch-3.1 had way too many conflicts, we can skip backport to that version. Thanks for your contribution! was (Author: snemeth): Hi [~pbacsko]! +1 on latest patch for trunk and branch-3.2, committed both of them! Thanks for your contribution! > Make tests more easy to comprehend in TestGpuResourceHandler > > > Key: YARN-9133 > URL: https://issues.apache.org/jira/browse/YARN-9133 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0, 3.2.1 > > Attachments: YARN-9133.001.patch, YARN-9133.001.patch, > YARN-9133.002.patch, YARN-9133.003.patch, YARN-9133.004.patch, > YARN-9133.005.patch, YARN-9133.006.patch, YARN-9133.006.patch, > YARN-9133.007.patch, YARN-9133.branch-3.2.001.patch > > > Tests are not quite easy to read: > - Some more helper methods would improve readability. > - Eliminating the boolean flag that controls if docker is used would also > improve readability and clarity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9133) Make tests more easy to comprehend in TestGpuResourceHandler
[ https://issues.apache.org/jira/browse/YARN-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907350#comment-16907350 ] Szilard Nemeth commented on YARN-9133: -- Hi [~pbacsko]! +1 on latest patch for trunk and branch-3.2, committed both of them! Thanks for your contribution! > Make tests more easy to comprehend in TestGpuResourceHandler > > > Key: YARN-9133 > URL: https://issues.apache.org/jira/browse/YARN-9133 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9133.001.patch, YARN-9133.001.patch, > YARN-9133.002.patch, YARN-9133.003.patch, YARN-9133.004.patch, > YARN-9133.005.patch, YARN-9133.006.patch, YARN-9133.006.patch, > YARN-9133.007.patch, YARN-9133.branch-3.2.001.patch > > > Tests are not quite easy to read: > - Some more helper methods would improve readability. > - Eliminating the boolean flag that controls if docker is used would also > improve readability and clarity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907340#comment-16907340 ] Hudson commented on YARN-9140: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17120 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17120/]) YARN-9140. Code cleanup in ResourcePluginManager.initialize and in (snemeth: rev e5e609384f68cc45b0c2bfbde0a49426c90017d3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/ResourcePluginManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/TestResourcePluginManager.java > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907328#comment-16907328 ] Szilard Nemeth edited comment on YARN-9140 at 8/14/19 3:02 PM: --- Hi [~pbacsko]! +1 on the latest patch for trunk and branch-3.2, committing them shortly. was (Author: snemeth): Hi [~pbacsko]! +1 on the latest patch for trunk, committing it. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907328#comment-16907328 ] Szilard Nemeth edited comment on YARN-9140 at 8/14/19 3:02 PM: --- Hi [~pbacsko]! +1 on the latest patch for trunk and branch-3.2, committing them shortly. As discussed offline, waiting for branch-3.1 patch, if applicable. was (Author: snemeth): Hi [~pbacsko]! +1 on the latest patch for trunk and branch-3.2, committing them shortly. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9140: --- Attachment: YARN-9140.branch-3.1.001.patch > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.1.001.patch, > YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907328#comment-16907328 ] Szilard Nemeth commented on YARN-9140: -- Hi [~pbacsko]! +1 on the latest patch for trunk, committing it. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9134) No test coverage for redefining FPGA / GPU resource types in TestResourceUtils
[ https://issues.apache.org/jira/browse/YARN-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907319#comment-16907319 ] Szilard Nemeth edited comment on YARN-9134 at 8/14/19 2:48 PM: --- Thanks [~pbacsko] for the 3.2 patch, pushed it! I agree with you: if it was so hard to backport the changes to 3.1 then we can ignore it and skip the backport. Resolved the jira. was (Author: snemeth): Thanks [~pbacsko] for the 3.2 patch, pushed it! > No test coverage for redefining FPGA / GPU resource types in TestResourceUtils > -- > > Key: YARN-9134 > URL: https://issues.apache.org/jira/browse/YARN-9134 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.3.0, 3.2.1 > > Attachments: YARN-9134.001.patch, YARN-9134.002.patch, > YARN-9134.003.patch, YARN-9134.004.patch, YARN-9134.branch-3.2.001.patch > > > The patch also includes some trivial code cleanup. > Also, setupResourceTypes has been deprecated as it is dangerous to use, see > the javadoc for details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9134) No test coverage for redefining FPGA / GPU resource types in TestResourceUtils
[ https://issues.apache.org/jira/browse/YARN-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907319#comment-16907319 ] Szilard Nemeth commented on YARN-9134: -- Thanks [~pbacsko] for the 3.2 patch, pushed it! > No test coverage for redefining FPGA / GPU resource types in TestResourceUtils > -- > > Key: YARN-9134 > URL: https://issues.apache.org/jira/browse/YARN-9134 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9134.001.patch, YARN-9134.002.patch, > YARN-9134.003.patch, YARN-9134.004.patch, YARN-9134.branch-3.2.001.patch > > > The patch also includes some trivial code cleanup. > Also, setupResourceTypes has been deprecated as it is dangerous to use, see > the javadoc for details. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8586) Extract log aggregation related fields and methods from RMAppImpl
[
https://issues.apache.org/jira/browse/YARN-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907305#comment-16907305
]
Hadoop QA commented on YARN-8586:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
34s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 107 unchanged - 10 fixed = 108 total (was 117)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 71m
32s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
27s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:63396be |
| JIRA Issue | YARN-8586 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977600/YARN-8586.branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux cd59b86594d9 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / f5661b6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/24563/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/2
[jira] [Commented] (YARN-9748) Allow capacity-scheduler configuration on HDFS
[ https://issues.apache.org/jira/browse/YARN-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907239#comment-16907239 ] Prabhu Joseph commented on YARN-9748: - [~cane] If you have not started working on this, shall i assign it to me. > Allow capacity-scheduler configuration on HDFS > -- > > Key: YARN-9748 > URL: https://issues.apache.org/jira/browse/YARN-9748 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907224#comment-16907224 ] Peter Bacsko commented on YARN-9140: ASF license warning can be ignored. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8586) Extract log aggregation related fields and methods from RMAppImpl
[ https://issues.apache.org/jira/browse/YARN-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-8586: --- Attachment: YARN-8586.branch-3.2.001.patch > Extract log aggregation related fields and methods from RMAppImpl > - > > Key: YARN-8586 > URL: https://issues.apache.org/jira/browse/YARN-8586 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-8586.001.patch, YARN-8586.002.patch, > YARN-8586.002.patch, YARN-8586.003.patch, YARN-8586.004.patch, > YARN-8586.branch-3.2.001.patch > > > Given that RMAppImpl is already above 2000 lines and it is very complex, as a > very simple > and straightforward step, all Log aggregation related fields and methods > could be extracted to a new class. > The clients of RMAppImpl may access the same methods and RMAppImpl would > delegate all those calls to the newly introduced class. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[
https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907189#comment-16907189
]
Hadoop QA commented on YARN-9140:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
38s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 6 unchanged - 5 fixed = 6 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
44s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | YARN-9140 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977578/YARN-9140.branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5b884390b253 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / f5661b6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24562/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-YARN-Build/24562/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 317 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yar
[jira] [Created] (YARN-9748) Allow capacity-scheduler configuration on HDFS
zhoukang created YARN-9748: -- Summary: Allow capacity-scheduler configuration on HDFS Key: YARN-9748 URL: https://issues.apache.org/jira/browse/YARN-9748 Project: Hadoop YARN Issue Type: Improvement Reporter: zhoukang -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9748) Allow capacity-scheduler configuration on HDFS
[ https://issues.apache.org/jira/browse/YARN-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated YARN-9748: --- Affects Version/s: 3.1.2 Component/s: capacityscheduler capacity scheduler > Allow capacity-scheduler configuration on HDFS > -- > > Key: YARN-9748 > URL: https://issues.apache.org/jira/browse/YARN-9748 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.2 >Reporter: zhoukang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2599) Standby RM should also expose some jmx and metrics
[ https://issues.apache.org/jira/browse/YARN-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907179#comment-16907179 ] Bibin A Chundatt edited comment on YARN-2599 at 8/14/19 11:15 AM: -- [~sunilg] # Change could cause incompatability w.r.t monitoring systems. Redirect need to be handled explicitly. Should we make this configurable ?? # Could you point to HTTPServlet handling */metrics* was (Author: bibinchundatt): [~sunilg] # Change could incompatability w.r.t monitoring systems. Redirect need to be handled explicitly. Should we make this configurable ?? # Could you point to HTTPServlet handling */metrics* > Standby RM should also expose some jmx and metrics > -- > > Key: YARN-2599 > URL: https://issues.apache.org/jira/browse/YARN-2599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1, 2.7.3, 3.0.0-alpha1 >Reporter: Karthik Kambatla >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-2599.patch > > > YARN-1898 redirects jmx and metrics to the Active. As discussed there, we > need to separate out metrics displayed so the Standby RM can also be > monitored. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2599) Standby RM should also expose some jmx and metrics
[ https://issues.apache.org/jira/browse/YARN-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907179#comment-16907179 ] Bibin A Chundatt commented on YARN-2599: [~sunilg] # Change could incompatability w.r.t monitoring systems. Redirect need to be handled explicitly. Should we make this configurable ?? # Could you point to HTTPServlet handling */metrics* > Standby RM should also expose some jmx and metrics > -- > > Key: YARN-2599 > URL: https://issues.apache.org/jira/browse/YARN-2599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1, 2.7.3, 3.0.0-alpha1 >Reporter: Karthik Kambatla >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-2599.patch > > > YARN-1898 redirects jmx and metrics to the Active. As discussed there, we > need to separate out metrics displayed so the Standby RM can also be > monitored. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9683) Remove reapDockerContainerNoPid left behind by YARN-9074
[ https://issues.apache.org/jira/browse/YARN-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907176#comment-16907176 ] Adam Antal commented on YARN-9683: -- Thanks for the commit [~eyang]. Resolved the issue. > Remove reapDockerContainerNoPid left behind by YARN-9074 > > > Key: YARN-9683 > URL: https://issues.apache.org/jira/browse/YARN-9683 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: kevin su >Priority: Trivial > Labels: newbie > Fix For: 3.3.0 > > > YARN-9074 has touched the ContainerCleanup.java but created a separate > function instead of using reapDockerContainerNoPid in ContainerCleanup.java. > Having no usages, that private function can be safely removed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5857) TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907158#comment-16907158
]
Hadoop QA commented on YARN-5857:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 27s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-5857 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977568/YARN-5857-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d89709aeaefe 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0e4b757 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24561/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24561/testReport/ |
| Max. process+thread count | 448 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/h
[jira] [Commented] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907137#comment-16907137 ] Peter Bacsko commented on YARN-9140: I managed to have this patch apply on branch-3.2. Let's wait for the build. > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9140) Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager
[ https://issues.apache.org/jira/browse/YARN-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9140: --- Attachment: YARN-9140.branch-3.2.001.patch > Code cleanup in ResourcePluginManager.initialize and in > TestResourcePluginManager > - > > Key: YARN-9140 > URL: https://issues.apache.org/jira/browse/YARN-9140 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Trivial > Attachments: YARN-9140.001.patch, YARN-9140.002.patch, > YARN-9140.003.patch, YARN-9140.004.patch, YARN-9140.005.patch, > YARN-9140.006.patch, YARN-9140.branch-3.2.001.patch > > > Some code cleanup is needed in ResourcePluginManager#initialize: > * There's a big code block that initializes resource plugins, this should be > extracted to a separate method. > * Exception handling could be simplified. > TestResourcePluginManager minor cleanup: > * Not thrown exceptions could be deleted from method signatures > * verify(obj, times(1)).() calls: times(1) parameter could be > deleted as it is the default if verify(obj) is invoked without the times > parameter. > * Some code exceeds the 80 character column limit. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2599) Standby RM should also expose some jmx and metrics
[ https://issues.apache.org/jira/browse/YARN-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907100#comment-16907100 ] Sunil Govindan commented on YARN-2599: -- Kicking this again. Having per process jmx is always better to debug any issues. If we redirect, its tough to know what has happened in the standby. [~rohithsharma] 's patch seems good to me. I will rebase this. cc [~leftnoteasy] [~cheersyang] [~vinodkv] > Standby RM should also expose some jmx and metrics > -- > > Key: YARN-2599 > URL: https://issues.apache.org/jira/browse/YARN-2599 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1, 2.7.3, 3.0.0-alpha1 >Reporter: Karthik Kambatla >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-2599.patch > > > YARN-1898 redirects jmx and metrics to the Active. As discussed there, we > need to separate out metrics displayed so the Standby RM can also be > monitored. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9133) Make tests more easy to comprehend in TestGpuResourceHandler
[ https://issues.apache.org/jira/browse/YARN-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907074#comment-16907074 ] Peter Bacsko commented on YARN-9133: ASF license issue can be ignored. > Make tests more easy to comprehend in TestGpuResourceHandler > > > Key: YARN-9133 > URL: https://issues.apache.org/jira/browse/YARN-9133 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9133.001.patch, YARN-9133.001.patch, > YARN-9133.002.patch, YARN-9133.003.patch, YARN-9133.004.patch, > YARN-9133.005.patch, YARN-9133.006.patch, YARN-9133.006.patch, > YARN-9133.007.patch, YARN-9133.branch-3.2.001.patch > > > Tests are not quite easy to read: > - Some more helper methods would improve readability. > - Eliminating the boolean flag that controls if docker is used would also > improve readability and clarity. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5857) TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T updated YARN-5857:
Attachment: YARN-5857-001.patch
> TestLogAggregationService.testFixedSizeThreadPool fails intermittently on
> trunk
> ---
>
> Key: YARN-5857
> URL: https://issues.apache.org/jira/browse/YARN-5857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Minor
> Attachments: YARN-5857-001.patch, testFixedSizeThreadPool failure
> reproduction
>
>
> {noformat}
> testFixedSizeThreadPool(org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService)
> Time elapsed: 0.11 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testFixedSizeThreadPool(TestLogAggregationService.java:1139)
> {noformat}
> Refer to https://builds.apache.org/job/PreCommit-YARN-Build/13829/testReport/
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5857) TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907070#comment-16907070
]
Bilwa S T commented on YARN-5857:
-
Hi [~ajithshetty] i would like to work on this
> TestLogAggregationService.testFixedSizeThreadPool fails intermittently on
> trunk
> ---
>
> Key: YARN-5857
> URL: https://issues.apache.org/jira/browse/YARN-5857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Minor
> Attachments: testFixedSizeThreadPool failure reproduction
>
>
> {noformat}
> testFixedSizeThreadPool(org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService)
> Time elapsed: 0.11 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testFixedSizeThreadPool(TestLogAggregationService.java:1139)
> {noformat}
> Refer to https://builds.apache.org/job/PreCommit-YARN-Build/13829/testReport/
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5857) TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk
[
https://issues.apache.org/jira/browse/YARN-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T reassigned YARN-5857:
---
Assignee: Bilwa S T (was: Ajith S)
> TestLogAggregationService.testFixedSizeThreadPool fails intermittently on
> trunk
> ---
>
> Key: YARN-5857
> URL: https://issues.apache.org/jira/browse/YARN-5857
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Minor
> Attachments: testFixedSizeThreadPool failure reproduction
>
>
> {noformat}
> testFixedSizeThreadPool(org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService)
> Time elapsed: 0.11 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService.testFixedSizeThreadPool(TestLogAggregationService.java:1139)
> {noformat}
> Refer to https://builds.apache.org/job/PreCommit-YARN-Build/13829/testReport/
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9747) Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs
[
https://issues.apache.org/jira/browse/YARN-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907020#comment-16907020
]
Prabhu Joseph commented on YARN-9747:
-
Thanks [~bibinchundatt].
> Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs
> ---
>
> Key: YARN-9747
> URL: https://issues.apache.org/jira/browse/YARN-9747
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9747-001.patch
>
>
> EntityGroupFSTimelineStore#cleanLogs creates additional Namenode RPC call.
> {code}
> cleanLogs:
> while (iter.hasNext()) {
> FileStatus stat = iter.next();
> Path clusterTimeStampPath = stat.getPath();
> if (isValidClusterTimeStampDir(clusterTimeStampPath)) {
> MutableBoolean appLogDirPresent = new MutableBoolean(false);
> { fs.getFileStatus(clusterTimeStampPath);}} in isValidClusterTimeStampDir*
> creates additional Namenode RPC call.
> {code}
> cc [~bibinchundatt]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9747) Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs
[
https://issues.apache.org/jira/browse/YARN-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907017#comment-16907017
]
Hudson commented on YARN-9747:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17116 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17116/])
YARN-9747. Reduce additional namenode call by (bibinchundatt: rev
89a53c7eb41098d85d238c39a62d3f98e5f51585)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
> Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs
> ---
>
> Key: YARN-9747
> URL: https://issues.apache.org/jira/browse/YARN-9747
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9747-001.patch
>
>
> EntityGroupFSTimelineStore#cleanLogs creates additional Namenode RPC call.
> {code}
> cleanLogs:
> while (iter.hasNext()) {
> FileStatus stat = iter.next();
> Path clusterTimeStampPath = stat.getPath();
> if (isValidClusterTimeStampDir(clusterTimeStampPath)) {
> MutableBoolean appLogDirPresent = new MutableBoolean(false);
> { fs.getFileStatus(clusterTimeStampPath);}} in isValidClusterTimeStampDir*
> creates additional Namenode RPC call.
> {code}
> cc [~bibinchundatt]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9747) Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs
[
https://issues.apache.org/jira/browse/YARN-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906967#comment-16906967
]
Hadoop QA commented on YARN-9747:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
47s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9747 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977556/YARN-9747-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 731a25accc60 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 48dc441 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24560/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24560/console |
| P
