[jira] [Created] (YARN-10781) The Thread of the NM aggregate log is exhausted and no other Application can aggregate the log
Xiping Zhang created YARN-10781: --- Summary: The Thread of the NM aggregate log is exhausted and no other Application can aggregate the log Key: YARN-10781 URL: https://issues.apache.org/jira/browse/YARN-10781 Project: Hadoop YARN Issue Type: Bug Components: yarn Affects Versions: 3.3.0, 2.9.2 Reporter: Xiping Zhang We observed more than 100 applications running on one NM.Most of these applications are SparkStreaming tasks, but these applications do not have running Containers.When the offline application running on it finishes, the log cannot be reported to HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9279) Remove the old hamlet package
[ https://issues.apache.org/jira/browse/YARN-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated YARN-9279: --- Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) > Remove the old hamlet package > - > > Key: YARN-9279 > URL: https://issues.apache.org/jira/browse/YARN-9279 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-9279.01.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > The old hamlet package was deprecated in HADOOP-11875. Let's remove this to > improve the maintenability. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10123) Error message around yarn app -stop/start can be improved to highlight that an implementation at framework level is needed for the stop/start functionality to work
[
https://issues.apache.org/jira/browse/YARN-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348943#comment-17348943
]
Hadoop QA commented on YARN-10123:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m
50s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} branch-3.3 Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 11m
48s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
8s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
1s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 15s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m
35s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
36s{color} | {color:blue}{color} | {color:blue}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
33s{color} | {color:blue}{color} | {color:blue}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs
{color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
53s{color} | {color:green}{color} | {color:green}
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348941#comment-17348941
]
Qi Zhu commented on YARN-10779:
---
Thanks [~pbacsko] for this work.
The patch LGTM, just to fix the only one checkstyle.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-001.patch, YARN-10779-002.patch,
> YARN-10779-003.patch, YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10123) Error message around yarn app -stop/start can be improved to highlight that an implementation at framework level is needed for the stop/start functionality to work
[
https://issues.apache.org/jira/browse/YARN-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348940#comment-17348940
]
Hadoop QA commented on YARN-10123:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
30s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} branch-3.3 Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m
0s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
43s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
55s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 25s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} branch-3.3 passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 24m
37s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
26s{color} | {color:blue}{color} | {color:blue}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 4s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs
{color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
37s{color} | {color:green}{color} | {color:green}
[jira] [Updated] (YARN-10123) Error message around yarn app -stop/start can be improved to highlight that an implementation at framework level is needed for the stop/start functionality to work
[
https://issues.apache.org/jira/browse/YARN-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10123:
---
Attachment: YARN-10123.branch-3.3.001.patch
> Error message around yarn app -stop/start can be improved to highlight that
> an implementation at framework level is needed for the stop/start
> functionality to work
> ---
>
> Key: YARN-10123
> URL: https://issues.apache.org/jira/browse/YARN-10123
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, documentation
>Affects Versions: 3.2.1
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Attachments: YARN-10123.001.patch, YARN-10123.branch-3.2.001.patch,
> YARN-10123.branch-3.3.001.patch
>
>
> A "stop" on a YARN application fails with the below error:
> {code}
> # yarn app -stop application_1581294743321_0002 -appTypes SPARK
> 20/02/10 06:24:27 INFO client.RMProxy: Connecting to ResourceManager at
> c3224-node2.squadron.support.hortonworks.com/172.25.34.128:8050
> 20/02/10 06:24:27 INFO client.AHSProxy: Connecting to Application History
> server at c3224-node2.squadron.support.hortonworks.com/172.25.34.128:10200
> Exception in thread "main" java.lang.IllegalArgumentException: App admin
> client class name not specified for type SPARK
> at
> org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClient(AppAdminClient.java:76)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:579)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:123)
> {code}
> From
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java#L76,
> it seems that this is because user does not have the setting:
> {code}
> yarn.application.admin.client.class.SPARK
> {code}
> set up in their client configuration.
> However, even if this setting is present, we still need to have an
> implementation available for the application type. From my internal
> discussions - Jobs don't have a notion of stop / resume functionality at YARN
> level. If some apps like Spark need it, it has to be implemented at those
> framework's level.
> Therefore, the above error message is a bit misleading in that, even if
> "yarn.application.admin.client.class.SPARK" is supplied (or for that matter -
> yarn.application.admin.client.class.MAPREDUCE), if there is no implementation
> actually available underneath to handle the stop/start functionality then, we
> will fail again, albeit with a different error here:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java#L85.
> As such, maybe this error message can be potentially improved to say
> something like:
> {code}
> Exception in thread "main" java.lang.IllegalArgumentException: App admin
> client class name not specified for type SPARK. Please ensure the App admin
> client class actually exists within SPARK to handle this functionality.
> {code}
> or something similar.
> Further, documentation around "-stop" and "-start" options will need to be
> improved here ->
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application_or_app
> as it does not mention anything about having an implementation at the
> framework level for the YARN stop/start command to succeed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10123) Error message around yarn app -stop/start can be improved to highlight that an implementation at framework level is needed for the stop/start functionality to work
[
https://issues.apache.org/jira/browse/YARN-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10123:
---
Attachment: YARN-10123.branch-3.2.001.patch
> Error message around yarn app -stop/start can be improved to highlight that
> an implementation at framework level is needed for the stop/start
> functionality to work
> ---
>
> Key: YARN-10123
> URL: https://issues.apache.org/jira/browse/YARN-10123
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, documentation
>Affects Versions: 3.2.1
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Attachments: YARN-10123.001.patch, YARN-10123.branch-3.2.001.patch
>
>
> A "stop" on a YARN application fails with the below error:
> {code}
> # yarn app -stop application_1581294743321_0002 -appTypes SPARK
> 20/02/10 06:24:27 INFO client.RMProxy: Connecting to ResourceManager at
> c3224-node2.squadron.support.hortonworks.com/172.25.34.128:8050
> 20/02/10 06:24:27 INFO client.AHSProxy: Connecting to Application History
> server at c3224-node2.squadron.support.hortonworks.com/172.25.34.128:10200
> Exception in thread "main" java.lang.IllegalArgumentException: App admin
> client class name not specified for type SPARK
> at
> org.apache.hadoop.yarn.client.api.AppAdminClient.createAppAdminClient(AppAdminClient.java:76)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:579)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:123)
> {code}
> From
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java#L76,
> it seems that this is because user does not have the setting:
> {code}
> yarn.application.admin.client.class.SPARK
> {code}
> set up in their client configuration.
> However, even if this setting is present, we still need to have an
> implementation available for the application type. From my internal
> discussions - Jobs don't have a notion of stop / resume functionality at YARN
> level. If some apps like Spark need it, it has to be implemented at those
> framework's level.
> Therefore, the above error message is a bit misleading in that, even if
> "yarn.application.admin.client.class.SPARK" is supplied (or for that matter -
> yarn.application.admin.client.class.MAPREDUCE), if there is no implementation
> actually available underneath to handle the stop/start functionality then, we
> will fail again, albeit with a different error here:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java#L85.
> As such, maybe this error message can be potentially improved to say
> something like:
> {code}
> Exception in thread "main" java.lang.IllegalArgumentException: App admin
> client class name not specified for type SPARK. Please ensure the App admin
> client class actually exists within SPARK to handle this functionality.
> {code}
> or something similar.
> Further, documentation around "-stop" and "-start" options will need to be
> improved here ->
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application_or_app
> as it does not mention anything about having an implementation at the
> framework level for the YARN stop/start command to succeed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348788#comment-17348788
]
Hadoop QA commented on YARN-10779:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
19s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
23s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 34s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 26m
42s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 4m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
5s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
5s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
35s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 43s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1006/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new +
234 unchanged - 0 fixed = 235 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
|
[jira] [Updated] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10779:
Attachment: YARN-10779-003.patch
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-001.patch, YARN-10779-002.patch,
> YARN-10779-003.patch, YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348629#comment-17348629
]
Hadoop QA commented on YARN-10779:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
57s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
5s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 18s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 24m
54s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 4m
2s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
35s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
35s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
32s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
32s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 50s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1005/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 3 new +
234 unchanged - 0 fixed = 237 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
|
[jira] [Updated] (YARN-10780) Optimise retrieval of configured node labels in CS queues
[
https://issues.apache.org/jira/browse/YARN-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10780:
Description:
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 5000 ~= over 12 million iteration + additional properties
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
was:
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 2500 * 5000 ~= over 300 million iteration + additional properties
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
> Optimise retrieval of configured node labels in CS queues
> -
>
> Key: YARN-10780
> URL: https://issues.apache.org/jira/browse/YARN-10780
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
>
> CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
> respect to queue numbers (its O(n*m), where n is the number of queues and m
> is the number of properties set by each queue). During CS reinit, the node
> labels are often queried, however looking at the code:
> {code:java}
> for (Entry stringStringEntry : this) {
> e = stringStringEntry;
> String key = e.getKey();
> if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
> + DOT)) {
> // Find in
> // .accessible-node-labels..property
> int labelStartIdx =
> key.indexOf(ACCESSIBLE_NODE_LABELS)
> + ACCESSIBLE_NODE_LABELS.length() + 1;
> int labelEndIndx = key.indexOf('.', labelStartIdx);
> String labelName = key.substring(labelStartIdx, labelEndIndx);
>
[jira] [Updated] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10779:
Attachment: YARN-10779-002.patch
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-001.patch, YARN-10779-002.patch,
> YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10780) Optimise retrieval of configured node labels in CS queues
[
https://issues.apache.org/jira/browse/YARN-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10780:
Description:
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 2500 * 5000 ~= over 300 million iteration + additional properties
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
was:
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 5000 ~= over 12 million iteration + additional properties
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
> Optimise retrieval of configured node labels in CS queues
> -
>
> Key: YARN-10780
> URL: https://issues.apache.org/jira/browse/YARN-10780
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
>
> CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
> respect to queue numbers (its O(n*m), where n is the number of queues and m
> is the number of properties set by each queue). During CS reinit, the node
> labels are often queried, however looking at the code:
> {code:java}
> for (Entry stringStringEntry : this) {
> e = stringStringEntry;
> String key = e.getKey();
> if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
> + DOT)) {
> // Find in
> // .accessible-node-labels..property
> int labelStartIdx =
> key.indexOf(ACCESSIBLE_NODE_LABELS)
> + ACCESSIBLE_NODE_LABELS.length() + 1;
> int labelEndIndx = key.indexOf('.', labelStartIdx);
> String labelName = key.substring(labelStartIdx, labelEndIndx);
>
[jira] [Updated] (YARN-10780) Optimise retrieval of configured node labels in CS queues
[
https://issues.apache.org/jira/browse/YARN-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Gyori updated YARN-10780:
Description:
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 5000 ~= over 12 million iteration + additional properties
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
was:
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 5000 ~= over 12 million iteration
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
> Optimise retrieval of configured node labels in CS queues
> -
>
> Key: YARN-10780
> URL: https://issues.apache.org/jira/browse/YARN-10780
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
>
> CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
> respect to queue numbers (its O(n*m), where n is the number of queues and m
> is the number of properties set by each queue). During CS reinit, the node
> labels are often queried, however looking at the code:
> {code:java}
> for (Entry stringStringEntry : this) {
> e = stringStringEntry;
> String key = e.getKey();
> if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
> + DOT)) {
> // Find in
> // .accessible-node-labels..property
> int labelStartIdx =
> key.indexOf(ACCESSIBLE_NODE_LABELS)
> + ACCESSIBLE_NODE_LABELS.length() + 1;
> int labelEndIndx = key.indexOf('.', labelStartIdx);
> String labelName = key.substring(labelStartIdx, labelEndIndx);
> configuredNodeLabels.add(labelName);
> }
>
[jira] [Created] (YARN-10780) Optimise retrieval of configured node labels in CS queues
Andras Gyori created YARN-10780:
---
Summary: Optimise retrieval of configured node labels in CS queues
Key: YARN-10780
URL: https://issues.apache.org/jira/browse/YARN-10780
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Andras Gyori
Assignee: Andras Gyori
CapacitySchedulerConfiguration#getConfiguredNodeLabels scales poorly with
respect to queue numbers (its O(n*m), where n is the number of queues and m is
the number of properties set by each queue). During CS reinit, the node labels
are often queried, however looking at the code:
{code:java}
for (Entry stringStringEntry : this) {
e = stringStringEntry;
String key = e.getKey();
if (key.startsWith(getQueuePrefix(queuePath) + ACCESSIBLE_NODE_LABELS
+ DOT)) {
// Find in
// .accessible-node-labels..property
int labelStartIdx =
key.indexOf(ACCESSIBLE_NODE_LABELS)
+ ACCESSIBLE_NODE_LABELS.length() + 1;
int labelEndIndx = key.indexOf('.', labelStartIdx);
String labelName = key.substring(labelStartIdx, labelEndIndx);
configuredNodeLabels.add(labelName);
}
}
{code}
This method iterates through ALL properties set in the configuration. For
example in case of initialising 2500 queues, each having at least 2 properties:
2500 * 5000 ~= over 12 million iteration
There are some ways to resolve this issue while keeping backward compatibility:
# Create a property like the original accessible-node-labels, which contains
predefined labels. If it is set, then getConfiguredNodeLabels get the value of
this property, otherwise it falls back to the old logic. I think
accessible-node-labels are not used for this purpose (though I have a feeling
that it should have been).
# Collect node labels for all queues at the beginning of parseQueue and only
iterate through the properties once. This will increase the space complexity in
exchange of not requiring intervention from user's perspective.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348319#comment-17348319
]
Hadoop QA commented on YARN-10779:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 13s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 24m
57s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 4m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
36s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 42s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1004/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 13 new +
234 unchanged - 0 fixed = 247 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
|
[jira] [Updated] (YARN-10505) Extend the maximum-capacity property to support Fair Scheduler migration
[ https://issues.apache.org/jira/browse/YARN-10505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10505: Description: The property root.users.maximum-capacity could mean the following things: * Parent Percentage: maximum capacity relative to its parent. If it’s set to 50, then it means that the capacity is capped with respect to the parent. * Cluster Percentage: maximum capacity expressed as a percentage of the overall cluster capacity. Note that Fair Scheduler supports the following settings: * Single percentage (cluster) * Two percentages (cluster) * Absolute resources It is recommended that all three formats are supported for maximum-capacity after introducing weight mode. was: The property root.users.maximum-capacity could mean the following things: * Parent Percentage: maximum capacity relative to its parent. If it’s set to 50, then it means that the capacity is capped with respect to the parent. * Cluster Percentage: maximum capacity expressed as a percentage of the overall cluster capacity. Note that Fair Scheduler supports the following settings: * Single percentage (absolute) * Two percentages (absolute) * Absolute resources It is recommended that all three formats are supported for maximum-capacity after introducing weight mode. > Extend the maximum-capacity property to support Fair Scheduler migration > > > Key: YARN-10505 > URL: https://issues.apache.org/jira/browse/YARN-10505 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > > The property root.users.maximum-capacity could mean the following things: > * Parent Percentage: maximum capacity relative to its parent. If it’s set to > 50, then it means that the capacity is capped with respect to the parent. > * Cluster Percentage: maximum capacity expressed as a percentage of the > overall cluster capacity. > > Note that Fair Scheduler supports the following settings: > * Single percentage (cluster) > * Two percentages (cluster) > * Absolute resources > > It is recommended that all three formats are supported for maximum-capacity > after introducing weight mode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10745) Change Log level from info to debug for few logs and remove unnecessary debuglog checks
[
https://issues.apache.org/jira/browse/YARN-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348301#comment-17348301
]
Hadoop QA commented on YARN-10745:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
56s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
26s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
2s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
37s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 23s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 40m
17s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 8m
3s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
2s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m
2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
28s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
28s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 1s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1003/artifact/out/diff-checkstyle-root.txt{color}
| {color:orange} root: The patch generated 6 new + 519 unchanged - 19 fixed =
525 total (was 538) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} |
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348213#comment-17348213
]
Peter Bacsko commented on YARN-10779:
-
TODO: new property needs a "." character after the prefix variable.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-001.patch, YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348202#comment-17348202
]
Peter Bacsko commented on YARN-10779:
-
Ok, uploaded patch v1, I forgot to update {{yarn-default.xml}}, that will
happen in the next patch.
[~gandras] [~snemeth] care to review?
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-001.patch, YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10779:
Attachment: YARN-10779-001.patch
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-001.patch, YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348147#comment-17348147
]
Peter Bacsko edited comment on YARN-10779 at 5/20/21, 8:34 AM:
---
"It is a possibility that Node A sends this PB message to Node B, which means
that this configuration must be set on Node B in order to take effect"
What you're saying is true, but this applies to the whole Hadoop ecosystem, not
just to this particular setting. Imagine having different container executors
on different nodes, or security enabled/disabled, that would wreak havoc.
Proper configuration management is mandatory and expected.
was (Author: pbacsko):
"It is a possibility that Node A sends this PB message to Node B, which means
that this configuration must be set on Node B in order to take effect"
What you're saying is true, but this applies to the whole Hadoop ecosystem, not
just to this particular setting. Imagine having different container executors
on different nodes, or security enabled/disabled, that would wreak havoc.
Proper configuration management is mandatory and assumed.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348147#comment-17348147
]
Peter Bacsko edited comment on YARN-10779 at 5/20/21, 8:33 AM:
---
"It is a possibility that Node A sends this PB message to Node B, which means
that this configuration must be set on Node B in order to take effect"
What you're saying is true, but this applies to the whole Hadoop ecosystem, not
just to this particular setting. Imagine having different container executors
on different nodes, or security enabled/disabled, that would wreak havoc.
Proper configuration management is mandatory and assumed.
was (Author: pbacsko):
"It is a possibility that Node A sends this PB message to Node B, which means
that this configuration must be set on Node B in order to take effect"
What you're saying is true, but this applies to the whole Hadoop ecosystem, not
just to this particular setting. Imagine having different container executors
on different nodes, or security enabled/disabled, that would wreak havoc.
Proper configuration management is a mandatory and assumed.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348147#comment-17348147
]
Peter Bacsko commented on YARN-10779:
-
"It is a possibility that Node A sends this PB message to Node B, which means
that this configuration must be set on Node B in order to take effect"
What you're saying is true, but this applies to the whole Hadoop ecosystem, not
just to this particular setting. Imagine having different container executors
on different nodes, or security enabled/disabled, that would wreak havoc.
Proper configuration management is a mandatory and assumed.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10543) Timeline Server V1.5 not supporting audit log
[ https://issues.apache.org/jira/browse/YARN-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348105#comment-17348105 ] ANANDA G B commented on YARN-10543: --- [~zhuqi] Thanks for your review. I will write UT and attach patch today. > Timeline Server V1.5 not supporting audit log > - > > Key: YARN-10543 > URL: https://issues.apache.org/jira/browse/YARN-10543 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Affects Versions: 3.1.1 >Reporter: ANANDA G B >Assignee: ANANDA G B >Priority: Major > Labels: TimeLine > Attachments: YARN-10543-001.patch, YARN-10543-002.patch > > > Like JHS, TS V1.5 can also support audit log when Timeline REST APIs are > accessed. This will helps to know the operation performed on TS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348099#comment-17348099
]
Andras Gyori commented on YARN-10779:
-
[~pbacsko] What I mean by "PB classes are shared throughout the network" is
that the actual PBImpl classes could probably be instantiated on any node
locally. It is a possibility that Node A sends this PB message to Node B, which
means that this configuration must be set on Node B in order to take effect. If
Node A sends the same type of PB message to Node C, which does not set this
property, it might be surprising for the user. I hope it makes more sense now.
However, I do not know if this is a problem or not.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10779) Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl
[
https://issues.apache.org/jira/browse/YARN-10779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348090#comment-17348090
]
Peter Bacsko commented on YARN-10779:
-
[~gandras] these classes are locally instantiated adapter classes that mostly
use generated ProtoBuf classes as an input. They are not sent over the wire. We
don't directly work on ProtoBuf classes, but on these PBImpl stuff that are the
implementations of the non-PBImpl classes like {{ApplicationSubmissionContext}}.
> Add option to disable lowercase conversion in GetApplicationsRequestPBImpl
> and ApplicationSubmissionContextPBImpl
> -
>
> Key: YARN-10779
> URL: https://issues.apache.org/jira/browse/YARN-10779
> Project: Hadoop YARN
> Issue Type: Task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10779-POC.patch
>
>
> In both {{GetApplicationsRequestPBImpl}} and
> {{ApplicationSubmissionContextPBImpl}}, there is a forced lowercase
> conversion:
> {noformat}
> checkTags(tags);
> // Convert applicationTags to lower case and add
> this.applicationTags = new TreeSet<>();
> for (String tag : tags) {
> this.applicationTags.add(StringUtils.toLowerCase(tag));
> }
> }
> {noformat}
> However, we encountered some cases where this is not desirable for "userid"
> tags.
> Proposed solution: since both classes are pretty low-level and can be often
> instantiated, a {{Configuration}} object which loads {{yarn-site.xml}} should
> be cached inside them. A new property should be created which tells whether
> lowercase conversion should occur or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
