[jira] [Commented] (YARN-9804) Update ATSv2 document for latest feature supports
[ https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920634#comment-16920634 ] Abhishek Modi commented on YARN-9804: - Thanks [~rohithsharma] for working on it. Some minor comments: Road map include -> Road map includes Simple authorization in terms of a configurable whitelist of users and groups who can read timeline data -> Support for simple authorization has been added in terms of a configurable whitelist of users and groups who can read timeline data. YARN Client integrates with ATSv2. -> YARN Client has been integrated with ATSv2. This enables fetching application/attempt/container report from TimelineReader if details not present in ResouceManager. -> This enables fetching application/attempt/container report from TimelineReader if details are not present in ResouceManager. It set true -> If set true Since Yarn Cli support has been added, should we remove this line: Currently there is no support for command line access. > Update ATSv2 document for latest feature supports > - > > Key: YARN-9804 > URL: https://issues.apache.org/jira/browse/YARN-9804 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Blocker > Attachments: YARN-9804.01.patch > > > Revisit ATSv2 documents and update for GA features. And also for the road map. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9789) Disable Option for Write Ahead Logs of LogMutation
[
https://issues.apache.org/jira/browse/YARN-9789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920632#comment-16920632
]
Prabhu Joseph commented on YARN-9789:
-
[~abmodi] Can you review this Jira when you get time. This stops reading write
ahead logs from {{ConfigurationStore}} (ZK, LevelDb) when maxLogs is set to
zero. Thanks.
> Disable Option for Write Ahead Logs of LogMutation
> --
>
> Key: YARN-9789
> URL: https://issues.apache.org/jira/browse/YARN-9789
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9789-001.patch
>
>
> When yarn.scheduler.configuration.store.max-logs is set to zero, the
> YARNConfigurationStore (ZK, LevelDB) reads the write ahead logs from the
> backend which is not needed.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9780) SchedulerConf Mutation Api does not Allow Stop and Remove Queue in a single call
[
https://issues.apache.org/jira/browse/YARN-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920630#comment-16920630
]
Prabhu Joseph commented on YARN-9780:
-
[~abmodi] Can you review this Jira when you get time. This allows SchedulerConf
to allow Stop and Remove Queue in a single Rest Call. Thanks.
> SchedulerConf Mutation Api does not Allow Stop and Remove Queue in a single
> call
>
>
> Key: YARN-9780
> URL: https://issues.apache.org/jira/browse/YARN-9780
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9780-001.patch, YARN-9780-002.patch,
> YARN-9780-003.patch
>
>
> SchedulerConf Mutation Api does not Allow Stop and Remove Queue in a single
> call. The queue has to be stopped before removing and so it is useful to
> allow both Stop and remove queue in a single call.
> *Repro:*
> {code:java}
> Capacity-Scheduler.xml:
> yarn.scheduler.capacity.root.queues = new, default, dummy
> yarn.scheduler.capacity.root.default.capacity = 60
> yarn.scheduler.capacity.root.dummy.capacity = 30
> yarn.scheduler.capacity.root.new.capacity = 10
> curl -v -X PUT -d @abc.xml -H "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf'
> abc.xml
>
>
> root.default
>
>
> capacity
> 70
>
>
>
>
> root.new
>
>
> state
> STOPPED
>
>
>
> root.new
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9781) SchedConfCli to get current stored scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920629#comment-16920629 ] Prabhu Joseph commented on YARN-9781: - [~abmodi] Can you review this Jira when you get time. This provides get Scheduler configuration option in SchedConfCLI. Thanks. > SchedConfCli to get current stored scheduler configuration > -- > > Key: YARN-9781 > URL: https://issues.apache.org/jira/browse/YARN-9781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9781-001.patch, YARN-9781-002.patch, > YARN-9781-003.patch > > > SchedConfCLI currently allows to add / remove / remove queue. It does not > support get configuration which RMWebServices provides as part of YARN-8559. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9400) Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId
[
https://issues.apache.org/jira/browse/YARN-9400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920625#comment-16920625
]
Prabhu Joseph commented on YARN-9400:
-
Thanks [~abmodi]
> Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId
> --
>
> Key: YARN-9400
> URL: https://issues.apache.org/jira/browse/YARN-9400
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9400-001.patch
>
>
> If clause to validate whether appIdStr starts with "application" is not
> required at EntityGroupFSTimelineStore#parseApplicationId
> {code}
> // converts the String to an ApplicationId or null if conversion failed
> private static ApplicationId parseApplicationId(String appIdStr) {
> ApplicationId appId = null;
> if (appIdStr.startsWith(ApplicationId.appIdStrPrefix)) {
> try {
> appId = ApplicationId.fromString(appIdStr);
> } catch (IllegalArgumentException e) {
> appId = null;
> }
> }
> return appId;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9400) Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId
[
https://issues.apache.org/jira/browse/YARN-9400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920624#comment-16920624
]
Abhishek Modi commented on YARN-9400:
-
Thanks [~Prabhu Joseph]. lgtm. will commit to trunk.
> Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId
> --
>
> Key: YARN-9400
> URL: https://issues.apache.org/jira/browse/YARN-9400
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9400-001.patch
>
>
> If clause to validate whether appIdStr starts with "application" is not
> required at EntityGroupFSTimelineStore#parseApplicationId
> {code}
> // converts the String to an ApplicationId or null if conversion failed
> private static ApplicationId parseApplicationId(String appIdStr) {
> ApplicationId appId = null;
> if (appIdStr.startsWith(ApplicationId.appIdStrPrefix)) {
> try {
> appId = ApplicationId.fromString(appIdStr);
> } catch (IllegalArgumentException e) {
> appId = null;
> }
> }
> return appId;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7982) Do ACLs check while retrieving entity-types per application
[
https://issues.apache.org/jira/browse/YARN-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920623#comment-16920623
]
Prabhu Joseph commented on YARN-7982:
-
Thanks [~abmodi].
> Do ACLs check while retrieving entity-types per application
> ---
>
> Key: YARN-7982
> URL: https://issues.apache.org/jira/browse/YARN-7982
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-7982-001.patch, YARN-7982-002.patch,
> YARN-7982-003.patch, YARN-7982-004.patch
>
>
> REST end point {{/apps/$appid/entity-types}} retrieves all the entity-types
> for given application. This need to be guarded with ACL check
> {code}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002?user.name=ambari-qa1;
> {"exception":"ForbiddenException","message":"java.lang.Exception: User
> ambari-qa1 is not allowed to read TimelineService V2
> data.","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002/entity-types?user.name=ambari-qa1;
> ["YARN_APPLICATION_ATTEMPT","YARN_CONTAINER"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8174) Add containerId to ResourceLocalizationService fetch failure log statement
[
https://issues.apache.org/jira/browse/YARN-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920622#comment-16920622
]
Prabhu Joseph commented on YARN-8174:
-
Thanks [~abmodi].
> Add containerId to ResourceLocalizationService fetch failure log statement
> --
>
> Key: YARN-8174
> URL: https://issues.apache.org/jira/browse/YARN-8174
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-8174.1.patch, YARN-8174.2.patch, YARN-8174.3.patch
>
>
> When a localization for a resource failed due to change in timestamp, there
> is no containerId logged to correlate.
> {code}
> 2018-04-18 07:31:46,033 WARN localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:processHeartbeat(1017)) - {
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo,
> 1524036694502, FILE, null } failed: Resource
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo
> changed on src filesystem (expected 1524036694502, was 1524036694502
> java.io.IOException: Resource
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo
> changed on src filesystem (expected 1524036694502, was 1524036694502
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:258)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:360)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:360)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8174) Add containerId to ResourceLocalizationService fetch failure log statement
[
https://issues.apache.org/jira/browse/YARN-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920619#comment-16920619
]
Abhishek Modi commented on YARN-8174:
-
v3 patch lgtm. Committed to trunk.
> Add containerId to ResourceLocalizationService fetch failure log statement
> --
>
> Key: YARN-8174
> URL: https://issues.apache.org/jira/browse/YARN-8174
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-8174.1.patch, YARN-8174.2.patch, YARN-8174.3.patch
>
>
> When a localization for a resource failed due to change in timestamp, there
> is no containerId logged to correlate.
> {code}
> 2018-04-18 07:31:46,033 WARN localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:processHeartbeat(1017)) - {
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo,
> 1524036694502, FILE, null } failed: Resource
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo
> changed on src filesystem (expected 1524036694502, was 1524036694502
> java.io.IOException: Resource
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo
> changed on src filesystem (expected 1524036694502, was 1524036694502
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:258)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:360)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:360)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8174) Add containerId to ResourceLocalizationService fetch failure log statement
[
https://issues.apache.org/jira/browse/YARN-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi updated YARN-8174:
Summary: Add containerId to ResourceLocalizationService fetch failure log
statement (was: Add containerId to ResourceLocalizationService fetch failure)
> Add containerId to ResourceLocalizationService fetch failure log statement
> --
>
> Key: YARN-8174
> URL: https://issues.apache.org/jira/browse/YARN-8174
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-8174.1.patch, YARN-8174.2.patch, YARN-8174.3.patch
>
>
> When a localization for a resource failed due to change in timestamp, there
> is no containerId logged to correlate.
> {code}
> 2018-04-18 07:31:46,033 WARN localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:processHeartbeat(1017)) - {
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo,
> 1524036694502, FILE, null } failed: Resource
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo
> changed on src filesystem (expected 1524036694502, was 1524036694502
> java.io.IOException: Resource
> hdfs://tarunhdp-1.openstacklocal:8020/user/ambari-qa/.staging/job_1523550428406_0016/job.splitmetainfo
> changed on src filesystem (expected 1524036694502, was 1524036694502
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:258)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:360)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:360)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7982) Do ACLs check while retrieving entity-types per application
[
https://issues.apache.org/jira/browse/YARN-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920610#comment-16920610
]
Abhishek Modi commented on YARN-7982:
-
Thanks [~Prabhu Joseph]. v4 patch looks good to me. Will commit it shortly.
> Do ACLs check while retrieving entity-types per application
> ---
>
> Key: YARN-7982
> URL: https://issues.apache.org/jira/browse/YARN-7982
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-7982-001.patch, YARN-7982-002.patch,
> YARN-7982-003.patch, YARN-7982-004.patch
>
>
> REST end point {{/apps/$appid/entity-types}} retrieves all the entity-types
> for given application. This need to be guarded with ACL check
> {code}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002?user.name=ambari-qa1;
> {"exception":"ForbiddenException","message":"java.lang.Exception: User
> ambari-qa1 is not allowed to read TimelineService V2
> data.","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002/entity-types?user.name=ambari-qa1;
> ["YARN_APPLICATION_ATTEMPT","YARN_CONTAINER"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5913) Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext
[ https://issues.apache.org/jira/browse/YARN-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-5913: -- Assignee: Yousef Abu-Salah > Consolidate "resource" and "amResourceRequest" in ApplicationSubmissionContext > -- > > Key: YARN-5913 > URL: https://issues.apache.org/jira/browse/YARN-5913 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yousef Abu-Salah >Priority: Minor > Labels: newbie > > Usage of these two variables overlaps and causes confusion. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6425) Move out FS state dump code out of method update()
[
https://issues.apache.org/jira/browse/YARN-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned YARN-6425:
--
Assignee: Yousef Abu-Salah
> Move out FS state dump code out of method update()
> --
>
> Key: YARN-6425
> URL: https://issues.apache.org/jira/browse/YARN-6425
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yousef Abu-Salah
>Priority: Major
> Labels: newbie++
>
> Better to move out FS state dump code out of update()
> {code}
> if (LOG.isDebugEnabled()) {
> if (--updatesToSkipForDebug < 0) {
> updatesToSkipForDebug = UPDATE_DEBUG_FREQUENCY;
> dumpSchedulerState();
> }
> }
> {code}
> And, after that we should distinct between update call and update thread
> duration like before YARN-6112.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8995) Log the event type of the too big AsyncDispatcher event queue size, and add the information to the metrics.
[ https://issues.apache.org/jira/browse/YARN-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920568#comment-16920568 ] Tao Yang commented on YARN-8995: Thanks [~zhuqi] for the update. Patch LGTM, could you please also fix the remaining check-style warnings? Hi, [~cheersyang], please help to review again, are these changes ok to you? > Log the event type of the too big AsyncDispatcher event queue size, and add > the information to the metrics. > > > Key: YARN-8995 > URL: https://issues.apache.org/jira/browse/YARN-8995 > Project: Hadoop YARN > Issue Type: Improvement > Components: metrics, nodemanager, resourcemanager >Affects Versions: 3.2.0, 3.3.0 >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: TestStreamPerf.java, YARN-8995.001.patch, > YARN-8995.002.patch, YARN-8995.003.patch, YARN-8995.004.patch, > YARN-8995.005.patch, YARN-8995.006.patch, YARN-8995.007.patch, > YARN-8995.008.patch, YARN-8995.009.patch, YARN-8995.010.patch, > YARN-8995.011.patch, YARN-8995.012.patch > > > In our growing cluster，there are unexpected situations that cause some event > queues to block the performance of the cluster, such as the bug of > https://issues.apache.org/jira/browse/YARN-5262 . I think it's necessary to > log the event type of the too big event queue size, and add the information > to the metrics, and the threshold of queue size is a parametor which can be > changed. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7982) Do ACLs check while retrieving entity-types per application
[
https://issues.apache.org/jira/browse/YARN-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920528#comment-16920528
]
Hadoop QA commented on YARN-7982:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-7982 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979075/YARN-7982-004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 81827086451f 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 751b5a1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24712/testReport/ |
| Max. process+thread count | 358 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24712/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Do ACLs check while retrieving entity-types per
[jira] [Commented] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920503#comment-16920503
]
Hudson commented on YARN-9791:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17219 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17219/])
YARN-9791. Queue Mutation API does not allow to remove a config. (abmodi: rev
751b5a1ac871734c46cb934d100a5367f1dbf791)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestMutableCSConfigurationProvider.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/MutableCSConfigurationProvider.java
> Queue Mutation API does not allow to remove a config
>
>
> Key: YARN-9791
> URL: https://issues.apache.org/jira/browse/YARN-9791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9791-001.patch, YARN-9791-002.patch
>
>
> Queue Mutation API does not allow to remove a config. When removing a node
> label from a queue and it's capacity config
> {code}
>
>
> root.batch
>
>
> accessible-node-labels
>
>
>
> accessible-node-labels.x.capacity
>
>
>
>
>
> {code}
> It fails with below.
> {code}
> Caused by: java.lang.NumberFormatException: empty String
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1632)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:682)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:185)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:157)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:139)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:259)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:785)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:497)
> ... 72 more
> {code}
> We can fix this by providing a separate XmlElement to remove config.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7982) Do ACLs check while retrieving entity-types per application
[
https://issues.apache.org/jira/browse/YARN-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-7982:
Attachment: YARN-7982-004.patch
> Do ACLs check while retrieving entity-types per application
> ---
>
> Key: YARN-7982
> URL: https://issues.apache.org/jira/browse/YARN-7982
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-7982-001.patch, YARN-7982-002.patch,
> YARN-7982-003.patch, YARN-7982-004.patch
>
>
> REST end point {{/apps/$appid/entity-types}} retrieves all the entity-types
> for given application. This need to be guarded with ACL check
> {code}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002?user.name=ambari-qa1;
> {"exception":"ForbiddenException","message":"java.lang.Exception: User
> ambari-qa1 is not allowed to read TimelineService V2
> data.","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002/entity-types?user.name=ambari-qa1;
> ["YARN_APPLICATION_ATTEMPT","YARN_CONTAINER"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9106) Add option to graceful decommission to not wait for applications
[ https://issues.apache.org/jira/browse/YARN-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920473#comment-16920473 ] Íñigo Goiri commented on YARN-9106: --- The main issue I see here is that this is an option that applies to everything running in the cluster. Not sure how to do it, but the best way would be to do this in a per app/job base. I guess that the case that [~abmodi] mentions would be the one where this makes sense to set it at cluster level. > Add option to graceful decommission to not wait for applications > > > Key: YARN-9106 > URL: https://issues.apache.org/jira/browse/YARN-9106 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Mikayla Konst >Assignee: Mikayla Konst >Priority: Major > Attachments: YARN-9106.patch > > > Add property > yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-applications. > If true (the default), the resource manager waits for all containers, as well > as all applications associated with those containers, to finish before > gracefully decommissioning a node. > If false, the resource manager only waits for containers, but not > applications, to finish. For map-only jobs or other jobs in which mappers do > not need to serve shuffle data, this allows nodes to be decommissioned as > soon as their containers are finished as opposed to when the job is done. > Add property > yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-app-masters. > If false, during graceful decommission, when the resource manager waits for > all containers on a node to finish, it will not wait for app master > containers to finish. Defaults to true. This property should only be set to > false if app master failure is recoverable. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920465#comment-16920465
]
Prabhu Joseph commented on YARN-9791:
-
Thanks [~abmodi].
> Queue Mutation API does not allow to remove a config
>
>
> Key: YARN-9791
> URL: https://issues.apache.org/jira/browse/YARN-9791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9791-001.patch, YARN-9791-002.patch
>
>
> Queue Mutation API does not allow to remove a config. When removing a node
> label from a queue and it's capacity config
> {code}
>
>
> root.batch
>
>
> accessible-node-labels
>
>
>
> accessible-node-labels.x.capacity
>
>
>
>
>
> {code}
> It fails with below.
> {code}
> Caused by: java.lang.NumberFormatException: empty String
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1632)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:682)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:185)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:157)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:139)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:259)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:785)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:497)
> ... 72 more
> {code}
> We can fix this by providing a separate XmlElement to remove config.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920462#comment-16920462
]
Abhishek Modi commented on YARN-9791:
-
Thanks [~Prabhu Joseph]. Latest patch looks good to me. Committed to trunk.
> Queue Mutation API does not allow to remove a config
>
>
> Key: YARN-9791
> URL: https://issues.apache.org/jira/browse/YARN-9791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9791-001.patch, YARN-9791-002.patch
>
>
> Queue Mutation API does not allow to remove a config. When removing a node
> label from a queue and it's capacity config
> {code}
>
>
> root.batch
>
>
> accessible-node-labels
>
>
>
> accessible-node-labels.x.capacity
>
>
>
>
>
> {code}
> It fails with below.
> {code}
> Caused by: java.lang.NumberFormatException: empty String
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1632)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:682)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:185)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:157)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:139)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:259)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:785)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:497)
> ... 72 more
> {code}
> We can fix this by providing a separate XmlElement to remove config.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920458#comment-16920458
]
Prabhu Joseph commented on YARN-9791:
-
Thanks [~abmodi] for reviewing. Have addressed the review comments in
[^YARN-9791-002.patch] .
> Queue Mutation API does not allow to remove a config
>
>
> Key: YARN-9791
> URL: https://issues.apache.org/jira/browse/YARN-9791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9791-001.patch, YARN-9791-002.patch
>
>
> Queue Mutation API does not allow to remove a config. When removing a node
> label from a queue and it's capacity config
> {code}
>
>
> root.batch
>
>
> accessible-node-labels
>
>
>
> accessible-node-labels.x.capacity
>
>
>
>
>
> {code}
> It fails with below.
> {code}
> Caused by: java.lang.NumberFormatException: empty String
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1632)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:682)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:185)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:157)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:139)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:259)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:785)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:497)
> ... 72 more
> {code}
> We can fix this by providing a separate XmlElement to remove config.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9787) Typo in analysesErrorMsg
[
https://issues.apache.org/jira/browse/YARN-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920440#comment-16920440
]
kevin su commented on YARN-9787:
[~surendrasingh] Could you help me review the patch
> Typo in analysesErrorMsg
>
>
> Key: YARN-9787
> URL: https://issues.apache.org/jira/browse/YARN-9787
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: kevin su
>Priority: Trivial
> Labels: newbie, noob
> Attachments: YARN-9787.001.patch
>
>
> {code:java}
> analysis.append("Please check whether your etc/hadoop/mapred-site.xml "
> + "contains the below configuration:\n");
> {code}
> I think it should be {{/etc/hadoop/mapred-site.xml}}
> https://github.com/apache/hadoop/blob/2064ca015d1584263aac0cc20c60b925a3aff612/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L788-L789
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920366#comment-16920366
]
Hadoop QA commented on YARN-9791:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m
52s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9791 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979067/YARN-9791-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0365df0c62ae 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18d74fe |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24710/testReport/ |
| Max. process+thread count | 817 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24710/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Queue Mutation API does not allow to remove a config
>
[jira] [Commented] (YARN-9792) Document examples of SchedulerConf with Node Labels
[
https://issues.apache.org/jira/browse/YARN-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920345#comment-16920345
]
Hadoop QA commented on YARN-9792:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9792 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979068/YARN-9792-003.patch |
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 31572180f71a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 18d74fe |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 449 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24711/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Document examples of SchedulerConf with Node Labels
> ---
>
> Key: YARN-9792
> URL: https://issues.apache.org/jira/browse/YARN-9792
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9792-001.patch, YARN-9792-002.patch,
> YARN-9792-003.patch
>
>
> Document examples of SchedulerConf with Node Labels.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9792) Document examples of SchedulerConf with Node Labels
[ https://issues.apache.org/jira/browse/YARN-9792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-9792: Attachment: YARN-9792-003.patch > Document examples of SchedulerConf with Node Labels > --- > > Key: YARN-9792 > URL: https://issues.apache.org/jira/browse/YARN-9792 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Attachments: YARN-9792-001.patch, YARN-9792-002.patch, > YARN-9792-003.patch > > > Document examples of SchedulerConf with Node Labels. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7982) Do ACLs check while retrieving entity-types per application
[
https://issues.apache.org/jira/browse/YARN-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920337#comment-16920337
]
Abhishek Modi edited comment on YARN-7982 at 9/1/19 8:07 AM:
-
Thanks [~Prabhu Joseph] for clarification.
Some minor comments:
# In FileSystemTimelineReaderImpl, we should set userId only if it is passed
as null.
# Would it be possible to write unit test covering case where user id is not
specified.
was (Author: abmodi):
Thanks [~Prabhu Joseph] for clarification.
Some minor comments:
# In FileSystemTimelineReaderImpl, we should set userId only if it is null.
# Would it be possible to write unit test covering case where user id is not
specified.
> Do ACLs check while retrieving entity-types per application
> ---
>
> Key: YARN-7982
> URL: https://issues.apache.org/jira/browse/YARN-7982
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-7982-001.patch, YARN-7982-002.patch,
> YARN-7982-003.patch
>
>
> REST end point {{/apps/$appid/entity-types}} retrieves all the entity-types
> for given application. This need to be guarded with ACL check
> {code}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002?user.name=ambari-qa1;
> {"exception":"ForbiddenException","message":"java.lang.Exception: User
> ambari-qa1 is not allowed to read TimelineService V2
> data.","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002/entity-types?user.name=ambari-qa1;
> ["YARN_APPLICATION_ATTEMPT","YARN_CONTAINER"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7982) Do ACLs check while retrieving entity-types per application
[
https://issues.apache.org/jira/browse/YARN-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920337#comment-16920337
]
Abhishek Modi commented on YARN-7982:
-
Thanks [~Prabhu Joseph] for clarification.
Some minor comments:
# In FileSystemTimelineReaderImpl, we should set userId only if it is null.
# Would it be possible to write unit test covering case where user id is not
specified.
> Do ACLs check while retrieving entity-types per application
> ---
>
> Key: YARN-7982
> URL: https://issues.apache.org/jira/browse/YARN-7982
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-7982-001.patch, YARN-7982-002.patch,
> YARN-7982-003.patch
>
>
> REST end point {{/apps/$appid/entity-types}} retrieves all the entity-types
> for given application. This need to be guarded with ACL check
> {code}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002?user.name=ambari-qa1;
> {"exception":"ForbiddenException","message":"java.lang.Exception: User
> ambari-qa1 is not allowed to read TimelineService V2
> data.","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> [yarn@yarn-ats-3 ~]$ curl
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1552297011473_0002/entity-types?user.name=ambari-qa1;
> ["YARN_APPLICATION_ATTEMPT","YARN_CONTAINER"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9791:
Attachment: YARN-9791-002.patch
> Queue Mutation API does not allow to remove a config
>
>
> Key: YARN-9791
> URL: https://issues.apache.org/jira/browse/YARN-9791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9791-001.patch, YARN-9791-002.patch
>
>
> Queue Mutation API does not allow to remove a config. When removing a node
> label from a queue and it's capacity config
> {code}
>
>
> root.batch
>
>
> accessible-node-labels
>
>
>
> accessible-node-labels.x.capacity
>
>
>
>
>
> {code}
> It fails with below.
> {code}
> Caused by: java.lang.NumberFormatException: empty String
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1632)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:682)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:185)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:157)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:139)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:259)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:785)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:497)
> ... 72 more
> {code}
> We can fix this by providing a separate XmlElement to remove config.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8678) Queue Management API - rephrase error messages
[
https://issues.apache.org/jira/browse/YARN-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920329#comment-16920329
]
Prabhu Joseph commented on YARN-8678:
-
Thanks [~abmodi].
> Queue Management API - rephrase error messages
> --
>
> Key: YARN-8678
> URL: https://issues.apache.org/jira/browse/YARN-8678
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Akhil PB
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8678-002.patch, YARN-8768-001.patch
>
>
> 1. When stopping a running parent queue, error message thrown by API was not
> meaningful.
> For example: When tried to stop root queue, error message thrown was
> {{Failed to re-init queues : The parent queue:root state is STOPPED, child
> queue:default state cannot be RUNNING.}}
> It is evident that root queue update failed, but the message says
> {{queue:root state is STOPPED}}.
> 2. While tried to delete a running leaf queue, error message thrown by API
> was not meaningful.
> For example: Error message was {{Failed to re-init queues : root.default.prod
> is deleted from the new capacity scheduler configuration, but the queue is
> not yet in stopped state. Current State : RUNNING}}.
> Clearly deletion of queue root.default.prod failed with error, but the
> message says {{queues : root.default.prod is deleted from the new capacity
> scheduler configuration}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9106) Add option to graceful decommission to not wait for applications
[ https://issues.apache.org/jira/browse/YARN-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920328#comment-16920328 ] Abhishek Modi commented on YARN-9106: - Gracefully decommissioning nodes without waiting for applications to finish also makes sense when shuffle data is offloaded to persistent storage or shuffle service is running completely outside nodes. While running Yarn on cloud, it is very common to offload shuffle data to persistent volumes and remove nodes. cc [~elgoiri] > Add option to graceful decommission to not wait for applications > > > Key: YARN-9106 > URL: https://issues.apache.org/jira/browse/YARN-9106 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Mikayla Konst >Assignee: Mikayla Konst >Priority: Major > Attachments: YARN-9106.patch > > > Add property > yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-applications. > If true (the default), the resource manager waits for all containers, as well > as all applications associated with those containers, to finish before > gracefully decommissioning a node. > If false, the resource manager only waits for containers, but not > applications, to finish. For map-only jobs or other jobs in which mappers do > not need to serve shuffle data, this allows nodes to be decommissioned as > soon as their containers are finished as opposed to when the job is done. > Add property > yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-app-masters. > If false, during graceful decommission, when the resource manager waits for > all containers on a node to finish, it will not wait for app master > containers to finish. Defaults to true. This property should only be set to > false if app master failure is recoverable. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9793) Remove duplicate sentence from TimelineServiceV2.md
[ https://issues.apache.org/jira/browse/YARN-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920324#comment-16920324 ] Hudson commented on YARN-9793: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17217 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17217/]) YARN-9793. Remove duplicate sentence from TimelineServiceV2.md. (abmodi: rev fef65b4c2bcd1ea452f3a66eda91e58881b64427) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md > Remove duplicate sentence from TimelineServiceV2.md > --- > > Key: YARN-9793 > URL: https://issues.apache.org/jira/browse/YARN-9793 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2, docs >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9793.001.patch > > > In the documentation of the ATSv2, TimelineEntity objects description part > there is a duplication: > * configs: A map from a string (config name) to a string (config value) > representing all configs associated with the entity. Users can post the whole > config or a part of it in the configs field. *Supported for application and > generic entities. Supported for application and generic entities.* > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9793) Remove duplicate sentence from TimelineServiceV2.md
[ https://issues.apache.org/jira/browse/YARN-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-9793: Component/s: ATSv2 > Remove duplicate sentence from TimelineServiceV2.md > --- > > Key: YARN-9793 > URL: https://issues.apache.org/jira/browse/YARN-9793 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2, docs >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9793.001.patch > > > In the documentation of the ATSv2, TimelineEntity objects description part > there is a duplication: > * configs: A map from a string (config name) to a string (config value) > representing all configs associated with the entity. Users can post the whole > config or a part of it in the configs field. *Supported for application and > generic entities. Supported for application and generic entities.* > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9793) Remove duplicate sentence from TimelineServiceV2.md
[ https://issues.apache.org/jira/browse/YARN-9793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920323#comment-16920323 ] Abhishek Modi commented on YARN-9793: - Thanks [~kmarton] for the patch and [~adam.antal] for review. LGTM. Will commit shortly. > Remove duplicate sentence from TimelineServiceV2.md > --- > > Key: YARN-9793 > URL: https://issues.apache.org/jira/browse/YARN-9793 > Project: Hadoop YARN > Issue Type: Bug > Components: docs >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: YARN-9793.001.patch > > > In the documentation of the ATSv2, TimelineEntity objects description part > there is a duplication: > * configs: A map from a string (config name) to a string (config value) > representing all configs associated with the entity. Users can post the whole > config or a part of it in the configs field. *Supported for application and > generic entities. Supported for application and generic entities.* > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9791) Queue Mutation API does not allow to remove a config
[
https://issues.apache.org/jira/browse/YARN-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920320#comment-16920320
]
Abhishek Modi commented on YARN-9791:
-
Thanks [~Prabhu Joseph] for the patch.
Some minor comments:
# Can we replace kv.getValue by keyValue everywhere as.
# Can we update test to check that other configs are unchanged after applying
mutation.
> Queue Mutation API does not allow to remove a config
>
>
> Key: YARN-9791
> URL: https://issues.apache.org/jira/browse/YARN-9791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9791-001.patch
>
>
> Queue Mutation API does not allow to remove a config. When removing a node
> label from a queue and it's capacity config
> {code}
>
>
> root.batch
>
>
> accessible-node-labels
>
>
>
> accessible-node-labels.x.capacity
>
>
>
>
>
> {code}
> It fails with below.
> {code}
> Caused by: java.lang.NumberFormatException: empty String
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1842)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.hadoop.conf.Configuration.getFloat(Configuration.java:1632)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:682)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:697)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:136)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:185)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:172)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:157)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:139)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:259)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.reinitializeQueues(CapacitySchedulerQueueManager.java:171)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:785)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:497)
> ... 72 more
> {code}
> We can fix this by providing a separate XmlElement to remove config.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
