[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[
https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17130411#comment-17130411
]
Peter Bacsko commented on YARN-9460:
Thanks for the new patch [~BilwaST]. There are some checkstyle
"VisibilityModifier" complaints. Please validate if those are legitimate or not
(I can see that they're package private). If they're accessed in the descendant
classes, make them protected. If package private is legit, then add
{{@SuppressWarnings("checkstyle:visibilitymodifier")}} to the class.
> QueueACLsManager and ReservationsACLManager should not use instanceof checks
>
>
> Key: YARN-9460
> URL: https://issues.apache.org/jira/browse/YARN-9460
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-9460.001.patch, YARN-9460.002.patch,
> YARN-9460.003.patch, YARN-9460.004.patch
>
>
> QueueACLsManager and ReservationsACLManager should not use instanceof checks
> for the scheduler type.
> Rather, we should abstract this into two classes: Capacity and Fair variants
> of these ACL classes.
> QueueACLsManager and ReservationsACLManager could be abstract classes, but
> the implementation is the decision of one who will work on this jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[
https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17130447#comment-17130447
]
Peter Bacsko commented on YARN-9460:
Another comment:
{noformat}
if (scheduler instanceof CapacityScheduler) {
reservationsACLsManager = new
CapacityReservationsACLsManager(scheduler,
conf);
} else if (scheduler instanceof FairScheduler) {
reservationsACLsManager = new
FairReservationsACLsManager(scheduler,
conf);
}
{noformat}
This is very similar to what I mentioned above. If neither CS nor FS is set,
then this can cause an NPE. We need a ReservationsACLsManager which does
nothing in a final else branch, eg. {{PassThroughReservationsACLsManager}}
which always return true for {{checkAccess()}}.
> QueueACLsManager and ReservationsACLManager should not use instanceof checks
>
>
> Key: YARN-9460
> URL: https://issues.apache.org/jira/browse/YARN-9460
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-9460.001.patch, YARN-9460.002.patch,
> YARN-9460.003.patch, YARN-9460.004.patch
>
>
> QueueACLsManager and ReservationsACLManager should not use instanceof checks
> for the scheduler type.
> Rather, we should abstract this into two classes: Capacity and Fair variants
> of these ACL classes.
> QueueACLsManager and ReservationsACLManager could be abstract classes, but
> the implementation is the decision of one who will work on this jira.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9930: --- Attachment: YARN-9930-004.patch > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[ https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133081#comment-17133081 ] Peter Bacsko commented on YARN-9460: Thanks [~BilwaST] I have no further comments. > QueueACLsManager and ReservationsACLManager should not use instanceof checks > > > Key: YARN-9460 > URL: https://issues.apache.org/jira/browse/YARN-9460 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9460.001.patch, YARN-9460.002.patch, > YARN-9460.003.patch, YARN-9460.004.patch > > > QueueACLsManager and ReservationsACLManager should not use instanceof checks > for the scheduler type. > Rather, we should abstract this into two classes: Capacity and Fair variants > of these ACL classes. > QueueACLsManager and ReservationsACLManager could be abstract classes, but > the implementation is the decision of one who will work on this jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133177#comment-17133177
]
Peter Bacsko commented on YARN-9930:
_"Thanks Peter Bacsko, it will be make sense to create a one-pager doc and talk
about what is the behavior looks like."_
Yes, the upstream documentation should be updated. Either in this ticket or in
a follow-up JIRA.
_"For example, how this feature related to maximum-am-limit."_
It's completely unrelated. If the app attempt is allowed to pass through by
{{CSMaxRunningAppsEnforcer}}, only then comes the max AM percent check, which
can still put the attempt on hold.
The order is: max number of apps --> max number of running apps --> max AM
percent.
_"And how refreshQueue works with this feature (increase #running-app-limit
seems fine, but how about shrink #running-app-limit?)."_
There is already a test for refreshQueue when you increase it. When you
decrease, I think running apps still keep running, but new submissions are not
accepted until the necessary condition is satisfied, but I have to try this out.
> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9930-001.patch, YARN-9930-002.patch,
> YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch,
> YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch,
> YARN-9930-POC05.patch, screenshot-1.png
>
>
> In FairScheduler, there has limitation for max running which will let
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[ https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133345#comment-17133345 ] Peter Bacsko commented on YARN-9460: +1 (non-binding) pending Jenkins > QueueACLsManager and ReservationsACLManager should not use instanceof checks > > > Key: YARN-9460 > URL: https://issues.apache.org/jira/browse/YARN-9460 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9460.001.patch, YARN-9460.002.patch, > YARN-9460.003.patch, YARN-9460.004.patch, YARN-9460.005.patch > > > QueueACLsManager and ReservationsACLManager should not use instanceof checks > for the scheduler type. > Rather, we should abstract this into two classes: Capacity and Fair variants > of these ACL classes. > QueueACLsManager and ReservationsACLManager could be abstract classes, but > the implementation is the decision of one who will work on this jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133368#comment-17133368 ] Peter Bacsko commented on YARN-9930: OK, I did some extra testing. # Limit max-parallel-apps to 2, submit 4 apps, then refresh it to 4. Result: pending apps transitioned to Running state. # Limit max-parallel-apps to 4, submit 4 apps, then refresh it to 2. Result: running apps were still running, but new apps stayed in Accepted state. >From that point on, only 2 apps were allowed to run at the same time. I also did other manual tests: # Limit max-parallel-apps to 2 for all users, then override to 4 for a specific user. Result: specific user could run 4 apps. Other user could only run 2 apps. # Limit max-parallel-apps to 2 for the "root" queue and submit apps to a leaf queue. Result: leaf queue could only run 2 apps in parallel, which is expected. To me it looks functionally correct. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9888) Capacity scheduler: add support for default maxRunningApps limit per user
[
https://issues.apache.org/jira/browse/YARN-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko resolved YARN-9888.
Resolution: Duplicate
This feature will be implemented in YARN-9930. Closing this as duplicate.
> Capacity scheduler: add support for default maxRunningApps limit per user
> -
>
> Key: YARN-9888
> URL: https://issues.apache.org/jira/browse/YARN-9888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> Fair scheduler has the setting {{}} which limits how many
> running applications each user can have.
> Capacity scheduler lacks this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10316) FS-CS converter: convert userMaxApps, maxRunningApps settins
Peter Bacsko created YARN-10316:
---
Summary: FS-CS converter: convert userMaxApps, maxRunningApps
settins
Key: YARN-10316
URL: https://issues.apache.org/jira/browse/YARN-10316
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Peter Bacsko
Assignee: Peter Bacsko
In YARN-9930, support for maximum running applications (called "max parallel
apps") has been introduced.
The converter now can handle the following settings in {{fair-scheduler.xml}}:
* {{ }} per user
* {{}} per queue
* {{}}
* {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert userMaxApps, maxRunningApps settins
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Description:
In YARN-9930, support for maximum running applications (called "max parallel
apps") has been introduced.
The converter now can handle the following settings in {{fair-scheduler.xml}}:
* {{}} per user
* {{}} per queue
* {{}}
* {{}}
was:
In YARN-9930, support for maximum running applications (called "max parallel
apps") has been introduced.
The converter now can handle the following settings in {{fair-scheduler.xml}}:
* {{ }} per user
* {{}} per queue
* {{}}
* {{}}
> FS-CS converter: convert userMaxApps, maxRunningApps settins
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Summary: FS-CS converter: convert maxAppsDefault, maxRunningApps settings
(was: FS-CS converter: convert userMaxApps, maxRunningApps settins)
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136648#comment-17136648 ] Peter Bacsko commented on YARN-9930: [~adam.antal] thanks for the comment. I suggest talking about IRL, because that would be more effective & then I'll summarize my answer later. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136648#comment-17136648 ] Peter Bacsko edited comment on YARN-9930 at 6/16/20, 1:17 PM: -- [~adam.antal] thanks for the comment. I suggest talking about IRL/video meeting, because that would be more effective & then I'll summarize my answer later. was (Author: pbacsko): [~adam.antal] thanks for the comment. I suggest talking about IRL, because that would be more effective & then I'll summarize my answer later. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136648#comment-17136648 ] Peter Bacsko edited comment on YARN-9930 at 6/16/20, 1:24 PM: -- [~adam.antal] thanks for the comment. I suggest talking about it IRL/video meeting, because that would be more effective & then I'll summarize my answer later. was (Author: pbacsko): [~adam.antal] thanks for the comment. I suggest talking about IRL/video meeting, because that would be more effective & then I'll summarize my answer later. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Attachment: YARN-10136-001.patch
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10136-001.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10292) FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136726#comment-17136726 ] Peter Bacsko commented on YARN-10292: - Is this ticket done? Please close it if so. > FS-CS converter: add an option to enable asynchronous scheduling in > CapacityScheduler > - > > Key: YARN-10292 > URL: https://issues.apache.org/jira/browse/YARN-10292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10292.001.patch, YARN-10292.002.branch-3.3.patch, > YARN-10292.002.patch, YARN-10292.003.branch-3.3.patch > > > FS doesn't have an equivalent setting to the CapacityScheduler's > yarn.scheduler.capacity.schedule-asynchronously.enable option so the FS to CS > converter won't add this to the yarn-site.xml. An optional command line > switch should be added to support this option during migration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Attachment: YARN-10316-002.patch
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138388#comment-17138388
]
Peter Bacsko commented on YARN-9930:
So after talkin about the problems over voice chat, here is our conclusion:
_"So AFAIU it is absolutely normal that some queue is above its limit if the
configurations have been changed. Doesn't it need some special attention in
your algorithm when you recursively update the parents to search for queues
where new apps could be submitted?"_
No, I tested this case manually, first 4 running apps were allowed to run, but
no more. Then it went down to 3, then to 2. After that, it stayed at 2 running
apps and everything else was accepted. Functionality was consistent during the
test run.
_"I'd prefer your solution as its more clear, but since we already have the
existing logic, the questions arises: why do we need a separate enforcer
object? Couldn't it be implemented similarly? Or am I missing something here?"_
Yes, this approach is different from max-applications calculation. In theory,
having a consistent implementation accross a module is often desirable, but
this patch duplicates a battle-tested algorithm from {{MaxRunningAppsEnforcer}}
which was then adapted to CS. So this class can be trusted. Rewriting the
current patch would take a lot of time. I'm being very practical here, but I
don't think it's a huge violation of coding principles (apart from the
duplication, but that was also necessary IMO).
_"The existing implementation for max apps (that considers both running and
pending ones) calls the {{OrderingPolicy#getNumSchedulableEntities()}} and
compare it the to limit inside {{LeafQueue"}}_
This could be a bug! Apps that were marked as non-runnable are actually missing
from {{schedulableEntities}} (precisely to prevent them from being scheduled).
Looks like this needs a little change plus a unit test.
[~bteke]'s comments are also valid.
I'll address these issues and upload patch v5 soon.
> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9930-001.patch, YARN-9930-002.patch,
> YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch,
> YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch,
> YARN-9930-POC05.patch, screenshot-1.png
>
>
> In FairScheduler, there has limitation for max running which will let
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9930: --- Attachment: YARN-9930-005.patch > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-POC01.patch, YARN-9930-POC02.patch, YARN-9930-POC03.patch, > YARN-9930-POC04.patch, YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9930: --- Attachment: YARN-9930-005.patch > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-POC01.patch, YARN-9930-POC02.patch, YARN-9930-POC03.patch, > YARN-9930-POC04.patch, YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9930: --- Attachment: (was: YARN-9930-005.patch) > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-POC01.patch, YARN-9930-POC02.patch, YARN-9930-POC03.patch, > YARN-9930-POC04.patch, YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9930: --- Attachment: YARN-9930-006.patch > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-006.patch, YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch, > screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140391#comment-17140391 ] Peter Bacsko commented on YARN-9930: [~snemeth] could you review & commit the patch? I think now it's in good shape. [~adam.antal] & [~bteke] you can also check and give non-binding votes if you have time. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-006.patch, YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch, > screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko reopened YARN-9930: I'll backport this to branch-3.3 (I believe it should apply cleanly). > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-006.patch, YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch, > screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-9930: --- Attachment: YARN-9930-branch-3.3.001.patch > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-006.patch, YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch, > YARN-9930-branch-3.3.001.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Attachment: YARN-10316-003.patch
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch,
> YARN-10316-003.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
Peter Bacsko created YARN-10325: --- Summary: Document max-parallel-apps for Capacity Scheduler Key: YARN-10325 URL: https://issues.apache.org/jira/browse/YARN-10325 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler, capacityscheduler Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10325: Description: New feature introduced by YARN-9930 should be reflected in the upstream documentation. > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Labels: fs2cs (was: )
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch,
> YARN-10316-003.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10325: Attachment: YARN-10325-001.patch > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10325-001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17142182#comment-17142182
]
Peter Bacsko commented on YARN-10316:
-
I'm not sure if negative numbers are treated as errors. I haven't seen any
preconditions & errors checks in FS either, so it's likely that all comparisons
will be performed against the value -1 (which doesn't really cause any harm).
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch,
> YARN-10316-003.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17142182#comment-17142182
]
Peter Bacsko edited comment on YARN-10316 at 6/22/20, 3:27 PM:
---
I'm not sure if negative numbers are treated as errors. I haven't seen any
preconditions & errors checks in FS either, so it's likely that all comparisons
will be performed against the negative value (which doesn't really cause any
harm during runtime).
was (Author: pbacsko):
I'm not sure if negative numbers are treated as errors. I haven't seen any
preconditions & errors checks in FS either, so it's likely that all comparisons
will be performed against the value -1 (which doesn't really cause any harm).
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch,
> YARN-10316-003.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17142189#comment-17142189 ] Peter Bacsko commented on YARN-10325: - [~gandras] could you quickly review this? html docs are generated by running "mvn site". Thanks. > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10325-001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10316:
Attachment: YARN-10316-branch-3.3.001.patch
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch,
> YARN-10316-003.patch, YARN-10316-branch-3.3.001.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10325: Attachment: YARN-10325-branch-3.3.001.patch > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10325-001.patch, YARN-10325-branch-3.3.001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10277) CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy
[ https://issues.apache.org/jira/browse/YARN-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17143721#comment-17143721 ] Peter Bacsko commented on YARN-10277: - Looks good to me, but can we get rid of the checkstyle problems? > CapacityScheduler test TestUserGroupMappingPlacementRule should build proper > hierarchy > -- > > Key: YARN-10277 > URL: https://issues.apache.org/jira/browse/YARN-10277 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10277.001.patch, YARN-10277.002.patch > > > Since the CapacityScheduler internal implementation depends more and more on > queue being hierarchical, the test gets really hard to maintain. A lot of > test cases were failing because they used non existing queues, but the older > placement rule solution ignored missing parents, but since the leaf queue > change in CS, we must be able to get a full path for any queue, since all > queues are referenced by their full path. > This test should reflect this and instead of creating and expecting the > existance of fictional queues, it should create a proper queue hierarchy, > with a way to describe it better. > Currently we set up a bunch of mockito "when" statements to simulate the > queue behavior, but this is a hassle to maintain, and easy to miss a few > method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17143726#comment-17143726 ] Peter Bacsko edited comment on YARN-10325 at 6/24/20, 10:07 AM: [~snemeth] could you guy review & commit this? was (Author: pbacsko): [~gandras], [~snemeth] could you guys review this? > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10325-001.patch, YARN-10325-branch-3.3.001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17143726#comment-17143726 ] Peter Bacsko commented on YARN-10325: - [~gandras], [~snemeth] could you guys review this? > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10325-001.patch, YARN-10325-branch-3.3.001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9903) Support reservations continue looking for Node Labels
[
https://issues.apache.org/jira/browse/YARN-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17144642#comment-17144642
]
Peter Bacsko commented on YARN-9903:
_"Please let me know if you are OK with this change. Without it, my tests were
failing because this code was only looking for reservations for NO_LABEL."_
This is very important. After we discovered and analyzed YARN-10283, we tried a
simple workaround where we return {{LOCALITY_SKIPPED}} if {{needToUnreserve}}
is true and we use node labels (basically what I mentioned in
https://issues.apache.org/jira/browse/YARN-10283?focusedCommentId=17112051&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17112051).
That eliminated the problem of starvation and we no longer saw the "Failed to
accept allocation proposal" messages. Everything looked fine until the same
problem re-appeared.
This time the logged messages are different but apps got stuck in ACCEPTED
regardless. Right now [~tarunparimi] is trying to figure out how this can
happen, but most likely we have to call {{application.findNodeToUnreserve()}}.
I'm not sure if it's really necessary to pass the label string, it depends on
what {{reservedContainers}} contains. If it cannot contain nodes that belong to
other partitions and we use exclusive labels then I think we are fine.
Anyway, I'll review the latest patch later today or tomorrow.
> Support reservations continue looking for Node Labels
> -
>
> Key: YARN-9903
> URL: https://issues.apache.org/jira/browse/YARN-9903
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Tarun Parimi
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-9903.001.patch, YARN-9903.002.patch,
> YARN-9903.003.patch, YARN-9903.004.patch
>
>
> YARN-1769 brought in reservations continue looking feature which improves the
> several resource reservation scenarios. However, it is not handled currently
> when nodes have a label assigned to them. This is useful and in many cases
> necessary even for Node Labels. So we should look to support this for node
> labels also.
> For example, in AbstractCSQueue.java, we have the below TODO.
> {code:java}
> // TODO, now only consider reservation cases when the node has no label
> if (this.reservationsContinueLooking && nodePartition.equals(
> RMNodeLabelsManager.NO_LABEL) && Resources.greaterThan( resourceCalculator,
> clusterResource, resourceCouldBeUnreserved, Resources.none())) {
> {code}
> cc [~sunilg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10277) CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy
[ https://issues.apache.org/jira/browse/YARN-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17145754#comment-17145754 ] Peter Bacsko commented on YARN-10277: - Thanks [~snemeth] for the patch, +1. > CapacityScheduler test TestUserGroupMappingPlacementRule should build proper > hierarchy > -- > > Key: YARN-10277 > URL: https://issues.apache.org/jira/browse/YARN-10277 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-10277.001.patch, YARN-10277.002.patch, > YARN-10277.003.patch > > > Since the CapacityScheduler internal implementation depends more and more on > queue being hierarchical, the test gets really hard to maintain. A lot of > test cases were failing because they used non existing queues, but the older > placement rule solution ignored missing parents, but since the leaf queue > change in CS, we must be able to get a full path for any queue, since all > queues are referenced by their full path. > This test should reflect this and instead of creating and expecting the > existance of fictional queues, it should create a proper queue hierarchy, > with a way to describe it better. > Currently we set up a bunch of mockito "when" statements to simulate the > queue behavior, but this is a hassle to maintain, and easy to miss a few > method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule
Peter Bacsko created YARN-10330:
---
Summary: Add missing test scenarios to
TestUserGroupMappingPlacementRule
Key: YARN-10330
URL: https://issues.apache.org/jira/browse/YARN-10330
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, capacityscheduler, test
Reporter: Peter Bacsko
Assignee: Peter Bacsko
After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned out
that there are at least 8-10 missing scenarios that are not covered.
Since we're planning to enhance mapping rule logic with extra features, it is
crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Description:
After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned out
that there are at least 8-10 missing test scenarios that are not covered.
Since we're planning to enhance mapping rule logic with extra features, it is
crucial to have good coverage so that we can verify backward compatibility.
was:
After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned out
that there are at least 8-10 missing scenarios that are not covered.
Since we're planning to enhance mapping rule logic with extra features, it is
crucial to have good coverage so that we can verify backward compatibility.
> Add missing test scenarios to TestUserGroupMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: YARN-10330-001.patch
> Add missing test scenarios to TestUserGroupMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Summary: Add missing test scenarios to TestUserGroupMappingPlacementRule
and TestAppNameMappingPlacementRule (was: Add missing test scenarios to
TestUserGroupMappingPlacementRule)
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: (was: YARN-10330-002.patch)
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: YARN-10330-002.patch
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: YARN-10330-002.patch
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch, YARN-10330-002.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: YARN-10330-003.patch
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch, YARN-10330-002.patch,
> YARN-10330-003.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: YARN-10330-004.patch
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch, YARN-10330-002.patch,
> YARN-10330-003.patch, YARN-10330-004.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149230#comment-17149230
]
Peter Bacsko commented on YARN-10330:
-
Yes, it's YARN-10329. TestDelegationTokenRenewer is also a long standing issue.
I uploaded patch v4 where I removed a TODO comment and fixed the checkstyle
problem.
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10330-001.patch, YARN-10330-002.patch,
> YARN-10330-003.patch, YARN-10330-004.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10330:
Attachment: YARN-10330-branch-3.3-001.patch
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10330-001.patch, YARN-10330-002.patch,
> YARN-10330-003.patch, YARN-10330-004.patch, YARN-10330-branch-3.3-001.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10370) [Umbrella] Reduce the feature gap between FS Placement Rules and CS Queue Mapping rules
[ https://issues.apache.org/jira/browse/YARN-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10370: Labels: capacity-scheduler capacityscheduler (was: ) > [Umbrella] Reduce the feature gap between FS Placement Rules and CS Queue > Mapping rules > --- > > Key: YARN-10370 > URL: https://issues.apache.org/jira/browse/YARN-10370 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Labels: capacity-scheduler, capacityscheduler > Attachments: MappingRuleEnhancements.pdf, Possible extensions of > mapping rule format in Capacity Scheduler.pdf > > > To continue closing the feature gaps between Fair Scheduler and Capacity > Scheduler to help users migrate between the scheduler more easy, we need to > add some of the Fair Scheduler placement rules to the capacity scheduler's > queue mapping functionality. > With [~snemeth] and [~pbacsko] we've created the following design docs about > the proposed changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10370) [Umbrella] Reduce the feature gap between FS Placement Rules and CS Queue Mapping rules
[ https://issues.apache.org/jira/browse/YARN-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166932#comment-17166932 ] Peter Bacsko commented on YARN-10370: - [~shuzirra] shouldn't the listed JIRA items be subtasks? That's the way we usually do things under an umbrella. I suggest converting them. > [Umbrella] Reduce the feature gap between FS Placement Rules and CS Queue > Mapping rules > --- > > Key: YARN-10370 > URL: https://issues.apache.org/jira/browse/YARN-10370 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Labels: capacity-scheduler, capacityscheduler > Attachments: MappingRuleEnhancements.pdf, Possible extensions of > mapping rule format in Capacity Scheduler.pdf > > > To continue closing the feature gaps between Fair Scheduler and Capacity > Scheduler to help users migrate between the scheduler more easy, we need to > add some of the Fair Scheduler placement rules to the capacity scheduler's > queue mapping functionality. > With [~snemeth] and [~pbacsko] we've created the following design docs about > the proposed changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10371) Create variable context class for CS queue mapping rules
[ https://issues.apache.org/jira/browse/YARN-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10371: Parent: YARN-10370 Issue Type: Sub-task (was: New Feature) > Create variable context class for CS queue mapping rules > > > Key: YARN-10371 > URL: https://issues.apache.org/jira/browse/YARN-10371 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As per the design document attached to the umbrella Jira (YARN-10370), we > need a class which is responsible for storing all the variables usable in > mapping rules. This class shall be able to store mutable / immutable > variables, and replace the known variables in a provided string. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10372) Create MappingRule class to represent each CS mapping rule
[ https://issues.apache.org/jira/browse/YARN-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10372: Parent: YARN-10370 Issue Type: Sub-task (was: New Feature) > Create MappingRule class to represent each CS mapping rule > -- > > Key: YARN-10372 > URL: https://issues.apache.org/jira/browse/YARN-10372 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As per the design document attached to the umbrella Jira (YARN-10370), we > need a class which represents a mapping rule, which can be evaluated, and > determines if the rule applies to the current application placement and > determines the action to be taken. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10374) Create Actions for CS mapping rules
[ https://issues.apache.org/jira/browse/YARN-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10374: Parent: YARN-10370 Issue Type: Sub-task (was: New Feature) > Create Actions for CS mapping rules > --- > > Key: YARN-10374 > URL: https://issues.apache.org/jira/browse/YARN-10374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As per the design document attached to the umbrella Jira (YARN-10370), we > need to create the classes which represent the different actions we can take > when a mapping rule applies to an application placement. There are multiple > actions to be implemented, like place to queue or reject application. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10373) Create Matchers for CS mapping rules
[ https://issues.apache.org/jira/browse/YARN-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10373: Parent: YARN-10370 Issue Type: Sub-task (was: New Feature) > Create Matchers for CS mapping rules > > > Key: YARN-10373 > URL: https://issues.apache.org/jira/browse/YARN-10373 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As per the design document attached to the umbrella Jira (YARN-10370), we > need to create the classes which represent the different ways we can decide > if a mapping rule applies to an application placement. There are multiple > matchers to be implemented, like user matcher, application name matcher, > group matcher, catch all. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10375) CS Mapping rule config parser should return MappingRule objects
[ https://issues.apache.org/jira/browse/YARN-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10375: Parent: YARN-10370 Issue Type: Sub-task (was: New Feature) > CS Mapping rule config parser should return MappingRule objects > --- > > Key: YARN-10375 > URL: https://issues.apache.org/jira/browse/YARN-10375 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As per the design document attached to the umbrella Jira (YARN-10370), we > need to change the current configuration parser to create the new object > structure when parsing the current mapping rule format. This should be 100% > backwards compatible, to make sure upon upgrade all mapping rules will behave > the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10376) Replace the UserGroupMappingPlacementRule and AppNameMappingPlacementRule classes with a generic one
[ https://issues.apache.org/jira/browse/YARN-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10376: Parent: YARN-10370 Issue Type: Sub-task (was: New Feature) > Replace the UserGroupMappingPlacementRule and AppNameMappingPlacementRule > classes with a generic one > > > Key: YARN-10376 > URL: https://issues.apache.org/jira/browse/YARN-10376 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As per the design document attached to the umbrella Jira (YARN-10370), we > need to replace the current PlacementRule classes with a new generic one, > which can utilize the more generic MappingRule approach. This way we can even > mix User and Application placement rules and define a custom order between > all the rules, not only the rules within the same category (user or > applcation placement rules) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10386) Create new JSON schema for Placement Rules
Peter Bacsko created YARN-10386: --- Summary: Create new JSON schema for Placement Rules Key: YARN-10386 URL: https://issues.apache.org/jira/browse/YARN-10386 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler, capacityscheduler Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10387) Implement logic which returns MappingRule objects based on mapping rules
Peter Bacsko created YARN-10387: --- Summary: Implement logic which returns MappingRule objects based on mapping rules Key: YARN-10387 URL: https://issues.apache.org/jira/browse/YARN-10387 Project: Hadoop YARN Issue Type: Sub-task Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171532#comment-17171532
]
Peter Bacsko commented on YARN-10386:
-
Attached v1 version of the schema.
Explanation:
* {{type}}: user / group / application --- pretty much the first column of the
current format.
* {{matches}}: it's either "\*" or a specific group/user -- the second column
of the current format (will use "*" instead of "%user").
* {{policy}}: a fixed set of mapping policies, basically the same as an FS
placement rule.
* {{queue}}: if the policy is {{user}}, {{primaryGroup}}, {{secondaryGroup}},
then this is prepended to the queue string. If the rule is {{defaultQueue}}
then "root.default" will be overridden.
* {{nestedUserRule}}: optional, used only if {{nestedUser}} policy is defined.
Note that unlike to Fair Scheduler, we don't have "outer" create flag, because
it would require very deep changes in Capacity Scheduler. So we only support
"innerCreate" for the "%user" part.
* {{fallbackResult}}: what happens if the placement fails, eg. the target queue
does not exist or cannot be created. Right now in CS, we return "null" for the
placement context which is interpreted as "root.default". In FS, we proceed to
the next rule. Here, we can fine-tune this behavior: {{skip}} is the FS
approach, {{placeDefault}} is the CS approach, {{reject}} is new and
straightforward (submission will be rejected on the client side).
* {{defaultQueue}}: what should be the default queue in case of
{{placeDefault}} fallback result.
* {{create}}: whether the queue should be created or not if it does not exist.
Note that the parent queue of the leaf must be a managed parent in order for
this switch to have effect.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Description: Tasks in this JIRA: # Create new JSON schema # Add Maven plugin which generates Java POJOs based on the schema # Add helper class which essentially does the same as #2 (for dev purposes) > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: MappingRulesDescription.json > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: (was: MappingRulesDescription.json) > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: MappingRulesDescription_v1.json > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175571#comment-17175571 ] Peter Bacsko commented on YARN-10386: - I've been thinking, probably "defaultQueue" should be renamed to something else. There's also a "defaultQueue" policy and users might think that they have to use "defaultQueue" to override "root.default". But it only affects "placeDefault" fallback rule, at least that's my intention right now. So we need to come up with some clever naming. I think "fallbackDefaultQueue" sounds better. > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-001.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176413#comment-17176413 ] Peter Bacsko commented on YARN-10386: - Wow, there are a lot of failures. I'll investigate the enforcer failure, because we added a new dependency to a pom. > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-002.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176572#comment-17176572 ] Peter Bacsko commented on YARN-10386: - [~snemeth] [~adam.antal] [~shuzirra] [~sunilg] - do you guys have any comments? The maven extension was contributed by [~bteke]. > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-003.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-004.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-005.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch, > YARN-10386-005.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-006.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch, > YARN-10386-005.patch, YARN-10386-006.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10408) Extract MockQueueHierarchyBuilder to a separate class
[ https://issues.apache.org/jira/browse/YARN-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185152#comment-17185152 ] Peter Bacsko commented on YARN-10408: - +1, checkstyle is not relevant. > Extract MockQueueHierarchyBuilder to a separate class > - > > Key: YARN-10408 > URL: https://issues.apache.org/jira/browse/YARN-10408 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10408.001.patch > > > We will need the functionality of the MockQueueHierarchyBuilder in multiple > tests so it should be extracted to a standalone class. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10409) Improve MockQueueHierarchyBuilder to detect queue ambiguity
[ https://issues.apache.org/jira/browse/YARN-10409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185153#comment-17185153 ] Peter Bacsko commented on YARN-10409: - [~shuzirra] does this depend on YARN-10408? Wouldn't it make sense to merge the two patches? > Improve MockQueueHierarchyBuilder to detect queue ambiguity > --- > > Key: YARN-10409 > URL: https://issues.apache.org/jira/browse/YARN-10409 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10409.001.patch > > > The current implementation of MockQueueHierarchyBuilder doesn't detect if a > queue short name reference is ambiguous. Since queue manager does have > detection for ambiguity so shall this class to be able to implement test > cases where queue references are ambiguous. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10408) Extract MockQueueHierarchyBuilder to a separate class
[ https://issues.apache.org/jira/browse/YARN-10408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185152#comment-17185152 ] Peter Bacsko edited comment on YARN-10408 at 8/26/20, 12:37 PM: pending +1. Checkstyle is not relevant. See YARN-10409 for a question. was (Author: pbacsko): +1, checkstyle is not relevant. > Extract MockQueueHierarchyBuilder to a separate class > - > > Key: YARN-10408 > URL: https://issues.apache.org/jira/browse/YARN-10408 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10408.001.patch > > > We will need the functionality of the MockQueueHierarchyBuilder in multiple > tests so it should be extracted to a standalone class. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10409) Improve MockQueueHierarchyBuilder to detect queue ambiguity
[ https://issues.apache.org/jira/browse/YARN-10409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185153#comment-17185153 ] Peter Bacsko edited comment on YARN-10409 at 8/26/20, 12:36 PM: [~shuzirra] does this depend on YARN-10408? Wouldn't it make sense to merge the two patches? This looks very small on its own. was (Author: pbacsko): [~shuzirra] does this depend on YARN-10408? Wouldn't it make sense to merge the two patches? > Improve MockQueueHierarchyBuilder to detect queue ambiguity > --- > > Key: YARN-10409 > URL: https://issues.apache.org/jira/browse/YARN-10409 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10409.001.patch > > > The current implementation of MockQueueHierarchyBuilder doesn't detect if a > queue short name reference is ambiguous. Since queue manager does have > detection for ambiguity so shall this class to be able to implement test > cases where queue references are ambiguous. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10371) Create variable context class for CS queue mapping rules
[ https://issues.apache.org/jira/browse/YARN-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185156#comment-17185156 ] Peter Bacsko commented on YARN-10371: - [~shuzirra] could take care of the checkstyle issues? Nothing serious, but let's try to get this in without warnings. > Create variable context class for CS queue mapping rules > > > Key: YARN-10371 > URL: https://issues.apache.org/jira/browse/YARN-10371 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10371.001.patch > > > As per the design document attached to the umbrella Jira (YARN-10370), we > need a class which is responsible for storing all the variables usable in > mapping rules. This class shall be able to store mutable / immutable > variables, and replace the known variables in a provided string. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10371) Create variable context class for CS queue mapping rules
[ https://issues.apache.org/jira/browse/YARN-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185158#comment-17185158 ] Peter Bacsko commented on YARN-10371: - Another thing: this and the remaining changes will introduce new code in "org.apache.hadoop.yarn.server.resourcemanager.placement". There are already FS-related placement stuff there. The new placement engine is not entirely scheduler agnostic, I think it deserves its own package. > Create variable context class for CS queue mapping rules > > > Key: YARN-10371 > URL: https://issues.apache.org/jira/browse/YARN-10371 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10371.001.patch > > > As per the design document attached to the umbrella Jira (YARN-10370), we > need a class which is responsible for storing all the variables usable in > mapping rules. This class shall be able to store mutable / immutable > variables, and replace the known variables in a provided string. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10373) Create Matchers for CS mapping rules
[
https://issues.apache.org/jira/browse/YARN-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185165#comment-17185165
]
Peter Bacsko edited comment on YARN-10373 at 8/26/20, 12:52 PM:
{noformat}
public static MappingRuleMatcher createGroupMatcher(String groupName) {
return new VariableMatcher("%primary_group", groupName);
}
{noformat}
I think it's OK to have a primary group matcher, although we won't need it
right now. What we need is a matcher that matches _all_ groups for a particular
user. For example:
{{g:somegroup:root.users.dev}}
This is a valid group mapping in the current code. "Somegroup" can be either
the primary or supplementary group. Therefore, I suggest creating an
{{AllGroupMatcher}} (or {{AllGroupMatcherForUser}}) in {{MappingRuleMatchers}}
which can determine whether the submitter user belong to the group which is
defined in the rule using the group mapping service.
was (Author: pbacsko):
{noformat}
public static MappingRuleMatcher createGroupMatcher(String groupName) {
return new VariableMatcher("%primary_group", groupName);
}
{noformat}
I think it's OK to have a primary group matcher, although we won't need it
right now. What we need is a matcher that matches _all_ groups. For example:
{{g:somegroup:root.users.dev}}
This is a valid group mapping in the current code. "Somegroup" can be either
the primary or supplementary group. Therefore, I suggest creating an
{{AllGroupMatcher}} (or {{AllGroupMatcherForUser}}) in {{MappingRuleMatchers}}
which can determine whether the submitter user belong to the group which is
defined in the rule using the group mapping service.
> Create Matchers for CS mapping rules
>
>
> Key: YARN-10373
> URL: https://issues.apache.org/jira/browse/YARN-10373
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10373.001.patch
>
>
> As per the design document attached to the umbrella Jira (YARN-10370), we
> need to create the classes which represent the different ways we can decide
> if a mapping rule applies to an application placement. There are multiple
> matchers to be implemented, like user matcher, application name matcher,
> group matcher, catch all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10373) Create Matchers for CS mapping rules
[
https://issues.apache.org/jira/browse/YARN-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185165#comment-17185165
]
Peter Bacsko commented on YARN-10373:
-
{noformat}
public static MappingRuleMatcher createGroupMatcher(String groupName) {
return new VariableMatcher("%primary_group", groupName);
}
{noformat}
I think it's OK to have a primary group matcher, although we won't need it
right now. What we need is a matcher that matches _all_ groups. For example:
{{g:somegroup:root.users.dev}}
This is a valid group mapping in the current code. "Somegroup" can be either
the primary or supplementary group. Therefore, I suggest creating an
{{AllGroupMatcher}} (or {{AllGroupMatcherForUser}}) in {{MappingRuleMatchers}}
which can determine whether the submitter user belong to the group which is
defined in the rule using the group mapping service.
> Create Matchers for CS mapping rules
>
>
> Key: YARN-10373
> URL: https://issues.apache.org/jira/browse/YARN-10373
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10373.001.patch
>
>
> As per the design document attached to the umbrella Jira (YARN-10370), we
> need to create the classes which represent the different ways we can decide
> if a mapping rule applies to an application placement. There are multiple
> matchers to be implemented, like user matcher, application name matcher,
> group matcher, catch all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9136) getNMResourceInfo NodeManager REST API method is not documented
[
https://issues.apache.org/jira/browse/YARN-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185194#comment-17185194
]
Peter Bacsko commented on YARN-9136:
[~snemeth] is right, this doesn't work for FPGA. I think it was an oversight
during development and nobody realized that
{{FpgaResourcePlugin#getNMResourceInfo}} returns null. This should be handled
in a follow-up JIRA.
> getNMResourceInfo NodeManager REST API method is not documented
> ---
>
> Key: YARN-9136
> URL: https://issues.apache.org/jira/browse/YARN-9136
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Hudáky Márton Gyula
>Priority: Major
> Attachments: YARN-9136.001.patch, YARN-9136.002.patch,
> YARN-9136.003.patch
>
>
> I cannot find documentation for the resources endpoint in NMWebServices:
> /ws/v1/node/resources/\{resourcename\}
> I looked in the file NodeManagerRest.md for documentation but haven't found
> any.
> This is supposedly unintentionally not documented:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10374) Create Actions for CS mapping rules
[
https://issues.apache.org/jira/browse/YARN-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185198#comment-17185198
]
Peter Bacsko commented on YARN-10374:
-
OK, I know this patch has dependencies, but still, here's the first round:
1. {{MappingQueuePath}}: this class completely lacks javadoc (at least describe
above the class definition what it's intended to store)
2. {{MappingQueuePath}} nit: {{return parent != null}} instead of {{return null
!= parent}}
3. {{MappingRuleAction}}: add javadoc to describe what this interface represents
4. {{setFallbackReject()}} / {{setFallbackSkip()}} /
{{setFallbackDefaultPlacement()}} – these methods all return
{{MappingRuleActionBase}}. In the implementation I can see it's because they
all return "this", but is chained invocation ever used? It's also weird to see
in {{MappingRuleAction}} that {{MappingRuleActionBase}} is returned, which is
the actual implementation of the interface. I suggest making these methods void.
5. {{VariableUpdateAction}} nit: both {{variableName}} and {{variableName}} can
be final
6. {{MappingRuleResult}}: there are separate methods for returning
{{RESULT_REJECT}}, {{RESULT_SKIP}}, {{RESULT_DEFAULT_PLACEMENT}}. They're also
called {{create*}} but they don't really create anything. I think it's simpler
to just make these constants public.
7. {{MappingRuleValidationContext}}: add a short description about this
interface
8. {{MappingRuleValidationContextImpl}}: avoid star imports like {{import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.*}}
9. {{MappingRuleValidationContextImpl}}: make {{queueManager}} final if possible
10. {{TestMappingRuleActions}} why does this extend
{{junit.framework.TestCase}} ?
11. {{TestMappingRuleValidationContextImpl}}: there's no need for
{{assertTrue(true)}}, just add a comment that the exception is expected
> Create Actions for CS mapping rules
> ---
>
> Key: YARN-10374
> URL: https://issues.apache.org/jira/browse/YARN-10374
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10374.001.patch
>
>
> As per the design document attached to the umbrella Jira (YARN-10370), we
> need to create the classes which represent the different actions we can take
> when a mapping rule applies to an application placement. There are multiple
> actions to be implemented, like place to queue or reject application.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10372) Create MappingRule class to represent each CS mapping rule
[
https://issues.apache.org/jira/browse/YARN-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185206#comment-17185206
]
Peter Bacsko commented on YARN-10372:
-
In this code:
{noformat}
public static MappingRule createLegacyRule(
String type, String source, String path) {
MappingRuleMatcher matcher;
MappingRuleAction action = new
MappingRuleActions.PlaceToQueueAction(path);
switch (type) {
case "u":
matcher = MappingRuleMatchers.createUserMatcher(source);
break;
case "g":
matcher = MappingRuleMatchers.createGroupMatcher(source);
break;
default:
matcher =
MappingRuleMatchers.createApplicationNameMatcher(source);
}
{noformat}
I would definitely add a separate case for "a" and then fail in in the
{{default}} branch with {{IllegalArgumentException}}. This adds an extra level
of safety net.
> Create MappingRule class to represent each CS mapping rule
> --
>
> Key: YARN-10372
> URL: https://issues.apache.org/jira/browse/YARN-10372
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10372.001.patch
>
>
> As per the design document attached to the umbrella Jira (YARN-10370), we
> need a class which represents a mapping rule, which can be evaluated, and
> determines if the rule applies to the current application placement and
> determines the action to be taken.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10372) Create MappingRule class to represent each CS mapping rule
[
https://issues.apache.org/jira/browse/YARN-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185206#comment-17185206
]
Peter Bacsko edited comment on YARN-10372 at 8/26/20, 2:12 PM:
---
In this code:
{noformat}
public static MappingRule createLegacyRule(
String type, String source, String path) {
MappingRuleMatcher matcher;
MappingRuleAction action = new
MappingRuleActions.PlaceToQueueAction(path);
switch (type) {
case "u":
matcher = MappingRuleMatchers.createUserMatcher(source);
break;
case "g":
matcher = MappingRuleMatchers.createGroupMatcher(source);
break;
default:
matcher =
MappingRuleMatchers.createApplicationNameMatcher(source);
}
{noformat}
I would definitely add a separate case for "a" and then fail in in the
{{default}} branch with {{IllegalArgumentException}}. This adds an extra level
of safety net.
It's perhaps even better to extract "u", "g" and "a" into final String
constants instead of using them as literals (this also applies to YARN-10375).
was (Author: pbacsko):
In this code:
{noformat}
public static MappingRule createLegacyRule(
String type, String source, String path) {
MappingRuleMatcher matcher;
MappingRuleAction action = new
MappingRuleActions.PlaceToQueueAction(path);
switch (type) {
case "u":
matcher = MappingRuleMatchers.createUserMatcher(source);
break;
case "g":
matcher = MappingRuleMatchers.createGroupMatcher(source);
break;
default:
matcher =
MappingRuleMatchers.createApplicationNameMatcher(source);
}
{noformat}
I would definitely add a separate case for "a" and then fail in in the
{{default}} branch with {{IllegalArgumentException}}. This adds an extra level
of safety net.
> Create MappingRule class to represent each CS mapping rule
> --
>
> Key: YARN-10372
> URL: https://issues.apache.org/jira/browse/YARN-10372
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10372.001.patch
>
>
> As per the design document attached to the umbrella Jira (YARN-10370), we
> need a class which represents a mapping rule, which can be evaluated, and
> determines if the rule applies to the current application placement and
> determines the action to be taken.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185738#comment-17185738 ] Peter Bacsko commented on YARN-10386: - Ok, the build what matters is https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/96/. ASF license warnings and the unit test failures can be ignored. Tests were executed for the whole project, but several times the forked process ran into resource limits. > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch, > YARN-10386-005.patch, YARN-10386-006.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185770#comment-17185770
]
Peter Bacsko commented on YARN-10386:
-
[~adam.antal] the entire project is findbugs/style checked and compiled many
times over due to the modification in {{hadoop-project/pom.xml}}. That makes
the build very slow and eventually it runs out of resources.
Anyway I'll make the changes.
Any thoughts about the JSON schema itself?
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185799#comment-17185799
]
Peter Bacsko commented on YARN-10386:
-
"Why is the "defaultQueue" property a separate option not a policy?"
That's what I explained, it's confusing. That's intended to set the fallback
default queue, in case a fallback occurs and the the {{fallbackResult}} is
{{placeToDefault}}.
"I think we are overusing the defaultQueue in the schema, we should use other
names like, changeDefaultQueue, placeToDefaultQueue, fallbackToDefaultQueue or
something along these lines to make sure it is clear what it will do."
Yep, that's what my first command is about :)
"Nested" is already a policy itself and the "nestedUserRule" is only
interpreted if this particular policy is used.
The reason for a separate nested policy is to prevent users from all kinds of
crazy configurations. Yes, the engine itself can happily resolve stuff like
"root.%default.%primary_group", but I don't think these kind of configurations
should be allowed. So the schema itself takes care of the allowed combinations.
Regarding changing the default, I think it's an overkill to have a separate
changeDefault policy. It's certainly how it works under the hood as an action,
but I don't see too much benefit of exposing this.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185799#comment-17185799
]
Peter Bacsko edited comment on YARN-10386 at 8/27/20, 11:54 AM:
"Why is the "defaultQueue" property a separate option not a policy?"
That's what I explained, it's confusing. That's intended to set the fallback
default queue, in case a fallback occurs and the {{fallbackResult}} is
{{placeToDefault}}.
"I think we are overusing the defaultQueue in the schema, we should use other
names like, changeDefaultQueue, placeToDefaultQueue, fallbackToDefaultQueue or
something along these lines to make sure it is clear what it will do."
Yep, that's what my first command is about :)
"Nested" is already a policy itself and the "nestedUserRule" is only
interpreted if this particular policy is used.
The reason for a separate nested policy is to prevent users from all kinds of
crazy configurations. Yes, the engine itself can happily resolve stuff like
"root.%default.%primary_group", but I don't think these kind of configurations
should be allowed. So the schema itself takes care of the allowed combinations.
Regarding changing the default, I think it's an overkill to have a separate
changeDefault policy. It's certainly how it works under the hood as an action,
but I don't see too much benefit of exposing this.
was (Author: pbacsko):
"Why is the "defaultQueue" property a separate option not a policy?"
That's what I explained, it's confusing. That's intended to set the fallback
default queue, in case a fallback occurs and the the {{fallbackResult}} is
{{placeToDefault}}.
"I think we are overusing the defaultQueue in the schema, we should use other
names like, changeDefaultQueue, placeToDefaultQueue, fallbackToDefaultQueue or
something along these lines to make sure it is clear what it will do."
Yep, that's what my first command is about :)
"Nested" is already a policy itself and the "nestedUserRule" is only
interpreted if this particular policy is used.
The reason for a separate nested policy is to prevent users from all kinds of
crazy configurations. Yes, the engine itself can happily resolve stuff like
"root.%default.%primary_group", but I don't think these kind of configurations
should be allowed. So the schema itself takes care of the allowed combinations.
Regarding changing the default, I think it's an overkill to have a separate
changeDefault policy. It's certainly how it works under the hood as an action,
but I don't see too much benefit of exposing this.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185899#comment-17185899
]
Peter Bacsko commented on YARN-10386:
-
Ok, had a discussion with [~shuzirra]. Now I can better understand what he
meant above.
The following changes will be introduced:
1. "nestedUserRule" will be removed, because it's used only in case of
"nestedUserRule" policy.
2. Add two new policies "secondaryGroupUser" and "primaryGroupUser".
3. Remove "defaultQueue" setting. The default queue will be set with a
different rule.
4. Also introduce "setDefaultQueue" policy. This will change the default queue
until it's set to something else.
5. The naming of "queue" is misleading, let's change that to sth like
"parentQueue" or "parentQueuePath".
6. Custom policy: we can keep this and add a "custom" field perhaps. This can
contain any random definition like
{{root.user.%primary_group.%secondary_group}} and it will not be rejected. The
tool won't validate whether the string makes any sense or not.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-007.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch, > YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10386) Create new JSON schema for Placement Rules
[ https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10386: Attachment: YARN-10386-008.patch > Create new JSON schema for Placement Rules > -- > > Key: YARN-10386 > URL: https://issues.apache.org/jira/browse/YARN-10386 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch, > YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch, > YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch, > YARN-10386-008.patch > > > Tasks in this JIRA: > # Create new JSON schema > # Add Maven plugin which generates Java POJOs based on the schema > # Add helper class which essentially does the same as #2 (for dev purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186087#comment-17186087
]
Peter Bacsko commented on YARN-10386:
-
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "reject"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch,
> YARN-10386-008.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186087#comment-17186087
]
Peter Bacsko edited comment on YARN-10386 at 8/27/20, 8:11 PM:
---
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "placeDefault"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
was (Author: pbacsko):
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "reject"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch,
> YARN-10386-008.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186087#comment-17186087
]
Peter Bacsko edited comment on YARN-10386 at 8/27/20, 8:15 PM:
---
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "placeDefault"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
This is also an alternative for {{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "custom",
"customPlacement": "root.users"
}
}
{noformat}
was (Author: pbacsko):
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "placeDefault"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch,
> YARN-10386-008.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10386) Create new JSON schema for Placement Rules
[
https://issues.apache.org/jira/browse/YARN-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186087#comment-17186087
]
Peter Bacsko edited comment on YARN-10386 at 8/27/20, 8:18 PM:
---
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "placeDefault"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
There is also an alternative for {{u:%user:root.users}} (in fact, all rules
could be rewritten with "custom", which is kind of a non-restriced way of
specifying a target queue with arbitrary combination of placeholder variables).
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "custom",
"customPlacement": "root.users"
}
}
{noformat}
was (Author: pbacsko):
Some examples:
{{u:%user:%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "user",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.dev.%primary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "primaryGroupUser",
"parentQueue": "root.dev",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:%secondary_group.%user}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "secondaryGroupUser",
"fallbackResult": "placeDefault"
}
}
{noformat}
{{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "setDefaultQueue",
"value": "root.users",
"fallbackResult": "skip"
},
{
"type": "user",
"matches": "*",
"policy": "defaultQueue",
"fallbackResult": "placeDefault"
}
}
{noformat}
As we can see, we need two rules for a hard-coded path like "root.users". This
might look weird, but this very similar to FS, where the rule is also called
{{}} with an overridden default queue. So in theory it's the same.
This is also an alternative for {{u:%user:root.users}}
{noformat}
"rules":
{
{
"type": "user",
"matches": "*",
"policy": "custom",
"customPlacement": "root.users"
}
}
{noformat}
> Create new JSON schema for Placement Rules
> --
>
> Key: YARN-10386
> URL: https://issues.apache.org/jira/browse/YARN-10386
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MappingRulesDescription_v1.json, YARN-10386-001.patch,
> YARN-10386-002.patch, YARN-10386-003.patch, YARN-10386-004.patch,
> YARN-10386-005.patch, YARN-10386-006.patch, YARN-10386-007.patch,
> YARN-10386-008.patch
>
>
> Tasks in this JIRA:
> # Create new JSON schema
> # Add Maven plugin which generates Java POJOs based on the schema
> # Add helper class which essentially does the same as #2 (for dev purposes)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-ma
[jira] [Created] (YARN-10413) Change fs2cs to generate mapping rules in the new format
Peter Bacsko created YARN-10413: --- Summary: Change fs2cs to generate mapping rules in the new format Key: YARN-10413 URL: https://issues.apache.org/jira/browse/YARN-10413 Project: Hadoop YARN Issue Type: Sub-task Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10413) Change fs2cs to generate mapping rules in the new format
[
https://issues.apache.org/jira/browse/YARN-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10413:
Description:
Currently, the converter tool {{fs2cs}} can convert placement rules to mapping
rules, but the differences are too big.
It should be modified to generate placement rules to the new engine and output
it to a separate JSON file.
> Change fs2cs to generate mapping rules in the new format
>
>
> Key: YARN-10413
> URL: https://issues.apache.org/jira/browse/YARN-10413
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> Currently, the converter tool {{fs2cs}} can convert placement rules to
> mapping rules, but the differences are too big.
> It should be modified to generate placement rules to the new engine and
> output it to a separate JSON file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
