[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136648#comment-17136648 ] Peter Bacsko edited comment on YARN-9930 at 6/16/20, 1:24 PM: -- [~adam.antal] thanks for the comment. I suggest talking about it IRL/video meeting, because that would be more effective & then I'll summarize my answer later. was (Author: pbacsko): [~adam.antal] thanks for the comment. I suggest talking about IRL/video meeting, because that would be more effective & then I'll summarize my answer later. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136648#comment-17136648 ] Peter Bacsko edited comment on YARN-9930 at 6/16/20, 1:17 PM: -- [~adam.antal] thanks for the comment. I suggest talking about IRL/video meeting, because that would be more effective & then I'll summarize my answer later. was (Author: pbacsko): [~adam.antal] thanks for the comment. I suggest talking about IRL, because that would be more effective & then I'll summarize my answer later. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, > YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, > YARN-9930-POC05.patch, screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126645#comment-17126645 ] Szilard Nemeth edited comment on YARN-9930 at 6/5/20, 10:36 AM: Hi [~pbacsko], 1. Overall feedback for the design: I like the approach so from my POV, the design is fine. I think it's very straightforward that you created a separate class of CSMaxRunningAppsEnforcer. What about keeping it for this round, and after this jira is merged, work on a followup jira that refactors this and make an abstract parent class for CSMaxRunningAppsEnforcer MaxRunningAppsEnforcer? You also mentioned about a separate property named as 'yarn.scheduler.capacity.maxrunningapps.reject' but I don't see it in the latest patch. 2. Review comments: AbstractCSQueue: Typo in name 'queueMaxRunninApps'. was (Author: snemeth): Hi [~pbacsko], 1. Overall feedback for the design: I like the approach so from my POV, the design is fine. I think it's very straightforward that you created a separate class of CSMaxRunningAppsEnforcer. What about keeping it for this round, and after this jira is merged, work on a followup jira that refactors this and make an abstract parent class for CSMaxRunningAppsEnforcer MaxRunningAppsEnforcer? 2. Review comments: AbstractCSQueue: Typo in name 'queueMaxRunninApps'. > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124932#comment-17124932 ] Peter Bacsko edited comment on YARN-9930 at 6/3/20, 4:02 PM: - Attached POC v4 with a new unit test which verifies the new functionality. Still a POC because there are failing UTs. TODO * fix failing UTs (most likely mocking should be fine tuned) * fix checkstyle * more tests for the new feature (eg. user limit exceeded) * check the visibility of new methods (package private/public) * check types/casts (the patch has probably too many of them) * ensure proper naming * add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from TestMaxRunningAppsEnforcer) was (Author: pbacsko): Attached POC v4 with a new unit test which verifies the new functionality. Still a POC because there are failing UTs. TODO * fix failing UTs (most likely mocking should be fine tuned) * fix checkstyle * more tests for the new feature (eg. user limit exceeded) * check visibility of new methods * add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from TestMaxRunningAppsEnforcer) > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124947#comment-17124947
]
Peter Bacsko edited comment on YARN-9930 at 6/3/20, 1:20 PM:
-
[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about
the POC?
Note that it actually does NOT interfere with the existing maxApps settings
because those are checked when the application is submitted. So the rejection
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The
maxParallelApps check comes later, when we submit the application attempt to
the leaf queue.
Also, to avoid confusion I decided to call the new setting "maxParallelApps" to
avoid confusion (it's called "maxRunningApps" in FS).
was (Author: pbacsko):
[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about
the POC?
Note that it actually does NOT interfere with the existing maxApps settings
because those is checked when the application is submitted. So the rejection
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The
maxParallelApps check comes later, when we submit the application attempt to
the leaf queue.
Also, to avoid confusion I decided to call the new setting "maxParallelApps" to
avoid confusion (it's called "maxRunningApps" in FS).
> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch,
> YARN-9930-POC03.patch, YARN-9930-POC04.patch
>
>
> In FairScheduler, there has limitation for max running which will let
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124947#comment-17124947
]
Peter Bacsko edited comment on YARN-9930 at 6/3/20, 1:20 PM:
-
[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about
the POC?
Note that it actually does NOT interfere with the existing maxApps settings
because those are checked when the application is submitted. So the rejection
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The
maxParallelApps check comes later, when we submit the application attempt to
the leaf queue.
Also, to avoid confusion I decided to call the new setting "maxParallelApps"
(it's called "maxRunningApps" in FS).
was (Author: pbacsko):
[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about
the POC?
Note that it actually does NOT interfere with the existing maxApps settings
because those are checked when the application is submitted. So the rejection
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The
maxParallelApps check comes later, when we submit the application attempt to
the leaf queue.
Also, to avoid confusion I decided to call the new setting "maxParallelApps" to
avoid confusion (it's called "maxRunningApps" in FS).
> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch,
> YARN-9930-POC03.patch, YARN-9930-POC04.patch
>
>
> In FairScheduler, there has limitation for max running which will let
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124932#comment-17124932 ] Peter Bacsko edited comment on YARN-9930 at 6/3/20, 1:02 PM: - Attached POC v4 with a new unit test which verifies the new functionality. Still a POC because there are failing UTs. TODO * fix failing UTs (most likely mocking should be fine tuned) * fix checkstyle * more tests for the new feature (eg. user limit exceeded) * check visibility of new methods * add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from TestMaxRunningAppsEnforcer) was (Author: pbacsko): Attached POC v4 with a new unit tests which verifies the new functionality. Still a POC because there are failing UTs. TODO * fix failing UTs (most likely mocking should be fine tuned) * fix checkstyle * more tests for the new feature (eg. user limit exceeded) * check visibility of new methods * add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from TestMaxRunningAppsEnforcer) > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118899#comment-17118899
]
Peter Bacsko edited comment on YARN-9930 at 5/28/20, 5:09 PM:
--
Created a POC based on the solution exists in FS. No tests yet at all.
Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so
that a single class could serve both FS and CS but it required way too many
changes. The class is heavily tied to FS. So I created
{{CSMaxRunningAppsEnforcer}}.
was (Author: pbacsko):
Created a POC based on the solution exists in FS. No tests yet at all.
Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so
that a single class could serve both FS and CS but the it required way too many
changes. The class is heavily tied to FS. So I created
{{CSMaxRunningAppsEnforcer}}.
> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch
>
>
> In FairScheduler, there has limitation for max running which will let
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118899#comment-17118899
]
Peter Bacsko edited comment on YARN-9930 at 5/28/20, 5:03 PM:
--
Created a POC based on the solution exists in FS. No tests yet at all.
Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so
that a single class could serve both FS and CS but the it required way too many
changes. The class is heavily tied to FS. So I created
{{CSMaxRunningAppsEnforcer}}.
was (Author: pbacsko):
Created a POC based on the solution exists in FS. No tests yet at all.
Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so
that a single class could serve both FS and CS but it just got too big. The
class is heavily tied to FS. So I created {{CSMaxRunningAppsEnforcer}}.
> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch
>
>
> In FairScheduler, there has limitation for max running which will let
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973280#comment-16973280 ] zhoukang edited comment on YARN-9930 at 11/13/19 11:55 AM: --- [~pbacsko]Thanks. The background is that in our production cluster we want to upgrade hadoop version to 3.x, and we used FairScheduler before. Now we want to use CapacityScheduler in new version 3.x. If we want to migrate from FS to CS , this behavior will be confused to users. [~epayne][~pbacsko]I agree with the point.Add a config like bq. "yarn.scheduler.capacity.maxrunningapps.reject" was (Author: cane): [~pbacsko]Thanks. The background is that in our production cluster we want to upgrade hadoop version to 3.x, and we used FairScheduler before. Now we want to use CapacityScheduler in new version 3.x. If we want to migrate from FS to CS , this behavior will be confused. [~epayne][~pbacsko]I agree with the point.Add a config like bq. "yarn.scheduler.capacity.maxrunningapps.reject" > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
