[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-16 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136648#comment-17136648
 ] 

Peter Bacsko edited comment on YARN-9930 at 6/16/20, 1:24 PM:
--

[~adam.antal] thanks for the comment. I suggest talking about it IRL/video 
meeting, because that would be more effective & then I'll summarize my answer 
later.


was (Author: pbacsko):
[~adam.antal] thanks for the comment. I suggest talking about IRL/video 
meeting, because that would be more effective & then I'll summarize my answer 
later.

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9930-001.patch, YARN-9930-002.patch, 
> YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, 
> YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, 
> YARN-9930-POC05.patch, screenshot-1.png
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-16 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136648#comment-17136648
 ] 

Peter Bacsko edited comment on YARN-9930 at 6/16/20, 1:17 PM:
--

[~adam.antal] thanks for the comment. I suggest talking about IRL/video 
meeting, because that would be more effective & then I'll summarize my answer 
later.


was (Author: pbacsko):
[~adam.antal] thanks for the comment. I suggest talking about IRL, because that 
would be more effective & then I'll summarize my answer later.

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9930-001.patch, YARN-9930-002.patch, 
> YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-POC01.patch, 
> YARN-9930-POC02.patch, YARN-9930-POC03.patch, YARN-9930-POC04.patch, 
> YARN-9930-POC05.patch, screenshot-1.png
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-05 Thread Szilard Nemeth (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126645#comment-17126645
 ] 

Szilard Nemeth edited comment on YARN-9930 at 6/5/20, 10:36 AM:


Hi [~pbacsko],

1. Overall feedback for the design: 
I like the approach so from my POV, the design is fine.
I think it's very straightforward that you created a separate class of 
CSMaxRunningAppsEnforcer.
What about keeping it for this round, and after this jira is merged, work on a 
followup jira that refactors this and make an abstract parent class for 
CSMaxRunningAppsEnforcer MaxRunningAppsEnforcer?

You also mentioned about a separate property named as 
'yarn.scheduler.capacity.maxrunningapps.reject' but I don't see it in the 
latest patch.



2. Review comments:
AbstractCSQueue: Typo in name 'queueMaxRunninApps'.


was (Author: snemeth):
Hi [~pbacsko],

1. Overall feedback for the design: 
I like the approach so from my POV, the design is fine.
I think it's very straightforward that you created a separate class of 
CSMaxRunningAppsEnforcer.
What about keeping it for this round, and after this jira is merged, work on a 
followup jira that refactors this and make an abstract parent class for 
CSMaxRunningAppsEnforcer MaxRunningAppsEnforcer?


2. Review comments:
AbstractCSQueue: Typo in name 'queueMaxRunninApps'.

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, 
> YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-03 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124932#comment-17124932
 ] 

Peter Bacsko edited comment on YARN-9930 at 6/3/20, 4:02 PM:
-

Attached POC v4 with a new unit test which verifies the new functionality. 
Still a POC because there are failing UTs.

TODO
* fix failing UTs (most likely  mocking should be fine tuned)
* fix checkstyle
* more tests for the new feature (eg. user limit exceeded)
* check the visibility of new methods (package private/public)
* check types/casts (the patch has probably too many of them)
* ensure proper naming
* add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from 
TestMaxRunningAppsEnforcer)


was (Author: pbacsko):
Attached POC v4 with a new unit test which verifies the new functionality. 
Still a POC because there are failing UTs.

TODO
* fix failing UTs (most likely  mocking should be fine tuned)
* fix checkstyle
* more tests for the new feature (eg. user limit exceeded)
* check  visibility of new methods
* add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from 
TestMaxRunningAppsEnforcer)

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, 
> YARN-9930-POC03.patch, YARN-9930-POC04.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-03 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124947#comment-17124947
 ] 

Peter Bacsko edited comment on YARN-9930 at 6/3/20, 1:20 PM:
-

[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about 
the POC?

Note that it actually does NOT interfere with the existing maxApps settings 
because those are checked when the application is submitted. So the rejection 
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The 
maxParallelApps check comes later, when we submit the application attempt to 
the leaf queue.

Also, to avoid confusion I decided to call the new setting "maxParallelApps" to 
avoid confusion (it's called "maxRunningApps" in FS).


was (Author: pbacsko):
[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about 
the POC?

Note that it actually does NOT interfere with the existing maxApps settings 
because those is checked when the application is submitted. So the rejection 
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The 
maxParallelApps check comes later, when we submit the application attempt to 
the leaf queue.

Also, to avoid confusion I decided to call the new setting "maxParallelApps" to 
avoid confusion (it's called "maxRunningApps" in FS).

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, 
> YARN-9930-POC03.patch, YARN-9930-POC04.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-03 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124947#comment-17124947
 ] 

Peter Bacsko edited comment on YARN-9930 at 6/3/20, 1:20 PM:
-

[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about 
the POC?

Note that it actually does NOT interfere with the existing maxApps settings 
because those are checked when the application is submitted. So the rejection 
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The 
maxParallelApps check comes later, when we submit the application attempt to 
the leaf queue.

Also, to avoid confusion I decided to call the new setting "maxParallelApps" 
(it's called "maxRunningApps" in FS).


was (Author: pbacsko):
[~epayne] [~cane] [~snemeth] [~sunilg] could you guys share your opinion about 
the POC?

Note that it actually does NOT interfere with the existing maxApps settings 
because those are checked when the application is submitted. So the rejection 
occurs immediately (see {{LeafQueue.validateSubmitApplication()}}). The 
maxParallelApps check comes later, when we submit the application attempt to 
the leaf queue.

Also, to avoid confusion I decided to call the new setting "maxParallelApps" to 
avoid confusion (it's called "maxRunningApps" in FS).

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, 
> YARN-9930-POC03.patch, YARN-9930-POC04.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-06-03 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124932#comment-17124932
 ] 

Peter Bacsko edited comment on YARN-9930 at 6/3/20, 1:02 PM:
-

Attached POC v4 with a new unit test which verifies the new functionality. 
Still a POC because there are failing UTs.

TODO
* fix failing UTs (most likely  mocking should be fine tuned)
* fix checkstyle
* more tests for the new feature (eg. user limit exceeded)
* check  visibility of new methods
* add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from 
TestMaxRunningAppsEnforcer)


was (Author: pbacsko):
Attached POC v4 with a new unit tests which verifies the new functionality. 
Still a POC because there are failing UTs.

TODO
* fix failing UTs (most likely  mocking should be fine tuned)
* fix checkstyle
* more tests for the new feature (eg. user limit exceeded)
* check  visibility of new methods
* add tests for CSMaxRunningAppsEnforcer (likely a copy-paste + edits from 
TestMaxRunningAppsEnforcer)

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch, YARN-9930-POC02.patch, 
> YARN-9930-POC03.patch, YARN-9930-POC04.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-05-28 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118899#comment-17118899
 ] 

Peter Bacsko edited comment on YARN-9930 at 5/28/20, 5:09 PM:
--

Created a POC based on the solution exists in FS. No tests yet at all.

Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so 
that a single class could serve both FS and CS but it required way too many 
changes. The class is heavily tied to FS. So I created 
{{CSMaxRunningAppsEnforcer}}.



was (Author: pbacsko):
Created a POC based on the solution exists in FS. No tests yet at all.

Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so 
that a single class could serve both FS and CS but the it required way too many 
changes. The class is heavily tied to FS. So I created 
{{CSMaxRunningAppsEnforcer}}.


> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2020-05-28 Thread Peter Bacsko (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118899#comment-17118899
 ] 

Peter Bacsko edited comment on YARN-9930 at 5/28/20, 5:03 PM:
--

Created a POC based on the solution exists in FS. No tests yet at all.

Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so 
that a single class could serve both FS and CS but the it required way too many 
changes. The class is heavily tied to FS. So I created 
{{CSMaxRunningAppsEnforcer}}.



was (Author: pbacsko):
Created a POC based on the solution exists in FS. No tests yet at all.

Note that I copy-pasted {{MaxRunningAppsEnforcer}}. I started to refactor it so 
that a single class could serve both FS and CS but it just got too big. The 
class is heavily tied to FS. So I created {{CSMaxRunningAppsEnforcer}}.


> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
> Attachments: YARN-9930-POC01.patch
>
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-9930) Support max running app logic for CapacityScheduler

2019-11-13 Thread zhoukang (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973280#comment-16973280
 ] 

zhoukang edited comment on YARN-9930 at 11/13/19 11:55 AM:
---

[~pbacsko]Thanks. The background  is that in our production cluster we want to 
upgrade hadoop version to 3.x, and we used FairScheduler before.
Now we want to use CapacityScheduler  in new version 3.x.
If we want to migrate from FS to CS , this behavior will be confused to users.

[~epayne][~pbacsko]I agree with the point.Add a config like
bq. "yarn.scheduler.capacity.maxrunningapps.reject"


was (Author: cane):
[~pbacsko]Thanks. The background  is that in our production cluster we want to 
upgrade hadoop version to 3.x, and we used FairScheduler before.
Now we want to use CapacityScheduler  in new version 3.x.
If we want to migrate from FS to CS , this behavior will be confused.

[~epayne][~pbacsko]I agree with the point.Add a config like
bq. "yarn.scheduler.capacity.maxrunningapps.reject"

> Support max running app logic for CapacityScheduler
> ---
>
> Key: YARN-9930
> URL: https://issues.apache.org/jira/browse/YARN-9930
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, capacityscheduler
>Affects Versions: 3.1.0, 3.1.1
>Reporter: zhoukang
>Assignee: zhoukang
>Priority: Major
>
> In FairScheduler, there has limitation for max running which will let 
> application pending.
> But in CapacityScheduler there has no feature like max running app.Only got 
> max app,and jobs will be rejected directly on client.
> This jira i want to implement this semantic for CapacityScheduler.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org