[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264855#comment-17264855
]
Gergely Pollak edited comment on YARN-10506 at 1/14/21, 3:46 PM:
-----------------------------------------------------------------
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently than CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
was (Author: shuzirra):
Hi [~gandras] [~wangda], [~zhuqi]
*1) How we deal with "create" flag of ApplicationPlacementContext?* **
I don't think we need this flag at all. The CapacityScheduler has the
information where can be dynamic queues created, it is supplied via the
configuration option {{queue-path.auto-queue-creation-v2.enabled}}
and
{\{queue-path.auto-queue-creation.enabled }}
So at any point the CS will be aware if a supplied queue path is creatable.
Mapping rules will return a path based on their configuration, but I don't
think the placement rule should be overriding the CS configuration. This can
cause some undesired issues, and we really don't get anything with it. It will
be a little more inconvenient for the user since they have to configure the
mapping rules AND the CS queues properly, but it doesn't look like a big deal.
On the other hand there are multiple places in the code which check if a queue
can be created and all these places use the queue hierarchy to determine
whether a queue is a managed parent, or not, however if the Mapping Rules are
allowed to circumvent this, it might lead to unforeseen inconsistencies. (As a
matter of fact the CSMappingPlacementRule currently uses the information from
CS to determine if a queue can be created or not, so it wouldn't set any of
these flags differently then CS is configured anyway)
So I would say let's omit the creation flags from the
ApoplicationPlacementContext, let the CSMappingPlacementRule rely on the
information provided by the CS about queue creation eligibility, and let it
pass back a "regular" ApplicationPlacementContext with the desired queue path,
then the CS will be able to create it if the configuration allows. This is a
much easier and stable solution, while we don't really put extra administration
overhead to the user, and also can be easily extended in the future.
As I see circumventing protections are never a good idea even if we are talking
about internal interfaces, and these flags would exactly do that, they would
force the CS to create queues which are against it's configuration.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
