This is an automated email from the ASF dual-hosted git repository.
benjobs pushed a commit to branch dev
in repository
https://gitbox.apache.org/repos/asf/incubator-streampark-website.git
The following commit(s) were added to refs/heads/dev by this push:
new 4f29e651 [Issue-2470][Feature][subtask] Add the documentation page
about yarn queue management. (#196)
4f29e651 is described below
commit 4f29e651d78e47ae64ae798b38770ffedc36b7aa
Author: Roc Marshal <[email protected]>
AuthorDate: Thu Apr 13 21:58:18 2023 +0800
[Issue-2470][Feature][subtask] Add the documentation page about yarn queue
management. (#196)
---
docs/user-guide/8-YarnQueueManagement.md | 107 +++++++++++++++++++++
.../available_queues_when_creating_application.png | Bin 0 -> 381753 bytes
.../available_queues_when_creating_cluster.png | Bin 0 -> 453549 bytes
static/doc/image/yarn-queue/existed_queues.png | Bin 0 -> 411339 bytes
static/doc/image/yarn-queue/flow_to_create.png | Bin 0 -> 384270 bytes
.../yarn-queue/flow_to_type_in_pure_queue.png | Bin 0 -> 392799 bytes
.../flow_to_type_in_pure_queue_labels.png | Bin 0 -> 395438 bytes
.../use_yarn_app_mode_to_create_application.png | Bin 0 -> 451721 bytes
...use_yarn_session_mode_to_create_application.png | Bin 0 -> 344748 bytes
9 files changed, 107 insertions(+)
diff --git a/docs/user-guide/8-YarnQueueManagement.md
b/docs/user-guide/8-YarnQueueManagement.md
new file mode 100644
index 00000000..5dcf6bd5
--- /dev/null
+++ b/docs/user-guide/8-YarnQueueManagement.md
@@ -0,0 +1,107 @@
+---
+id: 'YarnQueueManagement'
+title: 'Yarn Queue Management'
+sidebar_position: 8
+---
+
+## Background
+In the actual production environment for the `Yarn Deployment Mode`,
+users often need to type in `queue` or `queue`&`labels` to specify
+the configuration for the `yarn-application` mode flink applications
+or the `yarn-session` mode flink clusters.
+During this process, manual input by users may cause errors,
+which may result in specified non-existed queues or the `flink
application`/`flink cluster`
+being submitted to the wrong queues.
+
+If the yarn cluster doesn't include users specified queues,
+the deployment process of `flink applications`/`flink clusters`
+will be time-consuming and accompanied by poor user experience.
+If a task is submitted to an incorrect queue due to the error-input,
+it's likely to affect the stability of yarn applications on the queue and the
abuse of the queue resource.
+
+So StreamPark introduced the queue management feature to ensure that a set of
added queues are shared within the same team,
+that is, ensure that queues resource is isolated within the scope of the team.
It can generate the following benefits:
+- When deploying Flink `yarn-application` applications or Flink `yarn-session`
clusters,
+it could set quickly and accurately yarn queue(`yarn.application.queue`) &
labels(`yarn.application.node-label`).
+- Not only does it ensure the correctness of queue & labels input,
+but it also shortens the time consumption of incorrect queues leading to
applications failures,
+that is, to prompt users as early as possible whether the queue setting
results are correct, improving interaction efficiency.
+- Effectively preventing the abuse of queues resource.
+
+
+## How to create yarn queues
+- Premise
+You must keep a admin user account. In the case, The admin user `admin` was
used to login the `default` team of the system.
+- Locate the `Add New` button of the `Yarn Queue` page as followed steps and
the fig.
+ - Use the admin user account to login the system with a target team.
+ - Click `Settings`.
+ - Click `Yarn Queue`.
+ - Click `Add New`.
+ <img src="/doc/image/yarn-queue/flow_to_create.png"/><br></br><br></br>
+
+- Type in the queue fields.
+ - Create a pure queue.
+ - Type in the required yarn queue name
+ - Type in the optional description of the yarn queue if needed.
+ - Click `OK`.
+ <img
src="/doc/image/yarn-queue/flow_to_type_in_pure_queue.png"/><br></br><br></br>
+ - Create a queue with labels.
+ - Type in the required yarn queue-label name.
+ Queue label, eg. `queue1` represents queue name, `queue@label1,label2`
represents that queue name is `queue1` and labels of the queue are `label1` &
`label2`.
+ - Type in the optional description of the yarn queue if needed.
+ - Click `OK`.
+ <img
src="/doc/image/yarn-queue/flow_to_type_in_pure_queue_labels.png"/><br></br><br></br>
+- View the existed yarn queue(-labels)s.
+ <img src="/doc/image/yarn-queue/existed_queues.png"/><br></br><br></br>
+
+
+## How to use available queues
+
+- Create a flink cluster with a specified yarn queue.
+
+ <img
src="/doc/image/yarn-queue/available_queues_when_creating_cluster.png"/><br></br><br></br>
+
+- Create a yarn-application mode flink application with a specified yarn queue
+
+ <img
src="/doc/image/yarn-queue/use_yarn_app_mode_to_create_application.png"/><br></br><br></br>
+ <img
src="/doc/image/yarn-queue/available_queues_when_creating_application.png"/><br></br><br></br>
+
+
+
+## Items about the feature
+
+- Will this feature affect the `yarn-application` mode flink applications or
`yarn-session` mode flink-clusters whose yarn queue(`yarn.application.queue`) &
labels(`yarn.application.node-label`) specified by the old dynamic properties ?
+> The feature will not affect it at all. SteamPark still retains the highest
priority of dynamic properties and does
+not enforce verification on the specified queues and labels in the dynamic
properties, providing users with space for advanced configuration.
+
+- The isolation of Yarn queue isn't stricter.
+> In the current designs, the permissions of queues aren't strictly isolated
due to
+the relationship between the queues and the `yarn-session` mode flink
clusters.
+> Flink `yarn-session` clusters are visible to all team, as shown in the
figure.
+The target `yarn-session` flink cluster uses a queue in the `default` team,
but it can still be selected in the `test` team.
+> So, if a `yarn-session` flink cluster uses a queue located in a certain
team, as the resource usage of the target cluster increases,
+the consumption of queues resource used by the target resource will also
increase.
+This is the phenomenon that yarn queues can't be strictly isolated
+due to the relationship between the queues and the `yarn-session` mode flink
clusters.
+> In other words, in `yarn-session` mode, flink clusters sharing leads to
indirect sharing of queues resource between teams.
+
+ <img
src="/doc/image/yarn-queue/use_yarn_session_mode_to_create_application.png"/><br></br>
+
+- Session cluster is shared by all teams. Why is it that when creating a
`yarn-session` flink cluster, only the queues in the current team instead of
all teams can be used in the queues candidate list ?
+> Based on the mentioned above, StreamPark hopes that when creating a
`yarn-session` flink cluster,
+administrators can specify the queue belonged to current of the current team
only,
+which could be better for administrators to perceive the impact of current
operations on the current team.
+
+- Why not support the isolation for `flink yarn-session clusters / general
clusters` on team wide ?
+ - The impact range caused by changes in cluster visibility is larger than
that caused by changes in queue visibility.
+ - StreamPark need to face greater difficulties in backward compatibility
while also considering the user experience.
+ - At present, there is no exact research on the users group and applications
scale deployed using
+ `yarn-application` & `yarn-session` cluster modes in the community.
+ Based on this fact, the community didn't provide greater feature support.
+
+
+- If you have any related requirements about the feature, feel free to contact
us or provide feedback directly to the community email list, please.
+ The community will make the next evaluation and support based on the usage
status of the users group and applications scale.
+
+Any suggestion is appreciated.
+
diff --git
a/static/doc/image/yarn-queue/available_queues_when_creating_application.png
b/static/doc/image/yarn-queue/available_queues_when_creating_application.png
new file mode 100644
index 00000000..8a978831
Binary files /dev/null and
b/static/doc/image/yarn-queue/available_queues_when_creating_application.png
differ
diff --git
a/static/doc/image/yarn-queue/available_queues_when_creating_cluster.png
b/static/doc/image/yarn-queue/available_queues_when_creating_cluster.png
new file mode 100644
index 00000000..dd96ce0a
Binary files /dev/null and
b/static/doc/image/yarn-queue/available_queues_when_creating_cluster.png differ
diff --git a/static/doc/image/yarn-queue/existed_queues.png
b/static/doc/image/yarn-queue/existed_queues.png
new file mode 100644
index 00000000..04a0b43b
Binary files /dev/null and b/static/doc/image/yarn-queue/existed_queues.png
differ
diff --git a/static/doc/image/yarn-queue/flow_to_create.png
b/static/doc/image/yarn-queue/flow_to_create.png
new file mode 100644
index 00000000..1db42bcb
Binary files /dev/null and b/static/doc/image/yarn-queue/flow_to_create.png
differ
diff --git a/static/doc/image/yarn-queue/flow_to_type_in_pure_queue.png
b/static/doc/image/yarn-queue/flow_to_type_in_pure_queue.png
new file mode 100644
index 00000000..bb339cb6
Binary files /dev/null and
b/static/doc/image/yarn-queue/flow_to_type_in_pure_queue.png differ
diff --git a/static/doc/image/yarn-queue/flow_to_type_in_pure_queue_labels.png
b/static/doc/image/yarn-queue/flow_to_type_in_pure_queue_labels.png
new file mode 100644
index 00000000..a7012a32
Binary files /dev/null and
b/static/doc/image/yarn-queue/flow_to_type_in_pure_queue_labels.png differ
diff --git
a/static/doc/image/yarn-queue/use_yarn_app_mode_to_create_application.png
b/static/doc/image/yarn-queue/use_yarn_app_mode_to_create_application.png
new file mode 100644
index 00000000..f73638c8
Binary files /dev/null and
b/static/doc/image/yarn-queue/use_yarn_app_mode_to_create_application.png differ
diff --git
a/static/doc/image/yarn-queue/use_yarn_session_mode_to_create_application.png
b/static/doc/image/yarn-queue/use_yarn_session_mode_to_create_application.png
new file mode 100644
index 00000000..5a2ef52e
Binary files /dev/null and
b/static/doc/image/yarn-queue/use_yarn_session_mode_to_create_application.png
differ
