This is an automated email from the ASF dual-hosted git repository.
peacewong pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/linkis-website.git
The following commit(s) were added to refs/heads/dev by this push:
new 62ded72f8d Optimize Architecture Documentation (#642)
62ded72f8d is described below
commit 62ded72f8d43fe1e4dcc6ca35385a3ff12867580
Author: peacewong <[email protected]>
AuthorDate: Fri Dec 30 17:49:13 2022 +0800
Optimize Architecture Documentation (#642)
* Optimize Architecture Documentation
* update img url
* update img url
---
.../proxy-user.md | 0
.../{engine => }/engine-conn-manager.md | 0
.../engine/_category_.json | 2 +-
.../engine/engine-conn.md | 183 +++++++++++---------
.../linkis-manager/ec-history-arc.md | 79 ---------
.../engine-conn-history.md | 0
.../computation-governance-services/overview.md | 18 +-
.../proxy-user.md | 0
.../{engine => }/engine-conn-manager.md | 0
.../engine/_category_.json | 2 +-
.../engine/engine-conn.md | 185 ++++++++++++---------
.../linkis-manager/ec-history-arc.md | 79 ---------
.../engine-conn-history.md | 0
.../computation-governance-services/overview.md | 24 +--
static/Images/Architecture/engine/ec_arc_01.png | Bin 0 -> 165321 bytes
static/Images/Architecture/engine/ec_arc_02.png | Bin 0 -> 26457 bytes
16 files changed, 228 insertions(+), 344 deletions(-)
diff --git
a/docs/architecture/feature/computation-governance-services/proxy-user.md
b/docs/architecture/feature/commons/proxy-user.md
similarity index 100%
rename from
docs/architecture/feature/computation-governance-services/proxy-user.md
rename to docs/architecture/feature/commons/proxy-user.md
diff --git
a/docs/architecture/feature/computation-governance-services/engine/engine-conn-manager.md
b/docs/architecture/feature/computation-governance-services/engine-conn-manager.md
similarity index 100%
rename from
docs/architecture/feature/computation-governance-services/engine/engine-conn-manager.md
rename to
docs/architecture/feature/computation-governance-services/engine-conn-manager.md
diff --git
a/docs/architecture/feature/computation-governance-services/engine/_category_.json
b/docs/architecture/feature/computation-governance-services/engine/_category_.json
index 7e2578b7f3..98fe7a2eea 100644
---
a/docs/architecture/feature/computation-governance-services/engine/_category_.json
+++
b/docs/architecture/feature/computation-governance-services/engine/_category_.json
@@ -1,4 +1,4 @@
{
- "label": "Engine",
+ "label": "EngineConn",
"position": 5
}
\ No newline at end of file
diff --git
a/docs/architecture/feature/computation-governance-services/engine/engine-conn.md
b/docs/architecture/feature/computation-governance-services/engine/engine-conn.md
index 07e3272f6d..f3581a3f25 100644
---
a/docs/architecture/feature/computation-governance-services/engine/engine-conn.md
+++
b/docs/architecture/feature/computation-governance-services/engine/engine-conn.md
@@ -3,102 +3,137 @@ title: EngineConn Design
sidebar_position: 1
---
-EngineConn architecture design
-==================
+## 1. Overview
-EngineConn: Engine connector, a module that provides functions such as unified
configuration management, context service, physical library, data source
management, microservice management, and historical task query for other
microservice modules.
+ EngineConn: engine connector, which is used to connect to the underlying
computing and storage engine to complete task execution, task information push
and result return, etc. It is the basis for Linkis to provide computing and
storage capabilities.
-EngineConn architecture diagram
+## 2. Overall Design
-
+ The overall design idea of EngineConn is to complete the acquisition and
storage of the session information of the underlying engine when starting,
complete the connection between the EngineConn process and the underlying
engine, and then complete the scheduling of tasks to the underlying engine
Session stored in EngineConn through the Executor unit for execution. and get
execution-related information.
-Introduction to the second-level module:
-==============
+### 2.1 Technical Architecture
-linkis-computation-engineconn interactive engine connector
----------------------------------------------
+**Introduction to related terms:**
-The ability to provide interactive computing tasks.
+**EngineConn:** Used to store the session information of the underlying
engine. To complete the connection with the underlying engine, for example, the
Spark engine stores the SparkSession.
-| Core class | Core function
|
-|----------------------|------------------------------------------------------------|
-| EngineConnTask | Defines the interactive computing tasks submitted to
EngineConn |
-| ComputationExecutor | Defined interactive Executor, with interactive
capabilities such as status query and task kill. |
-| TaskExecutionService | Provides management functions for interactive
computing tasks |
+**Executor:** The scheduling executor used to accept the task passed by the
caller (such as: Entrance), and finally submit the task to the underlying
engine Session for execution. Different tasks implement different Executor
classes. The most used is the interactive ComputationExecutor, which is used to
accept tasks and push task information to the caller in real time. And the
non-interactive ManageableOnceExecutor that accepts only one task is used to
complete the submission and executi [...]
-linkis-engineconn-common engine connector common module
---------------------------------------------
+
-Define the most basic entity classes and interfaces in the engine connector.
EngineConn is used to create a connection session for the underlying computing
storage engine, which contains the session information between the engine and
the specific cluster, and is the client that communicates with the specific
engine.
+### 2.2 Business Architecture
-| Core Service | Core function
|
-|-----------------------|----------------------------------------------------------------------|
-| EngineCreationContext | Contains the context information of EngineConn
during startup |
-| EngineConn | Contains the specific information of EngineConn,
such as type, specific connection information with layer computing storage
engine, etc. |
-| EngineExecution | Provide Executor creation logic
|
-| EngineConnHook | Define the operations before and after each phase of
engine startup |
+|Component name|First-level module|Second-level module|Function points|
+|:----|:----|:----|:----|
+|Linkis|EngineConn|linkis-engineconn-common|The common module of engine conn,
which defines the most basic entity classes and interfaces in engine conn. |
+|Linkis|EngineConn|linkis-engineconn-core|The core module of the engine
connector, which defines the interfaces involved in the core logic of
EngineConn. |
+|Linkis|EngineConn|linkis-executor-core|The core module of the executor, which
defines the core classes related to the executor. |
+|Linkis|EngineConn|linkis-accessible-executor|The underlying abstraction of
the accessible Executor. You can interact with it through RPC requests to
obtain its status, load, concurrency and other basic indicators Metrics data |
+|Linkis|EngineConn|linkis-computation-engineconn|Related classes that provide
capabilities for interactive computing tasks. |
-The core logic of linkis-engineconn-core engine connector
-------------------------------------------
+## 3. Module design
-Defines the interfaces involved in the core logic of EngineConn.
+Input: The caller executes the task
-| Core class | Core function |
-|-------------------|------------------------------------|
-| EngineConnManager | Provide related interfaces for creating and obtaining
EngineConn |
-| ExecutorManager | Provide related interfaces for creating and obtaining
Executor |
-| ShutdownHook | Define the operation of the engine shutdown phase
|
+Output: return task information such as execution status, results, logs, etc.
-linkis-engineconn-launch engine connector startup module
-------------------------------------------
+Key logic: the timing diagram of the key logic of task execution
-Defines the logic of how to start EngineConn.
+
-| Core class | core function |
-|------------------|--------------------------|
-| EngineConnServer | EngineConn microservice startup class |
+Key Notes:
-The core logic of the linkis-executor-core executor
-------------------------------------
+1. If it is a serial Executor, after EngineConn receives a task, it will mark
EngineConn as Busy and cannot accept other tasks, and will judge whether the
lock of the task is consistent to prevent EngineConn from being submitted by
multiple callers at the same time. After the task is executed, it becomes the
Unlock state
+2. If it is a parallel Executor, after EngineConn receives the task, the state
is still in the Unlock state, and it can continue to accept the task. Only when
the number of concurrent tasks is reached or the machine index is abnormal will
it be marked as Busy state
+3. If it is an Once type task, EngineConn will automatically execute the task
after it is started, and the EngineConn process will exit after the task is
executed.
-> Defines the core classes related to the actuator. The executor is a real
computing scene executor, responsible for submitting user code to EngineConn.
+## 4. Data structure/storage design
-| Core class | Core function
|
-|----------------------------|------------------------------------------------------------|
-| Executor | It is the actual computational logic execution unit and provides
a top-level abstraction of the various capabilities of the engine. |
-| EngineConnAsyncEvent | Defines EngineConn-related asynchronous events |
-| EngineConnSyncEvent | Defines EngineConn-related synchronization events |
-| EngineConnAsyncListener | Defines EngineConn related asynchronous event
listener |
-| EngineConnSyncListener | Defines EngineConn related synchronization event
listener |
-| EngineConnAsyncListenerBus | Defines the listener bus for EngineConn
asynchronous events |
-| EngineConnSyncListenerBus | Defines the listener bus for EngineConn
synchronization events |
-| ExecutorListenerBusContext | Defines the context of the EngineConn event
listener |
-| LabelService | Provide label reporting function |
-| ManagerService | Provides the function of information transfer with
LinkisManager |
+not involving
-linkis-callback-service callback logic
--------------------------------
+## 5. Interface design
-| Core Class | Core Function |
-|--------------------|--------------------------|
-| EngineConnCallback | Define EngineConn's callback logic |
+**Brief introduction of other classes:**
-linkis-accessible-executor can be accessed executor
---------------------------------------------
+The common module of linkis-engineconn-common engine connector defines the
most basic entity classes and interfaces in the engine connector.
-Executor that can be accessed. You can interact with it through RPC requests
to get its status, load, concurrency and other basic indicators Metrics data.
+|Core Service|Core Function|
+|:----|:----|
+|EngineCreationContext|contains the context information of EngineConn during
startup|
+|EngineConn| contains the specific information of EngineConn, such as type,
specific connection information with layer computing storage engine, etc.|
+|EngineExecution|Provides the creation logic of Executor|
+|EngineConnHook|Defines the operations before and after each stage of engine
startup|
+The core module of linkis-engineconn-core engine connector defines the
interfaces involved in the core logic of EngineConn.
-| Core Class | Core Function |
-|--------------------------|-------------------------------------------------|
-| LogCache | Provide log cache function |
-| AccessibleExecutor | The Executor that can be accessed can interact with it
through RPC requests. |
-| NodeHealthyInfoManager | Manage Executor's Health Information |
-| NodeHeartbeatMsgManager | Manage the heartbeat information of Executor |
-| NodeOverLoadInfoManager | Manage Executor load information |
-| Listener | Provides events related to Executor and the corresponding
listener definition |
-| EngineConnTimedLock | Define Executor level lock |
-| AccessibleService | Provides the start-stop and status acquisition functions
of Executor |
-| ExecutorHeartbeatService | Provides heartbeat related functions of Executor |
-| LockService | Provide lock management function |
-| LogService | Provide log management functions |
\ No newline at end of file
+|Core Classes|Core Functions|
+|:----|:----|
+|EngineConnManager|Provides related interfaces for creating and obtaining
EngineConn|
+|ExecutorManager|Provides related interfaces for creating and obtaining
Executor|
+|ShutdownHook|Defines actions during engine shutdown|
+|EngineConnServer|Startup class of EngineConn microservice|
+
+linkis-executor-core is the core module of the executor, which defines the
core classes related to the executor. The executor is the real computing
execution unit, which is responsible for submitting user code to EngineConn for
execution.
+
+|Core Classes|Core Functions|
+|:----|:----|
+|Executor| is the actual computing logic execution unit, and provides
top-level abstraction of various capabilities of the engine. |
+|EngineConnAsyncEvent| defines EngineConn related asynchronous events|
+|EngineConnSyncEvent| defines the synchronization event related to EngineConn|
+|EngineConnAsyncListener| defines EngineConn-related asynchronous event
listeners|
+|EngineConnSyncListener| defines EngineConn-related synchronization event
listeners|
+|EngineConnAsyncListenerBus|Defines the listener bus for EngineConn
asynchronous events|
+|EngineConnSyncListenerBus|Defines the listener bus for EngineConn sync events|
+|ExecutorListenerBusContext| defines the context of the EngineConn event
listener|
+|LabelService|Provide label reporting function|
+|ManagerService|Provides the function of information transfer with
LinkisManager|
+
+linkis-accessible-executor: The underlying abstraction of the Executor that
can be accessed. You can interact with it through RPC requests to obtain basic
metrics such as its status, load, and concurrency.
+
+|Core Classes|Core Functions|
+|:----|:----|
+|LogCache|Provides the function of log caching|
+|AccessibleExecutor| An Executor that can be accessed and interacted with via
RPC requests. |
+|NodeHealthyInfoManager|Manage Executor's health information|
+|NodeHeartbeatMsgManager|Manage Executor's heartbeat information|
+|NodeOverLoadInfoManager|Manage Executor load information|
+|Listener-related|Provides events related to Executor and corresponding
listener definitions|
+|EngineConnTimedLock|Define Executor level lock|
+|AccessibleService|Provide the start-stop and status acquisition functions of
Executor|
+|ExecutorHeartbeatService|Provides Executor's heartbeat-related functions|
+|LockService|Provides lock management functions|
+|LogService|Provides log management functions|
+|EngineConnCallback|Define the callback logic of EngineConn|
+
+Related classes that provide capabilities for interactive computing tasks.
+
+|Core Classes|Core Functions|
+|:----|:----|
+|EngineConnTask| defines interactive computing tasks submitted to EngineConn|
+|ComputationExecutor| defines an interactive Executor, which has interactive
capabilities such as status query and task kill, and can only execute tasks
once by default. |
+|ConcurrentComputationExecutor|Interactive synchronous concurrent Executor,
inherited from ComputationExecutor, but supports executing multiple tasks at
the same time|
+|AsyncConcurrentComputationExecutor|Interactive asynchronous concurrent
Executor, inherited from ComputationExecutor, supports multiple tasks to be
executed at the same time, and the task does not occupy the execution thread
and adopts the form of asynchronous notification|
+|TaskExecutionService|Provides management functions for interactive computing
tasks|
+
+
+## 6. Non-functional design
+
+### 6.1 Security
+
+1. All the relevant information of the task can only be queried by submitting
the user
+2. The default startup user of the EngineConn process is the submission user
+### 6.2 Performance
+
+EngineConn that supports concurrency supports colleagues to run a large number
of tasks concurrently. For example, a single Trino EngineConn can run more than
300 trino tasks at the same time
+
+### 6.3 Capacity
+
+not involving
+
+### 6.4 High Availability
+
+EngineConn is a process started on demand and task. Support high availability
+
+### 6.5 Data Quality
+
+not involving
\ No newline at end of file
diff --git
a/docs/architecture/feature/computation-governance-services/linkis-manager/ec-history-arc.md
b/docs/architecture/feature/computation-governance-services/linkis-manager/ec-history-arc.md
deleted file mode 100644
index bbce512de8..0000000000
---
a/docs/architecture/feature/computation-governance-services/linkis-manager/ec-history-arc.md
+++ /dev/null
@@ -1,79 +0,0 @@
----
-title: EC History List Architecture Design
-sidebar_position: 4
----
-
-## 1. General
-### 1.1 Requirements Background
- Now LinkisManager only records the information
and resource usage of the running EengineConn, but this information is lost
after the task ends. It is inconvenient to do some statistics and view of
historical ECs, or to view logs of ECs that have ended. It is more important to
record the historical EC.
-### 1.2 Target
-1. Complete the persistence of EC information and resource information to DB
storage
-2. Support the viewing and searching of historical EC information through
restful
-3. Support to view the logs of EC that has ended
-
-## 2. Design
- The main modification of this feature is the RM
and AM modules under LinkisManager, and a new information record table is
added. It will be described in detail below.
-
-### 2.1 Technical Architecture
- Because this implementation needs to record EC
information and resource information, and resource information is divided into
three concepts, such as requesting resources, actual use of resources, and
release of resources, and all need to be recorded. Therefore, this
implementation is based on the life cycle of the EC in the ResourceManager.
When the EC completes the above three stages, the update operation of the EC
information is added. The whole is shown [...]
-
-
-### 2.2 Business Architecture
- This feature is mainly to complete the
information recording of historical EC and support the log viewing of
historical technical EC. The modules designed by the function point are as
follows:
-
-| Component name | First-level module | Second-level module | Function point |
-|---|---|---|---|
-| Linkis | LinkisManager | ResourceManager| Complete the EC information record
when the EC requests resources, reports the use of resources, and releases
resources|
-| Linkis | LinkisManager | AppManager| Provides an interface to list and
search all historical EC information|
-
-## 3. Module Design
-### Core execution flow
-[Input] The input is mainly the information input when creating the engine,
requesting resources, reporting the actual use of resources after the engine
starts, and releasing resources when the engine exits, mainly including the
requested label, resource, ec unique ticketid, and resource type.
-[Processing process] The information recording service processes the input
data, and parses the corresponding engine information, user, creator, and log
path through tags. Confirm the resource request, use, and release by the
resource type. Then talk about the information stored in the DB.
-The call sequence diagram is as follows:
-
-
-
-## 4. DDL:
-```sql
-# EC information resource record table
-DROP TABLE IF EXISTS `linkis_cg_ec_resource_info_record`;
-CREATE TABLE `linkis_cg_ec_resource_info_record` (
- `id` INT(20) NOT NULL AUTO_INCREMENT,
- `label_value` VARCHAR(255) NOT NULL COMMENT 'ec labels stringValue',
- `create_user` VARCHAR(128) NOT NULL COMMENT 'ec create user',
- `service_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ec
instance info',
- `ecm_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ecm
instance info ',
- `ticket_id` VARCHAR(100) NOT NULL COMMENT 'ec ticket id',
- `log_dir_suffix` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'log
path',
- `request_times` INT(8) COMMENT 'resource request times',
- `request_resource` VARCHAR(255) COMMENT 'request resource',
- `used_times` INT(8) COMMENT 'resource used times',
- `used_resource` VARCHAR(255) COMMENT 'used resource',
- `release_times` INT(8) COMMENT 'resource released times',
- `released_resource` VARCHAR(255) COMMENT 'released resource',
- `release_time` datetime DEFAULT NULL COMMENT 'released time',
- `used_time` datetime DEFAULT NULL COMMENT 'used time',
- `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time',
- PRIMARY KEY (`id`),
- KEY (`ticket_id`),
- UNIQUE KEY `label_value_ticket_id` (`ticket_id`, `label_value`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
-````
-## 5. Interface design:
-The API interface of the engine history management page, refer to the document
Management console to add a history engine page
-https://linkis.apache.org/docs/latest/api/http/linkis-cg-linkismanager-api/ec-resource-management-api
-
-## 6. Non-functional design:
-
-### 6.1 Security
-No security issues are involved, restful requires login authentication
-
-### 6.2 Performance
-Less impact on engine life cycle performance
-
-### 6.3 Capacity
-Requires regular cleaning
-
-### 6.4 High Availability
-not involving
\ No newline at end of file
diff --git
a/docs/architecture/feature/computation-governance-services/engine/engine-conn-history.md
b/docs/architecture/feature/computation-governance-services/linkis-manager/engine-conn-history.md
similarity index 100%
rename from
docs/architecture/feature/computation-governance-services/engine/engine-conn-history.md
rename to
docs/architecture/feature/computation-governance-services/linkis-manager/engine-conn-history.md
diff --git
a/docs/architecture/feature/computation-governance-services/overview.md
b/docs/architecture/feature/computation-governance-services/overview.md
index 3dbffcf761..813381c059 100644
--- a/docs/architecture/feature/computation-governance-services/overview.md
+++ b/docs/architecture/feature/computation-governance-services/overview.md
@@ -3,22 +3,10 @@ title: Overview
sidebar_position: 1
---
-## Background
-**The architecture of Linkis0.X mainly has the following problems**
-1. The boundary between the core processing flow and the hierarchical module
is blurred:
-- Entrance and EngineManager function boundaries are blurred.
+## **Overview**
-- The main process of task submission and execution is not clear enough.
+Computing Governance Service Group, CGS: Computation Governance Services. It
is the core module of Linkis to complete the main steps of computing tasks and
requests such as submission, preparation, execution, and return of results.
-- It is troublesome to extend the new engine, and it needs to implement the
code of multiple modules.
-
-- Only support computing request scenarios, storage request scenarios and
resident service mode (Cluster) are difficult to support.
-2. Demands for richer and more powerful computing governance functions:
-- Insufficient support for computing task management strategies.
-
-- The labeling capability is not strong enough, which restricts computing
strategies and resource managemen.
-
-The new architecture of Linkis1.0 computing governance service can solve these
problems well.
## Architecture Diagram
**Operation process optimization:** Linkis1.0 will optimize the overall
execution process of the job, from submission —\> preparation —\>
@@ -44,7 +32,7 @@ Perform three stages to fully upgrade Linkis's Job execution
architecture, as sh
[Enter LinkisManager Architecture Design](linkis-manager/overview.md)
### 4. Engine Manager
Engine conn Manager (ECM) is a
simplified and upgraded version of linkis0. X engine manager. The ECM under
linkis1.0 removes the application ability of the engine, and the whole
microservice is completely stateless. It will focus on supporting the startup
and destruction of all kinds of enginecon.
-[Enter EngineConnManager Architecture Design](engine/engine-conn-manager.md)
+[Enter EngineConnManager Architecture Design](engine-conn-manager.md)
### 5. EngineConn
EngineConn is an optimized
and upgraded version of Linkis0.X Engine. It will provide EngineConn and
Executor two modules. EngineConn is used to connect the underlying computing
storage engine and provide a session session that connects the underlying
computing storage engines; Executor is based on this Session session , Provide
full-stack computing support for interactive computing, streaming computing,
offline computing, and data storage.
[Enter EngineConn Architecture Design](engine/engine-conn.md)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/proxy-user.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/commons/proxy-user.md
similarity index 100%
rename from
i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/proxy-user.md
rename to
i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/commons/proxy-user.md
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn-manager.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine-conn-manager.md
similarity index 100%
rename from
i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn-manager.md
rename to
i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine-conn-manager.md
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/_category_.json
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/_category_.json
index 871f751594..7d5b0b6f6b 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/_category_.json
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/_category_.json
@@ -1,4 +1,4 @@
{
- "label": "引擎服务架构",
+ "label": "引擎连接器架构",
"position": 5
}
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn.md
index cba1a291c2..c215f424a1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn.md
@@ -4,101 +4,138 @@ sidebar_position: 1
---
-EngineConn架构设计
-==================
+## 1.总述
-EngineConn：引擎连接器，为其他微服务模块提供统一配置管理、上下文服务、物料库、数据源管理、微服务管理和历史任务查询等功能的模块。
+EngineConn：引擎连接器，用于连接底层的计算存储引擎，完成任务的执行、任务信息推送和结果返回等，是Linkis提供计算存储能力的基础。
-一、EngineConn架构图
+## 2. 总体设计
-
+
EngineConn的整体设计思路，启动的时候会完成底层引擎Session信息的获取和存储，完成EngineConn进程和底层引擎的连接，然后通过Executor执行器单元完成任务的调度到EngineConn存储的底层引擎Session中进行执行，并获取执行相关的信息。
-二级模块介绍：
-==============
+### 2.1 技术架构
-linkis-computation-engineconn交互式引擎连接器
----------------------------------------------
+**关名词介绍：**
-提供交互式计算任务的能力。
+**EngineConn：** 用于存储底层引擎的Session信息。来完成和底层引擎的连接，例如Spark引擎存储的是SparkSession。
-| 核心类 | 核心功能 |
-|----------------------|------------------------------------------------------------|
-| EngineConnTask | 定义了提交给EngineConn的交互式计算任务 |
-| ComputationExecutor | 定义了交互式Executor，具备状态查询、任务kill等交互式能力。 |
-| TaskExecutionService | 提供对交互式计算任务的管理功能 |
+**Executor：**用于接受调用方(如：Entrance)传过来的任务的调度执行器，并将任务最终提交给底层的引擎Session进行执行，不同的任务实现不同的Executor类。使用最多的是交互式ComputationExecutor，用于接受任务并实时推送任务信息给到调用方。以及非交互式的只接受一次任务的ManageableOnceExecutor，用于完成对EngineConn启动的任务的提交执行。
-linkis-engineconn-common引擎连接器的通用模块
---------------------------------------------
+
-1.
定义了引擎连接器中最基础的实体类和接口。EngineConn是用于创建一个底层计算存储引擎的连接会话Session，包含引擎与具体集群的会话信息，是与具体引擎通信的client。
+### 2.2 业务架构
-| 核心Service | 核心功能
|
-|-----------------------|----------------------------------------------------------------------|
-| EngineCreationContext | 包含了EngineConn在启动期间的上下文信息
|
-| EngineConn | 包含了EngineConn的具体信息，如类型、与层计算存储引擎的具体连接信息等 |
-| EngineExecution | 提供Executor的创建逻辑
|
-| EngineConnHook | 定义引擎启动各个阶段前后的操作
|
+|组件名|一级模块|二级模块|功能点|
+|:----|:----|:----|:----|
+|Linkis|EngineConn|linkis-engineconn-common|引擎连接器的通用模块，定义了引擎连接器中最基础的实体类和接口。|
+|Linkis|EngineConn|linkis-engineconn-core|引擎连接器的核心模块，定义了EngineConn的核心逻辑涉及的接口。|
+|Linkis|EngineConn|linkis-executor-core|执行器的核心模块，定义了执行器相关的核心类。|
+|Linkis|EngineConn|linkis-accessible-executor|能够被访问的Executor底层抽象。可以通过RPC请求与它交互，从而获取它的状态、负载、并发等基础指标Metrics数据|
+|Linkis|EngineConn|linkis-computation-engineconn|提供交互式计算任务能力的相关类。|
-linkis-engineconn-core引擎连接器的核心逻辑
-------------------------------------------
+## 3. 模块设计
-定义了EngineConn的核心逻辑涉及的接口。
+输入：调用方执行任务
-| 核心类 | 核心功能 |
-|-------------------|------------------------------------|
-| EngineConnManager | 提供创建、获取EngineConn的相关接口 |
-| ExecutorManager | 提供创建、获取Executor的相关接口 |
-| ShutdownHook | 定义引擎关闭阶段的操作 |
+输出：返回执行状态、结果、日志等任务信息
-linkis-engineconn-launch引擎连接器启动模块
-------------------------------------------
+关键逻辑：任务执行的关键逻辑的时序图
-定义了如何启动EngineConn的逻辑。
+
-| 核心类 | 核心功能 |
-|------------------|--------------------------|
-| EngineConnServer | EngineConn微服务的启动类 |
+关键说明：
-linkis-executor-core执行器的核心逻辑
-------------------------------------
+1.
如果是串行Executor，EngineConn接收到任务后就会将EngineConn标记为Busy状态，不能再接受其他任务，并且会判断任务的锁是否一致，防止EngineConn被多个调用方同时提交的情况。任务执行完变成Unlock状态
+2.
如果是并行的Executor，EngineConn接收到任务后，状态还是Unlock状态，可以继续接受任务，只有当达到任务并发数或者机器指标异常才会标记为Busy状态
+3. 如果是Once类型任务，EngineConn启动后就会自动执行任务，任务执行结束EngineConn进程退出
-> 定义了执行器相关的核心类。执行器是真正的计算场景执行器，负责将用户代码提交给EngineConn。
+## 4. 数据结构/存储设计
-| 核心类 | 核心功能
|
-|----------------------------|------------------------------------------------------------|
-| Executor | 是实际的计算逻辑执行单元，并提供对引擎各种能力的顶层抽象。 |
-| EngineConnAsyncEvent | 定义了EngineConn相关的异步事件
|
-| EngineConnSyncEvent | 定义了EngineConn相关的同步事件
|
-| EngineConnAsyncListener | 定义了EngineConn相关异步事件监听器 |
-| EngineConnSyncListener | 定义了EngineConn相关同步事件监听器 |
-| EngineConnAsyncListenerBus | 定义了EngineConn异步事件的监听器总线 |
-| EngineConnSyncListenerBus | 定义了EngineConn同步事件的监听器总线 |
-| ExecutorListenerBusContext | 定义了EngineConn事件监听器的上下文 |
-| LabelService | 提供标签上报功能
|
-| ManagerService | 提供与LinkisManager进行信息传递的功能 |
+不涉及
-linkis-callback-service回调逻辑
--------------------------------
+## 5. 接口设计
-| 核心类 | 核心功能 |
-|--------------------|--------------------------|
-| EngineConnCallback | 定义EngineConn的回调逻辑 |
+**其他类简要介绍：**
-linkis-accessible-executor能够被访问的执行器
---------------------------------------------
+linkis-engineconn-common引擎连接器的通用模块，定义了引擎连接器中最基础的实体类和接口。
-能够被访问的Executor。可以通过RPC请求与它交互，从而获取它的状态、负载、并发等基础指标Metrics数据。
+|核心Service|核心功能|
+|:----|:----|
+|EngineCreationContext|包含了EngineConn在启动期间的上下文信息|
+|EngineConn|包含了EngineConn的具体信息，如类型、与层计算存储引擎的具体连接信息等|
+|EngineExecution|提供Executor的创建逻辑|
+|EngineConnHook|定义引擎启动各个阶段前后的操作|
+
+linkis-engineconn-core引擎连接器的核心模块，定义了EngineConn的核心逻辑涉及的接口。
+
+|核心类|核心功能|
+|:----|:----|
+|EngineConnManager|提供创建、获取EngineConn的相关接口|
+|ExecutorManager|提供创建、获取Executor的相关接口|
+|ShutdownHook|定义引擎关闭阶段的操作|
+|EngineConnServer|EngineConn微服务的启动类|
+
+linkis-executor-core执行器的核心模块，定义了执行器相关的核心类。执行器是真正的计算执行单元，负责将用户代码提交给EngineConn进行执行。
+
+|核心类|核心功能|
+|:----|:----|
+|Executor|是实际的计算逻辑执行单元，并提供对引擎各种能力的顶层抽象。|
+|EngineConnAsyncEvent|定义了EngineConn相关的异步事件|
+|EngineConnSyncEvent|定义了EngineConn相关的同步事件|
+|EngineConnAsyncListener|定义了EngineConn相关异步事件监听器|
+|EngineConnSyncListener|定义了EngineConn相关同步事件监听器|
+|EngineConnAsyncListenerBus|定义了EngineConn异步事件的监听器总线|
+|EngineConnSyncListenerBus|定义了EngineConn同步事件的监听器总线|
+|ExecutorListenerBusContext|定义了EngineConn事件监听器的上下文|
+|LabelService|提供标签上报功能|
+|ManagerService|提供与LinkisManager进行信息传递的功能|
+
+linkis-accessible-executor：能够被访问的Executor底层抽象。可以通过RPC请求与它交互，从而获取它的状态、负载、并发等基础指标Metrics数据。
+
+|核心类|核心功能|
+|:----|:----|
+|LogCache|提供日志缓存的功能|
+|AccessibleExecutor|能够被访问的Executor，可以通过RPC请求与它交互。|
+|NodeHealthyInfoManager|管理Executor的健康信息|
+|NodeHeartbeatMsgManager|管理Executor的心跳信息|
+|NodeOverLoadInfoManager|管理Executor的负载信息|
+|Listener相关|提供与Executor相关的事件以及对应的监听器定义|
+|EngineConnTimedLock|定义Executor级别的锁|
+|AccessibleService|提供Executor的启停、状态获取功能|
+|ExecutorHeartbeatService|提供Executor的心跳相关功能|
+|LockService|提供锁管理功能|
+|LogService|提供日志管理功能|
+|EngineConnCallback|定义EngineConn的回调逻辑|
+
+提供交互式计算任务能力的相关类。
+
+|核心类|核心功能|
+|:----|:----|
+|EngineConnTask|定义了提交给EngineConn的交互式计算任务|
+|ComputationExecutor|定义了交互式Executor，具备状态查询、任务kill等交互式能力，默认每次只能执行一次任务。|
+|ConcurrentComputationExecutor|交互式同步并发Executor，继承于ComputationExecutor，但是支持同时执行多个任务|
+|AsyncConcurrentComputationExecutor|交互式异步并发Executor，继承ComputationExecutor，支持同时执行多个任务，并且任务不占用执行线程采用异步通知的形式|
+|TaskExecutionService|提供对交互式计算任务的管理功能|
+
+
+## 6. 非功能性设计
+
+### 6.1 安全
+
+1. 任务所有相关的信息，只能提交用户才能进行查询
+2. EngineConn进程的默认启动用户为提交用户
+### 6.2 性能
+
+支持并发的EngineConn支持同事并发跑大量任务，如单个trino的EngineConn可以同时跑300多个trino任务
+
+### 6.3 容量
+
+不涉及
+
+### 6.4 高可用
+
+EngineConn是按需、按任务启动的进程。支持高可用
+
+### 6.5 数据质量
+
+不涉及
-| 核心类 | 核心功能 |
-|--------------------------|-------------------------------------------------|
-| LogCache | 提供日志缓存的功能 |
-| AccessibleExecutor | 能够被访问的Executor，可以通过RPC请求与它交互。 |
-| NodeHealthyInfoManager | 管理Executor的健康信息 |
-| NodeHeartbeatMsgManager | 管理Executor的心跳信息 |
-| NodeOverLoadInfoManager | 管理Executor的负载信息 |
-| Listener | 提供与Executor相关的事件以及对应的监听器定义 |
-| EngineConnTimedLock | 定义Executor级别的锁 |
-| AccessibleService | 提供Executor的启停、状态获取功能 |
-| ExecutorHeartbeatService | 提供Executor的心跳相关功能 |
-| LockService | 提供锁管理功能 |
-| LogService | 提供日志管理功能 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/linkis-manager/ec-history-arc.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/linkis-manager/ec-history-arc.md
deleted file mode 100644
index 190e112a54..0000000000
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/linkis-manager/ec-history-arc.md
+++ /dev/null
@@ -1,79 +0,0 @@
----
-title: EngineConn历史引擎信息记录架构
-sidebar_position: 3
----
-
-## 1. 总述
-### 需求背景
- 现在LinkisManager只记录了在运行中的EengineConn的信息和资源使用，但是在任务结束后这些信息就丢失了。如果需要做历史EC的一些统计和查看，或者要去查看已经结束的EC的日志都不太方便。对于历史EC的记录显得较为重要。
-### 目标
-1. 完成EC信息和资源信息持久化到DB的存储
-2. 支持通过restful完成历史EC信息的查看和搜索
-3. 支持查看已经结束EC的日志
-
-## 2. 总体设计
- 此次特性新增主要修改为LinkisManager下的RM和AM模块，并新增了一种信息记录表。下面将详细介绍。
-
-### 2.1 技术架构
- 因为此次的实现需要记录EC的信息和资源信息，而资源信息分为请求资源、真实使用资源、释放资源等三个概念，而且都需要进行记录。所以此次实现为基于EC在ResourceManager的生命周期去进行实现，在EC完成以上三个阶段时，都加上EC信息的更新操作。整体如下图所示
-
-
-### 2.2 业务架构
- 此次的特性主要是为了完成历史EC的信息记录和支持历史技术EC的日志查看。功能点设计的模块如下：
-
-| 组件名| 一级模块 | 二级模块 | 功能点 |
-|---|---|---|---|
-| Linkis | LinkisManager | ResourceManager| 在EC请求资源、上报使用资源、释放资源时完成EC信息的记录|
-| Linkis | LinkisManager | AppManager| 提供list和搜索所有历史EC信息的接口|
-
-## 3. 模块设计
-### 核心执行流程
-- [输入端]
输入端主要为创建引擎时请求资源、引擎启动后上报真实使用资源、引擎退出时释放资源时输入的信息，主要包括请求的label、资源、ec唯一的ticketid、资源类型.
-- [处理流程]
信息记录service，对输入的数据进行处理，通过标签会解析出对应的引擎信息、用户、creator，以及日志路径。通过资源类型确认是资源请求、使用、释放。接着将这些信息存储到DB中。
-调用时序图如下：
-
-
-
-## 4. 数据结构：
-```sql
-# EC信息资源记录表
-DROP TABLE IF EXISTS `linkis_cg_ec_resource_info_record`;
-CREATE TABLE `linkis_cg_ec_resource_info_record` (
- `id` INT(20) NOT NULL AUTO_INCREMENT,
- `label_value` VARCHAR(255) NOT NULL COMMENT 'ec labels stringValue',
- `create_user` VARCHAR(128) NOT NULL COMMENT 'ec create user',
- `service_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ec
instance info',
- `ecm_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ecm
instance info ',
- `ticket_id` VARCHAR(100) NOT NULL COMMENT 'ec ticket id',
- `log_dir_suffix` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'log
path',
- `request_times` INT(8) COMMENT 'resource request times',
- `request_resource` VARCHAR(255) COMMENT 'request resource',
- `used_times` INT(8) COMMENT 'resource used times',
- `used_resource` VARCHAR(255) COMMENT 'used resource',
- `release_times` INT(8) COMMENT 'resource released times',
- `released_resource` VARCHAR(255) COMMENT 'released resource',
- `release_time` datetime DEFAULT NULL COMMENT 'released time',
- `used_time` datetime DEFAULT NULL COMMENT 'used time',
- `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time',
- PRIMARY KEY (`id`),
- KEY (`ticket_id`),
- UNIQUE KEY `label_value_ticket_id` (`ticket_id`,`label_value`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
-```
-## 5. 接口设计：
-引擎历史管理页面API接口，参考文档 管理台添加历史引擎页面
-https://linkis.apache.org/docs/latest/api/http/linkis-cg-linkismanager-api/ec-resource-management-api
-## 6. 非功能性设计：
-
-### 6.1 安全
-不涉及安全问题，restful需要登录认证
-
-### 6.2 性能
-对引擎生命周期性能影响较小
-
-### 6.3 容量
-需要定期进行清理
-
-### 6.4 高可用
-不涉及
-
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn-history.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/linkis-manager/engine-conn-history.md
similarity index 100%
rename from
i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/engine/engine-conn-history.md
rename to
i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/linkis-manager/engine-conn-history.md
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/overview.md
index 5dd885a72a..503efc0bff 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/architecture/feature/computation-governance-services/overview.md
@@ -4,27 +4,9 @@ sidebar_position: 1
---
-## **背景**
+## **总述**
-**Linkis0.X的架构主要存在以下问题**
-
-1.核心处理流程和层级模块边界模糊
-
-- Entrance 和 EngineManager 功能边界模糊
-
-- 任务提交执行主流程不够清晰
-
-- 扩展新引擎较麻烦，需要实现多个模块的代码
-
-- 只支持计算请求场景，存储请求场景和常驻服务模式（Cluster）难以支持
-
-2.更丰富强大计算治理功能需求
-
-- 计算任务管理策略支持度不够
-
-- 标签能力不够强大，制约计算策略和资源管理
-
-Linkis1.0计算治理服务的新架构可以很好的解决这些问题。
+计算治理服务组,CGS：Computation Governance
Services.完成计算任务和请求的提交、准备、执行、返回结果等主要步骤，是Linkis的核心模块
## **架构图**
@@ -66,7 +48,7 @@ Linkis1.0将优化Job的整体执行流程，从提交 —\> 准备 —\>
EngineConnManager (简称ECM)是 Linkis0.X EngineManager
的精简升级版。Linkis1.0下的ECM去除了引擎的申请能力，整个微服务完全无状态，将聚焦于支持各类 EngineConn 的启动和销毁。
- [进入EngineConnManager架构设计](engine/engine-conn-manager.md)
+ [进入EngineConnManager架构设计](engine-conn-manager.md)
### 5、EngineConn
diff --git a/static/Images/Architecture/engine/ec_arc_01.png
b/static/Images/Architecture/engine/ec_arc_01.png
new file mode 100644
index 0000000000..11255612c0
Binary files /dev/null and b/static/Images/Architecture/engine/ec_arc_01.png
differ
diff --git a/static/Images/Architecture/engine/ec_arc_02.png
b/static/Images/Architecture/engine/ec_arc_02.png
new file mode 100644
index 0000000000..43917acd7c
Binary files /dev/null and b/static/Images/Architecture/engine/ec_arc_02.png
differ
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
