This is an automated email from the ASF dual-hosted git repository.
casion pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-linkis-website.git
The following commit(s) were added to refs/heads/dev by this push:
new 224e84ab1f document translation (#440)
224e84ab1f is described below
commit 224e84ab1fd2ac80b274432ab60e187734715ea5
Author: 成彬彬 <[email protected]>
AuthorDate: Tue Jul 19 19:44:00 2022 +0800
document translation (#440)
---
.../engine/engine_conn_history.md | 89 ++++++++
.../engine/engine_conn_metrics.md | 76 +++++++
.../context_service/content_service_cleanup.md | 238 +++++++++++++++++++++
3 files changed, 403 insertions(+)
diff --git
a/docs/architecture/computation_governance_services/engine/engine_conn_history.md
b/docs/architecture/computation_governance_services/engine/engine_conn_history.md
new file mode 100644
index 0000000000..46e2d888d9
--- /dev/null
+++
b/docs/architecture/computation_governance_services/engine/engine_conn_history.md
@@ -0,0 +1,89 @@
+---
+title: EngineConn History Features
+sidebar_position: 5
+tags: [Feature]
+---
+
+## 1. Functional requirements
+### 1.1 Requirement Background
+Before version 1.1.3, LinkisManager only recorded the information and resource
usage of the running EngineConn, but the information was lost after the task
was completed. If you need to do some statistics and view of historical ECs, or
to view the logs of ECs that have ended, it is too cumbersome, so it is more
important to record historical ECs.
+
+### 1.2 Goals
+- Complete the storage of EC information and resource information persistent
to DB
+- Supports viewing and searching of historical EC information through the
restful interface
+- Support to view logs of EC that has ended
+
+## 2. Overall Design
+
+The main changes in this feature are the RM and AM modules under
LinkisManager, and an information record table has been added.
+
+### 2.1 Technical Architecture
+Because this implementation needs to record EC information and resource
information, and resource information is divided into three concepts, such as
requesting resources, actually using resources, and releasing resources, and
all of them need to be recorded. Therefore, the general plan for this
implementation is: based on the EC in the life cycle of the ResourceManager to
implement, and when the EC completes the above three stages, the update
operation of the EC information is added. Th [...]
+
+
+
+
+
+### 2.2 Business Architecture
+
+This feature is mainly to complete the information recording of historical ECs
and support the log viewing of historical technical ECs. The modules designed
by the function point are as follows:
+
+| First-level module | Second-level module | Function point |
+|---|---|---|
+| LinkisManager | ResourceManager| Complete the EC information record when the
EC requests resources, reports the use of resources, and releases resources|
+| LinkisManager | AppManager| Provides an interface to list and search all
historical EC information|
+
+## 3. Module Design
+### Core execution flow
+
+- \[Input] The input is mainly for the requested resource when the engine is
created, the real used resource reported after the engine is started, and the
information input when the resource is released when the engine exits, mainly
including the requested label, resource, EC's unique ticketid, resource type
etc.
+- \[Processing process] Information recording service, which processes the
input data, and parses the corresponding engine information, user, creator, and
log path through tags. Confirm the resource request, use, and release by the
resource type. Then talk about the information stored in the DB.
+
+The call sequence diagram is as follows:
+
+
+
+
+## 4. Data structure:
+```sql
+# EC information resource record table
+DROP TABLE IF EXISTS `linkis_cg_ec_resource_info_record`;
+CREATE TABLE `linkis_cg_ec_resource_info_record` (
+ `id` INT(20) NOT NULL AUTO_INCREMENT,
+ `label_value` VARCHAR(255) NOT NULL COMMENT 'ec labels stringValue',
+ `create_user` VARCHAR(128) NOT NULL COMMENT 'ec create user',
+ `service_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ec
instance info',
+ `ecm_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ecm
instance info ',
+ `ticket_id` VARCHAR(100) NOT NULL COMMENT 'ec ticket id',
+ `log_dir_suffix` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'log
path',
+ `request_times` INT(8) COMMENT 'resource request times',
+ `request_resource` VARCHAR(255) COMMENT 'request resource',
+ `used_times` INT(8) COMMENT 'resource used times',
+ `used_resource` VARCHAR(255) COMMENT 'used resource',
+ `release_times` INT(8) COMMENT 'resource released times',
+ `released_resource` VARCHAR(255) COMMENT 'released resource',
+ `release_time` datetime DEFAULT NULL COMMENT 'released time',
+ `used_time` datetime DEFAULT NULL COMMENT 'used time',
+ `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ KEY (`ticket_id`),
+ UNIQUE KEY `label_value_ticket_id` (`ticket_id`, `label_value`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
+````
+
+## 5. Interface Design
+Engine history management page API interface, refer to the document Add
history engine page to the management console
+
+## 6. Non-functional design
+
+### 6.1 Security
+No security issues are involved, the restful interface requires login
authentication
+
+### 6.2 Performance
+Less impact on engine life cycle performance
+
+### 6.3 Capacity
+Requires regular cleaning
+
+### 6.4 High Availability
+not involving
\ No newline at end of file
diff --git
a/docs/architecture/computation_governance_services/engine/engine_conn_metrics.md
b/docs/architecture/computation_governance_services/engine/engine_conn_metrics.md
new file mode 100644
index 0000000000..bfb89cac1f
--- /dev/null
+++
b/docs/architecture/computation_governance_services/engine/engine_conn_metrics.md
@@ -0,0 +1,76 @@
+---
+title: EngineConn Metrics reporting feature
+sidebar_position: 6
+tags: [Feature]
+---
+
+
+## 1. Functional requirements
+### 1.1 Requirement Background
+The reported information lacks engine information, and the reported resources
and progress interfaces are redundant, which reduces performance. It needs to
be aligned for optimization and adjustment, and an extension module should be
added to the reporting protocol.
+
+### 1.2 Goals
+- Added RPC protocol containing resources, progress, and additional
information, supporting reporting of these information in one request
+- Reconstruct existing resources and progress reporting links, and combine the
actions of reporting related information into one request
+
+## 2. Overall Design
+
+This requirement involves the `linkis-entrance,
linkis-computation-orchestrator, linkis-orchestrator-ecm-plugin,
linkis-computation-engineconn` modules. Add and refactor the reporting
information in the `computation-engineconn` module, and parse the information
and store it on the entry side.
+
+### 2.1 Technical Architecture
+
+The engine information reporting architecture is shown in the figure. After
the user submits the task to the entry, the entry applies to the linkismanager
for an engine.
+After applying to the engine, submit tasks to the application, and receive
regular reports of tasks (resources, progress, status). Until the task is
executed, the entry returns the final result when the user queries.
+For this modification, the engine metrics information needs to be added to the
entry into the database;
+Combine Resource and Progress interface information in Orchestrator, and add
additional information such as metrics;
+On the ComputationEngineConn side of the interactive engine, the reported
resources and progress information are combined, and engine statistics are
additionally reported.
+
+
+
+
+### 2.2 Business Architecture
+This feature involves the following function point modules:
+
+| First-level module | Second-level module | Function point |
+| :------------ | :------------ | :------------ |
+| Entrance | | Merge resource and progress interfaces; parse new engine
metrics |
+| Orchestrator | orchestrator-core | Merge resource and progress interfaces;
handle TaskRunningInfo messages |
+| Orchestrator | orchestrator-plugin-ecm | Resource and progress interfaces
for merging monitor engine information |
+| Orchestrator | computation-engineconn | Reporting interface for combining
resources and progress; new reporting engine example metrics |
+
+
+## 3. Module Design
+### Core execution flow
+-\[input] The input is the interactive engine `computation-engineconn`. When
the engine executes a task, it reports the running information
`TaskRunningInfo`, including the original `TaskProgressInfo` and
`TaskResourceInfo`, and adds the engine example information and the information
about the number of existing tasks of the engine.
+- \[Processing process] `orchestrator-plugin-ecm` is responsible for
monitoring the reporting information when the engine runs tasks, receiving the
reporting information, and generating the `TaskRunningInfoEvent` asynchronous
message,
+ Sent to `OrchestratorAsyncListenerBus` for processing. The
`TaskRunningInfoListener` registered to the `OrchestratorAsyncListener`
receives the message, triggers the `listener` method, and calls back to the
`TaskRunningInfo` callback method of the `Entrance` Job.
+ The callback method parses the resource, progress, and engine `metrancs`
information in `TaskRunningInfo` and persists them respectively.
+
+
+
+## 4. Data structure
+
+`RPC protocol TaskRunningInfo` has been added to the requirement, no db table
has been added
+
+## 5. Interface Design
+No external interface
+
+## 6. Non-functional design:
+### 6.1 Security
+RPC interface internal authentication, does not involve external security
issues
+
+### 6.2 Performance
+Combined two RPC interfaces to reduce the number of reports and improve
performance
+
+### 6.3 Capacity
+Less metrics information, no impact
+
+### 6.4 High Availability
+not involving
+
+
+
+
+
+
+
diff --git
a/docs/architecture/public_enhancement_services/context_service/content_service_cleanup.md
b/docs/architecture/public_enhancement_services/context_service/content_service_cleanup.md
new file mode 100644
index 0000000000..e12f2af08b
--- /dev/null
+++
b/docs/architecture/public_enhancement_services/context_service/content_service_cleanup.md
@@ -0,0 +1,238 @@
+---
+title: CS Cleanup Interface Features
+sidebar_position: 7
+tags: [Feature]
+---
+
+## 1. Functional requirements
+### 1.1 Background
+Before version 1.1.3, the ContextService unified context service lacked a
cleaning mechanism, and lacked the creation time, update time fields, and batch
cleaning interfaces.
+In the case of long-term accumulation, millions of data may appear, affecting
query efficiency.
+
+### 1.2 Goals
+- Modify 1ContextService` underlying library table, add creation time,
modification time, last access time fields, complete the update time of
`ContextID` and `ContextMap` related data into the database
+- Add `restful` interface for cleaning and cleaning, support batch and retail
cleaning interface according to time range and id list
+- Add the corresponding `java sdk` interface of `cs-client`
+
+## 2. Overall Design
+This requirement involves `cs-client`, `cs-persistence` and `cs-server`
modules under `ContextService`.
+Add 3 fields of the existing table in the `cs-persistence` module; add 3
`restful` interfaces in the `cs-server` module, and add 3 `sdk api` in the
`cs-client` module.
+
+### 2.1 Technical Architecture
+
+For the overall architecture of ContextService, please refer to the existing
document: [ContextService Architecture Document](overview.md)
+
+The access relationship of each module of ContestService is shown in the
following figure
+
+
+
+Table changes are made in the `cs-persistence` module. This change involves 5
tables `context_id, context_map, context_id_listener, context_key_listener,
context_history`, all of which need to add 3 fields `create_time, update_time,
access_time`. The `context_id and context_map` tables are enabled, and the
other three tables are not enabled. `create_time` adds the time before the
persistence module performs the insert operation. `update_time` and
`access_time` are actively called by the [...]
+
+The `update_time` field is updated in the cs-cache module, the ADD message is
detected when a new `context_id` is loaded from the db, and the `access_time`
is synchronized to the db at this time.
+Only the `create_time, update_time, access_time` of the `context_id` table are
recorded in the table. Subsequent search cleaning is also performed from the
context_id table.
+
+Add 3 cleanup related interfaces: `searchContextIDByTime, clearAllContextByID,
clearAllContextByTime`
+- `searchContextIDByTime` searches according to 3 time ranges and returns a
list of contextIDs
+- `clearAllContextByID` clears the content of the context_map table and
context_id table corresponding to the ID in the input contextID list
+- `clearAllContextByTime` searches according to 3 time ranges, and clears all
the contents of the context_map table and context_id table corresponding to the
searched contextID
+
+###2.2 Business Architecture
+This feature is to add related interfaces for batch query and cleanup to the
ContextService service, and to add fields such as the update time of the
underlying data table, so as to clean up expired data according to the access
situation. The function points involve the modules as shown in the table below.
+
+| First-level module | Second-level module | Function point |
+| :------------ | :------------ | :------------ |
+| linkis-ps-cs | cs-client | Added batch cleaning interface related java sdk
api interface |
+| Linkis-ps-cs | cs-server | Added restful interface related to batch cleaning
interface |
+| linkis-ps-cs | cs-persistence | Add 3 time-related fields of the underlying
table |
+
+
+##3. Module Design
+###Main execution process
+- Create ContextID. When the user creates the ContextID, the create_time will
be recorded, and this field will not be updated later
+- Update ContextID. When the user updates the ContextID, the update_time field
is updated. Note that if the update is from the cache at this time, the
access_time field will not be updated; if it is loaded from the db to the cache
and then updated with the new contextID, the access_time will be updated first,
and then the new update_time will be updated separately.
+- Query ContextID according to time. When the user queries the ContextID of
the corresponding time range, only a list of haid strings will be returned.
This interface has paging, the default is limited to 5000 pieces of data
+- Bulk cleanup of ContextIDs. All contextMap data and contextID data
corresponding to the incoming idList will be cleaned up in batches. The maximum
number of incoming arrays is 5000
+- Query and clear ContextID, first query and then batch clear
+
+The corresponding timing diagrams above are as follows:
+
+
+Two of them require additional attention:
+①The restful api in the cs-server service will encapsulate the request as a
Job and submit it to the queue and block waiting for the result. The operation
type of CLEAR is newly defined to facilitate matching to the cleanup related
interface.
+② To process the Service service of the Job in ①, the name needs to be defined
as not including the ContextID to avoid the dynamic proxy conversion of the
HighAvailable module. This conversion is only for the interface with only one
ContextID in the request, and it is meaningless and affects the batch cleanup
and batch query interface. performance.
+
+##4. Data structure
+````
+# The main involved context_id table structure is as follows, adding
create_time, update_time, expire_time fields
+CREATE TABLE `linkis_ps_cs_context_id` (
+ `id` int(11) NOT NULL AUTO_INCREMENT,
+ `user` varchar(32) DEFAULT NULL,
+ `application` varchar(32) DEFAULT NULL,
+ `source` varchar(255) DEFAULT NULL,
+ `expire_type` varchar(32) DEFAULT NULL,
+ `expire_time` datetime DEFAULT NULL,
+ `instance` varchar(128) DEFAULT NULL,
+ `backup_instance` varchar(255) DEFAULT NULL,
+ `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update
unix timestamp',
+ `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create
time',
+ `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last
access time',
+ PRIMARY KEY (`id`),
+ KEY `instance` (`instance`(128)),
+ KEY `backup_instance` (`backup_instance`(191)),
+ KEY `instance_2` (`instance`(128), `backup_instance`(128))
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+````
+
+##5. Interface Design
+###5.1 Restful interface
+
+1 Query ID interface `searchContextIDByTime`
+
+①Interface name
+GET `/api/rest_j/v1/contextservice/searchContextIDByTime`
+
+②Input parameters
+
+| Parameter name | Parameter description | Request type | Required | Data type
| schema |
+| -------- | -------- | ----- | -------- | -------- | ------ |
+|accessTimeEnd|Access end time|query|false|string|
+|accessTimeStart|Access start time|query|false|string|
+|createTimeEnd|Create end time|query|false|string|
+|createTimeStart|create time|query|false|string|
+|pageNow|page number|query|false|string|
+|pageSize|page size|query|false|string|
+|updateTimeEnd|Update end time|query|false|string|
+|updateTimeStart|Update time|query|false|string|
+
+
+③Example of output parameters
+````
+{
+ "method": "/api/contextservice/searchContextIDByTime",
+ "status": 0,
+ "message": "OK",
+ "data": {
+ "contextIDs": [
+ "8-8--cs_1_devcs_2_dev10493",
+ "8-8--cs_1_devcs_2_dev10494",
+ "8-8--cs_1_devcs_2_dev10495",
+ "8-8--cs_1_devcs_2_dev10496",
+ "8-8--cs_1_devcs_2_dev10497",
+ "8-8--cs_2_devcs_2_dev10498"
+ ]
+ }
+}
+````
+
+
+2. Clear the specified ID interface clearAllContextByID
+
+①Interface name `POST /api/rest_j/v1/contextservice/clearAllContextByID`
+② Example of input parameters
+````
+{
+"idList" : [
+"8-8--cs_1_devcs_1_dev2236"
+]
+}
+````
+
+③Example of output parameters
+````
+{
+ "method": "/api/contextservice/clearAllContextByID",
+ "status": 0,
+ "message": "OK",
+ "data": {
+ "num": "1"
+ }
+}
+````
+
+3. Clean up the interface `clearAllContextByTime` according to the time
+ ①Interface name
+ POST /api/rest_j/v1/contextservice/clearAllContextByTime
+ ② Example of input parameters
+ {
+ "createTimeStart": "2022-06-01 00:00:00",
+ "createTimeEnd": "2022-06-30 00:00:00"
+ }
+ ③Example of output parameters
+````
+{
+ "method": "/api/contextservice/clearAllContextByTime",
+ "status": 0,
+ "message": "OK",
+ "data": {
+ "num": "1"
+ }
+}
+````
+
+###5.2 JAVA SDK API
+````
+# import pom
+<dependency>
+ <groupId>org.apache.linkis</groupId>
+ <artifactId>linkis-cs-client</artifactId>
+ <version>1.1.3</version>
+</dependency>
+
+# Code reference is as follows
+
+String createTimeStart = "2022-05-26 22:04:00";
+ String createTimeEnd = "2022-06-01 24:00:00";
+
+ ContextClient contextClient =
ContextClientFactory.getOrCreateContextClient();
+
+# Interface 1 searchHAIDByTime
+ List<String> idList =
+ contextClient.searchHAIDByTime(
+ createTimeStart, createTimeEnd, null, null, null,
null, 0, 0);
+
+ for (String id : idList) {
+ System.out.println(id);
+ }
+
+ System.out.println("Got " + idList.size() + "ids.");
+
+ if (idList.size() > 0) {
+ String id1 = idList.get(0);
+ System.out.println("will clear context of id : " + id1);
+ }
+
+# Interface 2 batchClearContextByHAID
+ List<String> tmpList = new ArrayList<>();
+ tmpList.add(id1);
+ int num = contextClient.batchClearContextByHAID(tmpList);
+ System.out.println("Succeed to clear " + num + " ids.");
+
+# Interface 3 batchClearContextByTime
+ int num1 =
+ contextClient.batchClearContextByTime(
+ createTimeStart, createTimeEnd, null, null, null,
null);
+ System.out.println("Succeed to clear " + num1 + " ids by time.");
+
+````
+
+
+## 6. Non-functional design
+### 6.1 Security
+The resultful interface requires login authentication and requires an
administrator to operate. The administrator user is configured in the
properties file
+
+### 6.2 Performance
+- The query ID interface searchContextIDByTime has paging, no performance
impact
+- Clear the specified ID interface clearAllContextByID to limit the amount of
operation data, no performance impact
+- The interface clearAllContextByTime is cleared according to the time. If the
query time range is too large, the query may time out, but the task will not
fail. and the cleanup operation is a single operation and does not affect other
queries
+
+### 6.3 Capacity
+This requirement provides a time range query and batch cleaning interface,
which requires the upper-layer application that uses ContextService to actively
clean up data.
+
+### 6.4 High Availability
+The interface reuses the high availability of the ContextService microservice
itself.
+
+
+
+
+
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
