This is an automated email from the ASF dual-hosted git repository.
zhonghongsheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new 57f59d9a435 Revise CDC and migration doc (#29594)
57f59d9a435 is described below
commit 57f59d9a4356f0c1d89fb9c85a63d7114dd9b1f8
Author: Xinze Guo <[email protected]>
AuthorDate: Fri Dec 29 15:51:09 2023 +0800
Revise CDC and migration doc (#29594)
* Revise CDC and migration doc
* Improve CDC build doc
---
.../shardingsphere-proxy/cdc/_index.cn.md | 2 +-
.../shardingsphere-proxy/cdc/_index.en.md | 2 +-
.../shardingsphere-proxy/cdc/build.cn.md | 25 ++++----
.../shardingsphere-proxy/cdc/build.en.md | 75 +++++++++++-----------
.../shardingsphere-proxy/cdc/usage.en.md | 2 +-
.../shardingsphere-proxy/migration/build.cn.md | 9 ++-
.../shardingsphere-proxy/migration/build.en.md | 9 ++-
7 files changed, 60 insertions(+), 64 deletions(-)
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.cn.md
b/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.cn.md
index d19f9e57988..c847f56c4d6 100644
--- a/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.cn.md
+++ b/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.cn.md
@@ -5,4 +5,4 @@ weight = 9
CDC（Change Data Capture）增量数据捕捉。CDC 可以监控 ShardingSphere-Proxy
的存储节点中的数据变化，捕捉到数据操作事件，过滤并提取有用信息，最终将这些变化数据发送到指定的目标上。
-CDC 可以用于数据同步，数据备份和恢复等方面。通常情况下，目前支持 openGauss、MySQL 和 PostgreSQL。
+CDC 可以用于数据同步，数据备份和恢复等方面，目前支持 openGauss、MySQL 和 PostgreSQL。
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.en.md
b/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.en.md
index 5052863886e..f16c7569120 100644
--- a/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.en.md
+++ b/docs/document/content/user-manual/shardingsphere-proxy/cdc/_index.en.md
@@ -5,4 +5,4 @@ weight = 9
CDC (Change Data Capture) captures incremental data changes. CDC can monitor
data changes in the storage nodes of ShardingSphere-Proxy, capture data
operation events, filter and extract useful information, and finally send these
changed data to a specified target.
-CDC can be used for data synchronization, data backup and recovery, etc. Under
normal circumstances, it currently supports openGauss, MySQL, and PostgreSQL.
\ No newline at end of file
+CDC can be used for data synchronization, data backup and recovery, it
currently supports openGauss, MySQL, and PostgreSQL.
\ No newline at end of file
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.cn.md
b/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.cn.md
index 6ffeb3b4040..93be4fe27d3 100644
--- a/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.cn.md
+++ b/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.cn.md
@@ -88,7 +88,6 @@ authority:
props:
system-log-level: INFO
- check-table-metadata-enabled: false
proxy-default-port: 3307 # Proxy default port
cdc-server-port: 33071 # CDC Server 端口，必须配置
proxy-frontend-database-protocol-type: openGauss # 和后端数据库的类型一致
@@ -96,7 +95,7 @@ props:
2. 引入 JDBC 驱动。
-proxy 已包含 PostgreSQL、openGauss JDBC 驱动。
+proxy 已包含 PostgreSQL JDBC 和 openGauss JDBC 驱动。
如果后端连接以下数据库，请下载相应 JDBC 驱动 jar 包，并将其放入 `${shardingsphere-proxy}/ext-lib` 目录。
@@ -164,7 +163,7 @@ STREAM_CHANNEL
(TYPE(NAME='MEMORY',PROPERTIES('block-queue-size'='2000')))
```sql
ALTER STREAMING RULE (
READ( -- 数据读取配置。如果不配置则部分参数默认生效。
- WORKER_THREAD=20, -- 影响全量、增量任务，从源端摄取数据的线程池大小。不配置则使用默认值。需要确保该值不低于分库的数量
+ WORKER_THREAD=20, -- 影响全量、增量任务，从源端摄取数据的线程池大小，不配置则使用默认值，需要确保该值不低于分库的数量。
BATCH_SIZE=1000, -- 影响全量、增量任务，一次查询操作返回的最大记录数。如果一个事务中的数据量大于该值，增量情况下可能超过设定的值。
SHARDING_SIZE=10000000, -- 影响全量任务，存量数据分片大小。如果不配置则使用默认值。
RATE_LIMITER ( -- 影响全量、增量任务，限流算法。如果不配置则不限流。
@@ -211,13 +210,13 @@ CDC Client 不需要额外部署，只需要通过 maven 引入 CDC Client 的
`org.apache.shardingsphere.data.pipeline.cdc.client.CDCClient` 是 CDC Client
的入口类，用户可以通过该类和 CDC Server 进行交互。主要的和新方法如下。
-| 方法名
| 返回值 | 说明
|
-|-----------------------------------------------------------------------------------------------------------------------------|-------------|---------------------------------------------------------------------------------------------------------|
-| connect(Consumer<List<Record>> dataConsumer, ExceptionHandler
exceptionHandler, ServerErrorResultHandler errorResultHandler | void |
和服务端进行连接，连接的时候需要指定 <br/>1. 数据的消费处理逻辑 <br/>2. 消费时候的异常处理逻辑 <br/>3. 服务端错误的异常处理逻辑
|
-| login(CDCLoginParameter parameter)
| void | CDC登陆，参数
<br/>username：用户名 <br/>password：密码
|
-| startStreaming(StartStreamingParameter parameter)
| streamingId | 开启 CDC 订阅，
StartStreamingParameter 参数 <br/> database：逻辑库名称 <br/> schemaTables：订阅的表名 <br/>
full：是否订阅全量数据 |
-| restartStreaming(String streamingId)
| void | 重启订阅
|
-| stopStreaming(String streamingId)
| void | 停止订阅
|
-| dropStreaming(String streamingId)
| void | 删除订阅
|
-| await()
| void | 阻塞 CDC 线程，等待
channel 关闭
|
-| close()
| void | 关闭 channel，流程结束
|
+| 方法名
| 返回值 | 说明
|
+|-----------------------------------------------------------------------------------------------------------------------------|-------------|--------------------------------------------------------------------------------------------------------------|
+| connect(Consumer<List<Record>> dataConsumer, ExceptionHandler
exceptionHandler, ServerErrorResultHandler errorResultHandler | void |
和服务端进行连接，连接的时候需要指定 <br/>1. 数据的消费处理逻辑 <br/>2. 消费时候的异常处理逻辑 <br/>3. 服务端错误的异常处理逻辑
|
+| login(CDCLoginParameter parameter)
| void | CDC登陆，参数
<br/>username：用户名 <br/>password：密码
|
+| startStreaming(StartStreamingParameter parameter)
| streamingId | 开启 CDC 订阅 <br/>
StartStreamingParameter 参数 <br/> database：逻辑库名称 <br/> schemaTables：订阅的表名 <br/>
full：是否订阅全量数据 |
+| restartStreaming(String streamingId)
| void | 重启订阅
|
+| stopStreaming(String streamingId)
| void | 停止订阅
|
+| dropStreaming(String streamingId)
| void | 删除订阅
|
+| await()
| void | 阻塞 CDC 线程，等待
channel 关闭
|
+| close()
| void | 关闭 channel，流程结束
|
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.en.md
b/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.en.md
index 7a6e1916791..748e36c66dc 100644
--- a/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.en.md
+++ b/docs/document/content/user-manual/shardingsphere-proxy/cdc/build.en.md
@@ -87,7 +87,6 @@ authority:
props:
system-log-level: INFO
- check-table-metadata-enabled: false
proxy-default-port: 3307 # Proxy default port.
cdc-server-port: 33071 # CDC Server port, must be configured
proxy-frontend-database-protocol-type: openGauss # Consistent with the type
of backend database
@@ -95,13 +94,13 @@ props:
2. Introduce JDBC driver.
-Proxy already includes PostgreSQL, openGauss JDBC driver.
+Proxy has included JDBC driver of PostgreSQL and openGauss.
-If the backend connects to the following databases, please download the
corresponding JDBC driver jar package and put it in the
`${shardingsphere-proxy}/ext-lib` directory.
+If the backend is connected to the following databases, download the
corresponding JDBC driver jar package and put it into the
`${shardingsphere-proxy}/ext-lib` directory.
-| Database | JDBC Driver
|
-|-----------|---------------------------------------------------------------------------------------------------------------------------------|
-| MySQL |
[mysql-connector-java-8.0.31.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.31/)
|
+| Database | JDBC Driver
|
+|----------|------------------------------------------------------------------------------------------------------|
+| MySQL |
[mysql-connector-java-8.0.31.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.31/)
|
4. Start ShardingSphere-Proxy:
@@ -109,15 +108,15 @@ If the backend connects to the following databases,
please download the correspo
sh bin/start.sh
```
-5. View the proxy log `logs/stdout.log`, and see in the log:
+5. View the proxy log `logs/stdout.log`. If you see the following statements:
```
[INFO ] [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy
Cluster mode started successfully
```
-Confirm successful startup.
+The startup will have been successful.
-6. Configure CDC task synchronization configuration as needed
+6. Configure CDC on demand.
6.1. Query configuration.
@@ -135,11 +134,11 @@ The default configuration is as follows:
+--------------------------------------------------------------+--------------------------------------+-------------------------------------------------------+
```
-6.2. Modify configuration (optional).
+6.2. Alter configuration (optional).
-Because the streaming rule has a default value, no creation is required, only
the ALTER statement is provided.
+Since the streaming rule has default values, there is no need to create it,
only the `ALTER` statement is provided.
-Complete configuration DistSQL example:
+A completely configured DistSQL is as follows.
```sql
ALTER STREAMING RULE (
@@ -162,32 +161,32 @@ Configuration item description:
```sql
ALTER STREAMING RULE (
-READ( -- Data reading configuration. If not configured, some parameters will
take effect by default.
- WORKER_THREAD=20, -- Affects full and incremental tasks, the size of the
thread pool for fetching data from the source end. If not configured, the
default value will be used. It needs to ensure that this value is not lower
than the number of database shards
- BATCH_SIZE=1000, -- Affects full and incremental tasks, the maximum number
of records returned by a query operation. If the amount of data in a
transaction is greater than this value, the incremental situation may exceed
the set value.
- SHARDING_SIZE=10000000, -- Affects full tasks, the size of stock data
sharding. If not configured, the default value will be used.
- RATE_LIMITER ( -- Affects full and incremental tasks, rate limiting
algorithm. If not configured, no rate limiting.
- TYPE( -- Algorithm type. Optional: QPS
+READ( -- Data reading configuration. If it is not configured, part of the
parameters will take effect by default.
+ WORKER_THREAD=20, -- Affects full and incremental tasks, obtain the thread
pool size of all the data from the source side. If it is not configured, the
default value is used. It needs to ensure that this value is not lower than the
number of database shards.
+ BATCH_SIZE=1000, -- Affects full and incremental tasks, the maximum number
of records returned by a query operation. If it is not configured, the default
value is used. If the amount of data in a transaction is greater than this
value, the incremental situation may exceed the set value.
+ SHARDING_SIZE=10000000, -- Affects full tasks, sharding size of all the
data. If it is not configured, the default value is used.
+ RATE_LIMITER ( -- Affects full and incremental tasks, traffic limit
algorithm. If it is not configured, traffic is not limited.
+ TYPE( -- Algorithm type. Option: QPS
NAME='QPS',
- PROPERTIES( -- Algorithm properties
+ PROPERTIES( -- Algorithm property
'qps'='500'
)))
),
-WRITE( -- Data writing configuration. If not configured, some parameters will
take effect by default.
- WORKER_THREAD=20, -- Affects full and incremental tasks, the size of the
thread pool for writing data to the target end. If not configured, the default
value will be used.
- BATCH_SIZE=1000, -- Affects full and incremental tasks, the maximum number
of records for a batch write operation in a stock task. If not configured, the
default value will be used. If the amount of data in a transaction is greater
than this value, the incremental situation may exceed the set value.
- RATE_LIMITER ( -- Rate limiting algorithm. If not configured, no rate
limiting.
- TYPE( -- Algorithm type. Optional: TPS
+WRITE( -- Data writing configuration. If it is not configured, part of the
parameters will take effect by default.
+ WORKER_THREAD=20, -- Affects full and incremental tasks, the size of the
thread pool on which data is written into the target side. If it is not
configured, the default value is used.
+ BATCH_SIZE=1000, -- Affects full and incremental tasks, the maximum number
of records for a batch write operation. If it is not configured, the default
value is used. If the amount of data in a transaction is greater than this
value, the incremental situation may exceed the set value.
+ RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic
is not limited.
+ TYPE( -- Algorithm type. Option: TPS
NAME='TPS',
- PROPERTIES( -- Algorithm properties
+ PROPERTIES( -- Algorithm property.
'tps'='2000'
)))
),
-STREAM_CHANNEL ( -- Data channel, connecting producers and consumers, used for
read and write links. If not configured, the MEMORY type is used by default.
-TYPE( -- Algorithm type. Optional: MEMORY
+STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used
for reading and writing procedures. If it is not configured, the MEMORY type is
used by default.
+TYPE( -- Algorithm type. Option: MEMORY
NAME='MEMORY',
-PROPERTIES( -- Algorithm properties
-'block-queue-size'='2000' -- Property: Blocking queue size
+PROPERTIES( -- Algorithm property
+'block-queue-size'='2000' -- Property: blocking queue size.
)))
);
```
@@ -210,13 +209,13 @@ If necessary, users can also implement a CDC Client
themselves to consume data a
`org.apache.shardingsphere.data.pipeline.cdc.client.CDCClient` is the entry
class of the CDC Client. Users can interact with the CDC Server through this
class. The main new methods are as follows.
-| Method Name
| Return Value | Description
|
-|-----------------------------------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Method Name
| Return Value | Description
|
+|-----------------------------------------------------------------------------------------------------------------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| connect(Consumer<List<Record>> dataConsumer, ExceptionHandler
exceptionHandler, ServerErrorResultHandler errorResultHandler | void |
Connect with the server, when connecting, you need to specify <br/>1. Data
consumption processing function <br/>2. Exception handling logic during
consumption <br/>3. Server error exception handling function |
-| login(CDCLoginParameter parameter)
| void | CDC login,
parameters <br/>username: username <br/>password: password
|
-| startStreaming(StartStreamingParameter parameter)
| streamingId | Start CDC
subscription, StartStreamingParameter parameters <br/> database: logical
database name <br/> schemaTables: subscribed table name <br/> full: whether to
subscribe to full data |
-| restartStreaming(String streamingId)
| void | Restart
subscription
|
-| stopStreaming(String streamingId)
| void | Stop
subscription
|
-| dropStreaming(String streamingId)
| void | Delete
subscription
|
-| await()
| void | Block the CDC
thread and wait for the channel to close
|
-| close()
| void | Close the
channel, the process ends
|
+| login(CDCLoginParameter parameter)
| void | CDC login,
parameters <br/>username: username <br/>password: password
|
+| startStreaming(StartStreamingParameter parameter)
| streamingId | Start CDC
subscription<br/> StartStreamingParameter parameters <br/> database: logical
database name <br/> schemaTables: subscribed table name <br/> full: whether to
subscribe to full data |
+| restartStreaming(String streamingId)
| void | Restart
subscription
|
+| stopStreaming(String streamingId)
| void | Stop
subscription
|
+| dropStreaming(String streamingId)
| void | Delete
subscription
|
+| await()
| void | Block the CDC
thread and wait for the channel to close
|
+| close()
| void | Close the
channel, the process ends
|
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/cdc/usage.en.md
b/docs/document/content/user-manual/shardingsphere-proxy/cdc/usage.en.md
index 1cb1bc89618..0090daa0409 100644
--- a/docs/document/content/user-manual/shardingsphere-proxy/cdc/usage.en.md
+++ b/docs/document/content/user-manual/shardingsphere-proxy/cdc/usage.en.md
@@ -1,5 +1,5 @@
+++
-title = "User Manual"
+title = "Manual"
weight = 2
+++
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/migration/build.cn.md
b/docs/document/content/user-manual/shardingsphere-proxy/migration/build.cn.md
index 9d65053a532..865347aef70 100644
---
a/docs/document/content/user-manual/shardingsphere-proxy/migration/build.cn.md
+++
b/docs/document/content/user-manual/shardingsphere-proxy/migration/build.cn.md
@@ -37,14 +37,13 @@ mode:
3. 引入 JDBC 驱动。
-proxy 已包含 PostgreSQL JDBC 驱动。
+proxy 已包含 PostgreSQL JDBC 和 openGauss JDBC 驱动。
如果后端连接以下数据库，请下载相应 JDBC 驱动 jar 包，并将其放入 `${shardingsphere-proxy}/ext-lib` 目录。
-| 数据库 | JDBC 驱动
|
-|-----------|---------------------------------------------------------------------------------------------------------------------------------------|
-| MySQL | [mysql-connector-java-5.1.49.jar](
https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar
) |
-| openGauss | [opengauss-jdbc-3.0.0.jar](
https://repo1.maven.org/maven2/org/opengauss/opengauss-jdbc/3.0.0/opengauss-jdbc-3.0.0.jar
) |
+| 数据库 | JDBC 驱动
|
+|-------|------------------------------------------------------------------------------------------------------|
+| MySQL |
[mysql-connector-java-8.0.31.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.31/)
|
如果是异构迁移，源端支持范围更广的数据库。JDBC 驱动处理方式同上。
diff --git
a/docs/document/content/user-manual/shardingsphere-proxy/migration/build.en.md
b/docs/document/content/user-manual/shardingsphere-proxy/migration/build.en.md
index 8321d56efd7..4f6e68844f3 100644
---
a/docs/document/content/user-manual/shardingsphere-proxy/migration/build.en.md
+++
b/docs/document/content/user-manual/shardingsphere-proxy/migration/build.en.md
@@ -37,14 +37,13 @@ mode:
3. Introduce JDBC driver.
-Proxy has included JDBC driver of PostgreSQL.
+Proxy has included JDBC driver of PostgreSQL and openGauss.
If the backend is connected to the following databases, download the
corresponding JDBC driver jar package and put it into the
`${shardingsphere-proxy}/ext-lib` directory.
-| Database | JDBC Driver
|
-|-----------|---------------------------------------------------------------------------------------------------------------------------------------|
-| MySQL | [mysql-connector-java-5.1.49.jar](
https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar
) |
-| openGauss | [opengauss-jdbc-3.0.0.jar](
https://repo1.maven.org/maven2/org/opengauss/opengauss-jdbc/3.0.0/opengauss-jdbc-3.0.0.jar
) |
+| Database | JDBC Driver
|
+|----------|------------------------------------------------------------------------------------------------------|
+| MySQL |
[mysql-connector-java-8.0.31.jar](https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.31/)
|
If you are migrating to a heterogeneous database, then you could use more
types of database. Introduce JDBC driver as above too.
