This is an automated email from the ASF dual-hosted git repository.
zhonghongsheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new 408615f Improve scaling user-manual doc (#14062)
408615f is described below
commit 408615fc1c93436328da51c881a13b029214e4df
Author: JunChao <[email protected]>
AuthorDate: Wed Dec 15 19:05:35 2021 +0800
Improve scaling user-manual doc (#14062)
* Update build.cn.md
* Update usage.cn.md
* Update usage.en.md
* Update build.en.md
---
.../shardingsphere-scaling-quick-start.cn.md | 2 +-
.../shardingsphere-scaling-quick-start.en.md | 2 +-
.../user-manual/shardingsphere-scaling/build.cn.md | 33 ++++++++++---
.../user-manual/shardingsphere-scaling/build.en.md | 27 ++++++++---
.../user-manual/shardingsphere-scaling/usage.cn.md | 54 ++++++++++++++++++----
.../user-manual/shardingsphere-scaling/usage.en.md | 51 ++++++++++++++++----
6 files changed, 136 insertions(+), 33 deletions(-)
diff --git
a/docs/document/content/quick-start/shardingsphere-scaling-quick-start.cn.md
b/docs/document/content/quick-start/shardingsphere-scaling-quick-start.cn.md
index 34adad3..39b526f 100644
--- a/docs/document/content/quick-start/shardingsphere-scaling-quick-start.cn.md
+++ b/docs/document/content/quick-start/shardingsphere-scaling-quick-start.cn.md
@@ -34,5 +34,5 @@ sh %SHARDINGSPHERE_PROXY_HOME%/bin/start.sh
- [功能#弹性伸缩](/cn/features/scaling/)：核心概念、使用规范
- [用户手册#弹性伸缩](/cn/user-manual/shardingsphere-scaling/)：运行部署、使用手册
--
[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)：弹性伸缩的DistSQL
+-
[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)：弹性伸缩的DistSQL
- [开发者手册#弹性伸缩](/cn/dev-manual/scaling/)：SPI接口及实现类
diff --git
a/docs/document/content/quick-start/shardingsphere-scaling-quick-start.en.md
b/docs/document/content/quick-start/shardingsphere-scaling-quick-start.en.md
index c0a1c37..44952ae 100644
--- a/docs/document/content/quick-start/shardingsphere-scaling-quick-start.en.md
+++ b/docs/document/content/quick-start/shardingsphere-scaling-quick-start.en.md
@@ -34,5 +34,5 @@ Please refer to [Usage
Manual](/en/user-manual/shardingsphere-scaling/usage/) fo
- [Features#Scaling](/en/features/scaling/) : Core Concept, User Norms
- [User Manual#Scaling](/en/user-manual/shardingsphere-scaling/) : Build,
Manual
--
[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/ral/#scaling)
: DistSQL for Scaling
+-
[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#scaling)
: DistSQL for Scaling
- [Dev Manual#Scaling](/en/dev-manual/scaling/) : SPI interfaces and
implementations
diff --git
a/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
b/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
index 0b9077b..9b2a6a1 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/build.cn.md
@@ -18,7 +18,12 @@ mvn clean install -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true -Drat.skip=tr
或者通过[下载页面]( https://shardingsphere.apache.org/document/current/cn/downloads/
)获取安装包。
-2. 解压缩 proxy 发布包，修改配置文件 `conf/server.yaml`，这里主要是开启 `scaling` 和 `mode` 配置：
+> Scaling还是实验性质的功能，建议使用master分支最新版本，点击此处[下载最新版本](
https://github.com/apache/shardingsphere#nightly-builds )
+
+2. 解压缩
proxy发布包，修改配置文件`conf/config-sharding.yaml`，配置`schemaName`，`dataSources`，`rules`里的`shardingAlgorithms`和`keyGenerators`，[config-sharding.yaml示例](
https://github.com/apache/shardingsphere/blob/master/examples/docker/shardingsphere-proxy/sharding/conf/config-sharding.yaml
)：
+
+3. 修改配置文件 `conf/server.yaml`，如果配置的`Mode`是`Cluster`，需要提前启动对应的注册中心。
+这里主要是开启 `scaling` 和 `mode` 配置：
```yaml
scaling:
blockQueueSize: 10000
@@ -50,13 +55,22 @@ mode:
可以通过`ScalingClusterAutoSwitchAlgorithm`接口自定义一个SPI实现，通过`ScalingDataConsistencyCheckAlgorithm`接口自定义一个SPI实现。详情请参见[开发者手册#弹性伸缩](/cn/dev-manual/scaling/)。
+overwrite字段含义为：控制配置文件是否覆盖`注册中心`元数据，一般可在测试时使用。
+
3. 启动 ShardingSphere-Proxy：
```
sh bin/start.sh
```
-4. 查看 proxy 日志 `logs/stdout.log`，确保启动成功。
+4. 查看 proxy 日志 `logs/stdout.log`，看到日志中出现：
+```
+[INFO ] 2021-12-08 15:28:48.336 [main] o.a.s.p.frontend.ShardingSphereProxy -
ShardingSphere-Proxy start success
+```
+
+确认启动成功
+
+
## 结束
@@ -67,8 +81,13 @@ sh bin/start.sh
## 应用配置项
应用现有配置项如下，相应的配置可在 `conf/server.yaml` 中修改：
-
-| 名称 | 说明 | 默认值 |
-| -------------- | -------------------------------------- | ------ |
-| blockQueueSize | 数据传输通道队列大小 | 10000 |
-| workerThread | 工作线程池大小，允许同时运行的迁移任务线程数 | 40 |
+| 一级Key | 二级Key | 三级Key
| 说明
| 默认值 |
+| ------- | ----------------------------- |
------------------------------------------------------------ |
------------------------------------------------------------ | --------- |
+| Scaling | | blockQueueSize
| 数据传输通道队列大小
| 10000 |
+| | | workerThread
| 工作线程池大小，允许同时运行的迁移任务线程数 | 40 |
+| | clusterAutoSwitchAlgorithm | type
| 开启自动检测任务是否完成及切换配置，目前系统提供了IDLE类型实现 | - |
+| | |
props:incremental-task-idle-minute-threshold |
增量同步空闲时间最大值，超过该值，进入下一状态 | 30 (分钟) |
+| | dataConsistencyCheckAlgorithm | type
|
配置数据校验算法，关闭该配置系统将不进行数据校验。目前系统提供了DEFAULT类型实现，DEFAULT算法目前支持的数据库：MySQL。其他数据库还不能打开这个配置项，相关支持还在开发中。详情请参见[开发者手册#弹性伸缩#数据一致性算法](/cn/dev-manual/scaling/#scalingdataconsistencycheckalgorithm)。
| - |
+| mode | type | Cluster
|
| - |
+| | repository | type、props |
注册中心，当前支持Zookeeper，Etcd | - |
+| | overwrite | -
| 控制配置文件是否覆盖注册中心元数据，一般可在测试时使用。 | false |
diff --git
a/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
b/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
index 91d107f..0cd8b8c 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/build.en.md
@@ -18,7 +18,12 @@ The binary packages:
Or get binary package from [download page](
https://shardingsphere.apache.org/document/current/en/downloads/ ).
-2. Unzip the proxy distribution package, modify the configuration file
`conf/server.yaml`, enable `scaling` and `mode`:
+> Scaling is an experimental feature，we suggest use master branch latest
version，click here[download latest version](
https://github.com/apache/shardingsphere#nightly-builds )
+
+2. Unzip the proxy distribution package, modify the configuration file
`conf/config-sharding.yaml`，edit `schemaName`，`dataSources`，in `rules`, you can
change `shardingAlgorithms` or`keyGenerators`，[config-sharding.yaml example](
https://github.com/apache/shardingsphere/blob/master/examples/docker/shardingsphere-proxy/sharding/conf/config-sharding.yaml
)：
+
+3. Modify the configuration file `conf/server.yaml`, if the `Mode` is
`Cluster`, please open the registry center you use before you run `proxy`.
+Now enable `scaling` and `mode`
```yaml
scaling:
blockQueueSize: 10000
@@ -56,7 +61,11 @@ You could customize an auto switch algorithm by implementing
`ScalingClusterAuto
sh bin/start.sh
```
-4. See the proxy log file `logs/stdout.log`，ensure startup successfully.
+4. See the proxy log file `logs/stdout.log`，if your log look like this:
+```
+[INFO ] 2021-12-08 15:28:48.336 [main] o.a.s.p.frontend.ShardingSphereProxy -
ShardingSphere-Proxy start success
+```
+Then `proxy` startup successfully：
## Shutdown
@@ -68,7 +77,13 @@ sh bin/stop.sh
The existing configuration items are as follows, we can modify them in
`conf/server.yaml`:
-| Name | Description
| Default value |
-| -------------- |
-----------------------------------------------------------------------------------------
| ------------- |
-| blockQueueSize | Queue size of data transmission channel
| 10000 |
-| workerThread | Worker thread pool size, the number of migration task
threads allowed to run concurrently | 40 |
+| 一级Key | 二级Key | 三级Key
| 说明
| 默认值 |
+| ------- | ----------------------------- |
------------------------------------------------------------ |
------------------------------------------------------------ | --------- |
+| Scaling | | blockQueueSize
| Queue size of data transmission channel
| 10000 |
+| | | workerThread
| Worker thread pool size, the number of migration
task threads allowed to run concurrently | 40 |
+| | clusterAutoSwitchAlgorithm | type
| Enable automatic detection of task completion and
switch configuration, currently the system provides IDLE type implementation |
- |
+| | |
props:incremental-task-idle-minute-threshold | The maximum idle
time of incremental synchronization, if it exceeds this value, enter the next
state | 30 (minutes) |
+| | dataConsistencyCheckAlgorithm | type
| Configure the dataConsistencyCheckAlgorithm,
closing the configuration system will not perform data verification. At
present, the system provides the implementation of the DEFAULT type, and the
database currently supported by the DEFAULT algorithm: MySQL. Other databases
cannot open this configuration item, and related support is still under
development.for more detail[/dev-manual/s [...]
+| mode | type | Cluster
|
| - |
+| | repository | type、props | registry center，now
support Zookeeper，Etcd | - |
+| | overwrite | -
| Control whether the configuration file covers the
registry center metadata, which can generally be used during testing. |
false |
diff --git
a/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
b/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
index eae8aa6..b9fd202 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/usage.cn.md
@@ -13,9 +13,9 @@ weight = 2
| 源端 | 目标端 |
| --------------------- | -------------------- |
-| MySQL(5.1.15 ~ 5.7.x) | MySQL |
-| PostgreSQL(9.4 ~ ) | PostgreSQL |
-| openGauss(2.1.0) | openGauss |
+| MySQL(5.1.15 ~ 5.7.x) | MySQL(5.1.15 ~ 5.7.x) |
+| PostgreSQL(9.4 ~ ) | PostgreSQL(9.4 ~ ) |
+| openGauss(2.1.0) | openGauss(2.1.0) |
**注意**：
@@ -40,8 +40,16 @@ weight = 2
还没开启`自动建表`的数据库需要手动创建分表。
### 权限要求
-
+#### MySQL
MySQL 需要开启 `binlog`，且迁移时所使用用户需要赋予 Replication 相关权限。
+执行以下命令，确认是否有开启binlog：
+
+```
+show variables like '%log_bin%';
+show variables like '%binlog%';
+```
+
+如以下显示，则说明binlog已开启
```
+-----------------------------------------+---------------------------------------+
@@ -51,7 +59,14 @@ MySQL 需要开启 `binlog`，且迁移时所使用用户需要赋予 Replicatio
| binlog_format | ROW
|
| binlog_row_image | FULL
|
+-----------------------------------------+---------------------------------------+
+```
+
+执行以下命令，查看该用户是否有迁移权限
+```
+SHOW GRANTS 'user';
+```
+```
+------------------------------------------------------------------------------+
|Grants for ${username}@${host}
|
+------------------------------------------------------------------------------+
@@ -60,6 +75,7 @@ MySQL 需要开启 `binlog`，且迁移时所使用用户需要赋予 Replicatio
+------------------------------------------------------------------------------+
```
+#### PostgreSQL
PostgreSQL 需要开启
[test_decoding](https://www.postgresql.org/docs/9.4/test-decoding.html)
### DistSQL 接口
@@ -110,7 +126,11 @@ ADD RESOURCE ds_2 (
详情请参见[RDL#数据分片](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/rdl/rdl-sharding-rule/)。
-`SHARDING TABLE RULE`支持2种类型：`TableRule`和`AutoTableRule`。对于同一个逻辑表，不能混合使用这2种格式。
+`SHARDING TABLE
RULE`支持2种类型：`TableRule`和`AutoTableRule`。对于同一个逻辑表，不能混合使用这2种格式，以下是两种分片规则的对比：
+| 类型 | AutoTableRule（自动分片） |
TableRule（自定义分片） |
+| ------------ | ------------------------------------------------------------
| ------------------------------------------------------------ |
+| 定义 | 5.x版本之后，引入的自动分片技术交由 ShardingSphere
自动管理分片，用户只需要指定分片数量和使用的数据源，无需再关心表的具体分布详情可参见：[#自动化分片算法](/cn/features/sharding/concept/sharding/#自动化分片算法)
| 需要自定义分片配置，可设置物理数据节点，数据分片、表分片等 |
+| 推荐使用场景 |
不需要关心实际表在哪个库、哪个库有几张表等问题只需考虑：SHARDING_COLUMN：设置用作分片键的列TYPE：设置分片算法和数量GENERATED_KEY：设置分布式自增ID
|
需要设置以下参数：DATANODES：物理数据节点表达式DATABASE_STRATEGY：数据库分片策略TABLE_STRATEGY：表分片策略GENERATED_KEY：设置分布式自增ID
|
`AutoTableRule`修改示例：
```sql
@@ -134,11 +154,11 @@
GENERATED_KEY(COLUMN=order_id,TYPE(NAME=snowflake,PROPERTIES("worker-id"=123)))
);
```
-**注意**：当前版本不支持通过修改`TableRule`触发迁移。
+**注意**：当前版本不支持通过修改`TableRule`触发迁移。对于同一个逻辑表，不能混合使用这2种格式。
#### 查询所有迁移任务
-详情请参见[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
+详情请参见[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
示例：
```sql
@@ -185,12 +205,26 @@ mysql> show scaling status 660152090995195904;
| RUNNING | 运行中
|
| EXECUTE_INVENTORY_TASK | 全量迁移中
|
| EXECUTE_INCREMENTAL_TASK | 增量迁移中
|
-| ALMOST_FINISHED | 基本完成
|
-| FINISHED | 已完成
|
+| FINISHED | 已完成（整个流程完成了，新规则已生效）
|
| PREPARING_FAILURE | 准备阶段失败
|
| EXECUTE_INVENTORY_TASK_FAILURE | 全量迁移阶段失败
|
| EXECUTE_INCREMENTAL_TASK_FAILURE | 增量迁移阶段失败
|
+如果`status`出现失败的情况，可以查看`proxy`的日志查看错误堆栈分析问题。
+
+
+#### 手动模式
+`Sharding-Scaling`提供了一些命令，可以手动执行。详情可见：[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
+```
+check scaling {jobId};
+```
+
+check和checkout是给手动模式使用的，就是clusterAutoSwitchAlgorithm没配置的情况。并且各自有使用条件。
+
+check做数据一致性校验，那需要判断全量和增量都完成的情况下执行才有可能校验通过。目前的默认实现是对两端的全量数据进行CRC32计算、然后比对结果
+
+checkout是让新规则生效，前置步骤没完成是不会执行成功的。
+
#### 预览新的分片规则是否生效
示例：
@@ -215,4 +249,4 @@ mysql> preview select count(1) from t_order;
```
#### 其他DistSQL
-详情请参见[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
+详情请参见[RAL#弹性伸缩](/cn/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#%E5%BC%B9%E6%80%A7%E4%BC%B8%E7%BC%A9)。
diff --git
a/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
b/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
index f04a33f..b28bf5a 100644
--- a/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
+++ b/docs/document/content/user-manual/shardingsphere-scaling/usage.en.md
@@ -13,9 +13,9 @@ The migration scene we support:
| Source | Target |
| -------------------------- | -------------------- |
-| MySQL(5.1.15 ~ 5.7.x) | MySQL |
-| PostgreSQL(9.4 ~ ) | PostgreSQL |
-| openGauss(2.1.0) | openGauss |
+| MySQL(5.1.15 ~ 5.7.x) | MySQL(5.1.15 ~ 5.7.x) |
+| PostgreSQL(9.4 ~ ) | PostgreSQL(9.4 ~ ) |
+| openGauss(2.1.0) | openGauss(2.1.0) |
**Attention**:
@@ -43,6 +43,15 @@ For RDBMS which `Create table automatically` feature is not
supported, we need t
We need to enable `binlog` for MySQL. Privileges of users scaling used should
include Replication privileges.
+Execute the following command to confirm whether binlog is turned on:
+
+```sql
+show variables like '%log_bin%';
+show variables like '%binlog%';
+```
+
+As shown below, it means that binlog has been turned on：
+
```
+-----------------------------------------+---------------------------------------+
| Variable_name | Value
|
@@ -51,7 +60,15 @@ We need to enable `binlog` for MySQL. Privileges of users
scaling used should in
| binlog_format | ROW
|
| binlog_row_image | FULL
|
+-----------------------------------------+---------------------------------------+
+```
+
+Execute the following command to check whether the user has migration
permission
+
+```sql
+SHOW GRANTS 'user';
+```
+```
+------------------------------------------------------------------------------+
|Grants for ${username}@${host}
|
+------------------------------------------------------------------------------+
@@ -110,7 +127,11 @@ ADD RESOURCE ds_2 (
Please refer to
[RDL#Sharding](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/rdl/rdl-sharding-rule/)
for more details.
-`SHARDING TABLE RULE` support two types: `TableRule` and `AutoTableRule`. For
each logic table, we could not use mixture of these two types.
+`SHARDING TABLE RULE` support two types: `TableRule` and `AutoTableRule`. For
each logic table, we could not use mixture of these two types. The following is
a comparison of the two sharding rules：
+| Tyoe | AutoTableRule | TableRule
|
+| ------------ | ------------------------------------------------------------
| ------------------------------------------------------------ |
+| Definition | After the 5.x version, the automatic sharding technology
introduced is handed over to ShardingSphere to automatically manage the
sharding. Users only need to specify the number of shards and the data source
used, and no longer need to care about the specific distribution of the table.
For details, please refer
to[#auto-sharding-algorithm](/en/features/sharding/concept/sharding/#auto-sharding-algorithm)
| Need to customize the sharding configuration, you can set up physi [...]
+| Recommended usage scenarios | No need to care about which library the actual
table is in, which library has several tables, etc. Just consider:
SHARDING_COLUMN: set the column used as the sharding key TYPE: set the sharding
algorithm and number GENERATED_KEY: set the distributed auto-increment ID |
Need to set these params: DATANODES: physical data node expression.
DATABASE_STRATEGY：database sharding strategy. TABLE_STRATEGY：table sharding
strategy. GENERATED_KEY：set the distributed au [...]
Example of alter `AutoTableRule`:
```sql
@@ -134,11 +155,11 @@
GENERATED_KEY(COLUMN=order_id,TYPE(NAME=snowflake,PROPERTIES("worker-id"=123)))
);
```
-**Attention**: We could not emit scaling job by altering `TableRule` in
current version.
+**Attention**: We could not emit scaling job by altering `TableRule` in
current version. For the same logical table, these two formats can not use at
the same time.
#### List scaling jobs
-Please refer to
[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/ral/#scaling)
for more details.
+Please refer to
[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#scaling)
for more details.
Example:
```sql
@@ -186,11 +207,25 @@ Current scaling job is finished, new sharding rule should
take effect, and not i
| EXECUTE_INVENTORY_TASK | inventory task running
|
| EXECUTE_INCREMENTAL_TASK | incremental task running
|
| ALMOST_FINISHED | almost finished
|
-| FINISHED | finished
|
+| FINISHED | finished(The whole
process is completed, and the new rules have taken effect)
|
| PREPARING_FAILURE | preparation failed
|
| EXECUTE_INVENTORY_TASK_FAILURE | inventory task failed
|
| EXECUTE_INCREMENTAL_TASK_FAILURE | incremental task failed
|
+If `status` fails, you can check the log of `proxy` to view the error stack
and analyze the problem.
+
+#### manual mode
+`Sharding-Scaling`提供了一些命令，可以手动执行。详情可见：[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#scaling)。
+```
+check scaling {jobId};
+```
+
+Check and checkout are used in manual mode, which is the case when
clusterAutoSwitchAlgorithm is not configured. And each has its own conditions
of use.
+
+The check is used for data consistency verification. It is possible to pass
the verification only when it is executed when the full amount and the
increment are completed. The current default implementation is to perform CRC32
calculation on the full amount of data at both ends, and then compare the
results.
+
+Checkout is to make the new rules take effect, and the pre-steps will not be
executed successfully if the pre-steps are not completed.
+
#### Preview new sharding rule
Example:
@@ -215,4 +250,4 @@ mysql> preview select count(1) from t_order;
```
#### Other DistSQL
-Please refer to
[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/ral/#scaling)
for more details.
+Please refer to
[RAL#Scaling](/en/user-manual/shardingsphere-proxy/usage/distsql/syntax/ral/#scaling)
for more details.
