This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 5882a6b324 [INLONG-689][Doc] Add incremental snapshot doc for oracle
cdc (#690)
5882a6b324 is described below
commit 5882a6b324912498271cc2be969e7075086f2123
Author: emhui <[email protected]>
AuthorDate: Fri Feb 24 15:39:15 2023 +0800
[INLONG-689][Doc] Add incremental snapshot doc for oracle cdc (#690)
---
docs/data_node/extract_node/oracle-cdc.md | 64 +++++++++++++++++
.../current/data_node/extract_node/oracle-cdc.md | 84 +++++++++++++++++++---
2 files changed, 138 insertions(+), 10 deletions(-)
diff --git a/docs/data_node/extract_node/oracle-cdc.md
b/docs/data_node/extract_node/oracle-cdc.md
index 0299f8551d..4d868c10d7 100644
--- a/docs/data_node/extract_node/oracle-cdc.md
+++ b/docs/data_node/extract_node/oracle-cdc.md
@@ -176,6 +176,7 @@ Overall, the steps for configuring CDB database is quite
similar to non-CDB data
GRANT SELECT ANY TRANSACTION TO flinkuser CONTAINER=ALL;
GRANT LOGMINING TO flinkuser CONTAINER=ALL;
GRANT CREATE TABLE TO flinkuser CONTAINER=ALL;
+ -- Don’t need to execute this statement, If you set
'scan.incremental.snapshot.enabled=true' (default).
GRANT LOCK ANY TABLE TO flinkuser CONTAINER=ALL;
GRANT CREATE SEQUENCE TO flinkuser CONTAINER=ALL;
@@ -335,6 +336,69 @@ TODO: It will be supported in the future.
<td>Boolean</td>
<td>Whether to enable multiple schema and table migration. If it is'
true ', Oracle Extract Node will compress the physical field of the table into
a special meta field 'data_canal' in the format of 'canal json'.</td>
</tr>
+ <tr>
+ <td>scan.incremental.snapshot.enabled</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>true</td>
+ <td>Boolean</td>
+ <td>Incremental snapshot is a new mechanism to read snapshot of a
table. Compared to the old snapshot mechanism, the incremental snapshot has
many advantages, including: (1) source can be parallel during snapshot reading,
(2) source can perform checkpoints in the chunk granularity during snapshot
reading, (3) source doesn't need to acquire ROW SHARE MODE lock before snapshot
reading.</td>
+ </tr>
+ <tr>
+ <td>scan.incremental.snapshot.chunk.size</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>8096</td>
+ <td>Integer</td>
+ <td>The chunk size (number of rows) of table snapshot, captured tables
are split into multiple chunks when read the snapshot of table.</td>
+ </tr>
+ <tr>
+ <td>scan.snapshot.fetch.size</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>1024</td>
+ <td>Integer</td>
+ <td>The maximum fetch size for per poll when read table snapshot.</td>
+ </tr>
+ <tr>
+ <td>connect.max-retries</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>3</td>
+ <td>Integer</td>
+ <td>The max retry times that the connector should retry to build Oracle
database server connection.</td>
+ </tr>
+ <tr>
+ <td>chunk-meta.group.size</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>1000</td>
+ <td>Integer</td>
+ <td>The group size of chunk meta, if the meta size exceeds the group
size, the meta will be divided into multiple groups.</td>
+ </tr>
+ <tr>
+ <td>connect.timeout</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>30s</td>
+ <td>Duration</td>
+ <td>The maximum time that the connector should wait after trying to
connect to the Oracle database server before timing out.</td>
+ </tr>
+ <tr>
+ <td>chunk-key.even-distribution.factor.lower-bound</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>0.05d</td>
+ <td>Double</td>
+ <td>The lower bound of chunk key distribution factor. The distribution
factor is used to determine whether the table is evenly distribution or not.
The table chunks would use evenly calculation optimization when the data
distribution is even, and the query for splitting would happen when it is
uneven. The distribution factor could be calculated by (MAX(id) - MIN(id) + 1)
/ rowCount.</td>
+ </tr>
+ <tr>
+ <td>chunk-key.even-distribution.factor.upper-bound</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>1000.0d</td>
+ <td>Double</td>
+ <td>The upper bound of chunk key distribution factor. The distribution
factor is used to determine whether the table is evenly distribution or not.
The table chunks would use evenly calculation optimization when the data
distribution is even, and the query for splitting would happen when it is
uneven. The distribution factor could be calculated by (MAX(id) - MIN(id) + 1)
/ rowCount.</td>
+ </tr>
+ <tr>
+ <td>connection.pool.size</td>
+ <td>optional</td>
+ <td style={{wordWrap: 'break-word'}}>20</td>
+ <td>Integer</td>
+ <td>The connection pool size.</td>
+ </tr>
</tbody>
</table>
</div>
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/oracle-cdc.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/oracle-cdc.md
index d1740608a9..d3c2a1d2ed 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/oracle-cdc.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/oracle-cdc.md
@@ -176,6 +176,7 @@ Oracle Extract 节点允许从 Oracle 数据库中读取快照数据和增量数
GRANT SELECT ANY TRANSACTION TO flinkuser CONTAINER=ALL;
GRANT LOGMINING TO flinkuser CONTAINER=ALL;
GRANT CREATE TABLE TO flinkuser CONTAINER=ALL;
+ -- 如果设置 scan.increative.snapshot.enabled=true （默认值），则不需要执行这条语句
GRANT LOCK ANY TABLE TO flinkuser CONTAINER=ALL;
GRANT CREATE SEQUENCE TO flinkuser CONTAINER=ALL;
@@ -249,63 +250,63 @@ TODO: 将在未来支持此功能。
<tbody>
<tr>
<td>connector</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>指定要使用的连接器，这里应该是 <code>'oracle-cdc-inlong'</code>。</td>
</tr>
<tr>
<td>hostname</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>Oracle 数据库服务器的 IP 地址或主机名。</td>
</tr>
<tr>
<td>username</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>连接到 Oracle 数据库服务器时要使用的 Oracle 数据库的名称。</td>
</tr>
<tr>
<td>password</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>连接到 Oracle 数据库服务器时使用的密码。</td>
</tr>
<tr>
<td>database-name</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>要监视的 Oracle 服务器的数据库名称。</td>
</tr>
<tr>
<td>schema-name</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>要监视的 Oracle 数据库的 Schema 名称。</td>
</tr>
<tr>
<td>table-name</td>
- <td>required</td>
+ <td>必选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>要监视的 Oracle
数据库的表名。格式为<i><schema_name>.<table_name></i></td>
</tr>
<tr>
<td>port</td>
- <td>optional</td>
+ <td>可选</td>
<td style={{wordWrap: 'break-word'}}>1521</td>
<td>Integer</td>
<td>Oracle 数据库服务器的整数端口号。</td>
</tr>
<tr>
<td>scan.startup.mode</td>
- <td>optional</td>
+ <td>可选</td>
<td style={{wordWrap: 'break-word'}}>initial</td>
<td>String</td>
<td>
@@ -315,7 +316,7 @@ Oracle CDC 消费者的可选启动模式，有效枚举为"initial"
</tr>
<tr>
<td>debezium.*</td>
- <td>optional</td>
+ <td>可选</td>
<td style={{wordWrap: 'break-word'}}>(none)</td>
<td>String</td>
<td>将 Debezium 的属性整合到用于从 Oracle 服务器捕获数据更改的 Debezium Embedded Engine。
@@ -336,6 +337,69 @@ Oracle CDC 消费者的可选启动模式，有效枚举为"initial"
<td>Boolean</td>
<td>是否开启多模式、表同步功能，如果为 'true'，Oracle Extract Node 则将表的物理字段压缩成
'canal-json' 格式的特殊元字段 'data_canal'。</td>
</tr>
+ <tr>
+ <td>scan.incremental.snapshot.enabled</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>true</td>
+ <td>Boolean</td>
+ <td>增量快照是一种读取表快照的新机制。与旧的快照机制相比，增量快照具有许多优点，包括：（1）在快照读取期间 Source
可以是并行的，（2）Source 可以在快照读取过程中执行 Chunk 粒度中的检查点，（3）Source 不需要在快照读取之前获取 ROW SHARE
MODE 锁。</td>
+ </tr>
+ <tr>
+ <td>scan.incremental.snapshot.chunk.size</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>8096</td>
+ <td>Integer</td>
+ <td>表快照的块大小（行数），读取表的快照时，表的快照被分成多个块。</td>
+ </tr>
+ <tr>
+ <td>scan.snapshot.fetch.size</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>1024</td>
+ <td>Integer</td>
+ <td>读取表快照时每次轮询的最大获取大小。</td>
+ </tr>
+ <tr>
+ <td>connect.max-retries</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>3</td>
+ <td>Integer</td>
+ <td>连接器应重试以建立 Oracle 数据库服务器连接的最大重试次数。</td>
+ </tr>
+ <tr>
+ <td>chunk-meta.group.size</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>1000</td>
+ <td>Integer</td>
+ <td>Chunk meta 组大小，如果 meta 大小超过组大小，则 meta 将被分成多个组。</td>
+ </tr>
+ <tr>
+ <td>connect.timeout</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>30s</td>
+ <td>Duration</td>
+ <td>连接器在尝试连接到 Oracle 数据库服务器后在超时之前应等待的最长时间。</td>
+ </tr>
+ <tr>
+ <td>chunk-key.even-distribution.factor.lower-bound</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>0.05d</td>
+ <td>Double</td>
+ <td>Chunk Key
分布因子的下限。分布系数用于确定表格是否均匀分布。当数据分布均匀时，表块将均匀地使用计算优化，当数据分布不均匀时，将进行拆分查询。分布因子可以通过（MAX（id）-MIN（id）+1）/rowCount计算。</td>
+ </tr>
+ <tr>
+ <td>chunk-key.even-distribution.factor.upper-bound</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>1000.0d</td>
+ <td>Double</td>
+ <td>Chunk Key
分布因子的上限。分布系数用于确定表格是否均匀分布。当数据分布均匀时，表块将均匀地使用计算优化，当数据分布不均匀时，将进行拆分查询。分布因子可以通过（MAX（id）-MIN（id）+1）/rowCount计算。</td>
+ </tr>
+ <tr>
+ <td>connection.pool.size</td>
+ <td>可选</td>
+ <td style={{wordWrap: 'break-word'}}>20</td>
+ <td>Integer</td>
+ <td>连接池大小。</td>
+ </tr>
</tbody>
</table>
</div>
