This is an automated email from the ASF dual-hosted git repository.
critas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iotdb-docs.git
The following commit(s) were added to refs/heads/main by this push:
new 70b97a6b add data delete in table model data sync (#675)
70b97a6b is described below
commit 70b97a6b251a4c3a7663ab9740caea117bd4491d
Author: leto-b <[email protected]>
AuthorDate: Fri Mar 28 10:10:01 2025 +0800
add data delete in table model data sync (#675)
* add data delete in table model data sync
* add schema sync and source.mode.double-living parameter
---
.../Master/Table/User-Manual/Data-Sync_apache.md | 66 ++++++++++++++++++----
.../Master/Table/User-Manual/Data-Sync_timecho.md | 66 ++++++++++++++++++----
.../latest-Table/User-Manual/Data-Sync_apache.md | 66 ++++++++++++++++++----
.../latest-Table/User-Manual/Data-Sync_timecho.md | 66 ++++++++++++++++++----
.../Master/Table/User-Manual/Data-Sync_apache.md | 66 ++++++++++++++++++----
.../Master/Table/User-Manual/Data-Sync_timecho.md | 66 ++++++++++++++++++----
.../latest-Table/User-Manual/Data-Sync_apache.md | 66 ++++++++++++++++++----
.../latest-Table/User-Manual/Data-Sync_timecho.md | 66 ++++++++++++++++++----
8 files changed, 432 insertions(+), 96 deletions(-)
diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
index 099a8a20..1d2f5096 100644
--- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
+++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
@@ -35,7 +35,47 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the
source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in
the `sink` section of the SQL statement.
-By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved.
+By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved. Currently, data
synchronization supports the synchronization of the following information, and
you can select the synchronization scope when creating a synchronization task
(the default is data.insert, which means synchronizing newly written data):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>Synchronization Scope</th>
+ <th>Synchronization Content </th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>All scopes</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(Data)</td>
+ <td>insert</td>
+ <td>Synchronize newly written data</td>
+ </tr>
+ <tr>
+ <td>delete</td>
+ <td>Synchronize deleted data</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema</td>
+ <td>database</td>
+ <td>Synchronize database creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>table</td>
+ <td>Synchronize table creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>TTL</td>
+ <td>Synchronize the data retention time</td>
+ </tr>
+ <tr>
+ <td>auth</td>
+ <td>-</td>
+ <td>Synchronize user permissions and access control</td>
+ </tr>
+</tbody>
+</table>
### 1.2 Functional Limitations and Notes
@@ -468,17 +508,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source parameter
-| **Parameter** | **Description**
| **Value Range** |
**Required** | **Default Value** |
-| :----------------------- |
:----------------------------------------------------------- |
:----------------------------------------------------------- | :----------- |
:---------------------------------------------------------- |
-| source | iotdb-source
| String: iotdb-source | Yes
| - |
-| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
-| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. | Boolean: true / false [...]
-| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
-| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched. | String:
Database name or database regular expression pattern string, which can match
uncreated or non - existent dat [...]
-| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched. | String: Data table
name or data table regular expression pattern string, which can be uncreated or
non - existent tables. | No [...]
-| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| forwarding-pipe-requests | Specifies whether to forward data that was
synchronized via the pipe to external clusters. Typically used for setting up
**active-active clusters**. In active-active cluster mode, this parameter
should be set to `false` to prevent **infinite circular synchronization**. |
Boolean: true / false | No |
true |
+| **Parameter** | **Description**
[...]
+| :-----------------------
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| source | iotdb-source
[...]
+| inclusion | Used to specify the range of data to be
synchronized in the data synchronization task, including data, schema and auth
[...]
+| inclusion.exclusion | Used to exclude specific operations from the
range specified by inclusion, reducing the amount of data synchronized
[...]
+| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
+| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. [...]
+| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
+| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched.
[...]
+| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched.
[...]
+| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| mode.double-living | Whether to enable full dual-active mode. When
enabled, the system will ignore the `-sql_dialect` connection method to capture
all tree-table model data and not forward data synced from another pipe (to
avoid circular synchronization).
[...]
> 💎 **Note:** The difference between the values of true and false for the
> data extraction mode `mode.streaming`
>
diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
index e8383920..4a7a6ff1 100644
--- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
+++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
@@ -34,7 +34,47 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the
source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in
the `sink` section of the SQL statement.
-By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved.
+By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved.Currently, data
synchronization supports the synchronization of the following information, and
you can select the synchronization scope when creating a synchronization task
(the default is data.insert, which means synchronizing newly written data):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>Synchronization Scope</th>
+ <th>Synchronization Content </th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>All scopes</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(Data)</td>
+ <td>insert</td>
+ <td>Synchronize newly written data</td>
+ </tr>
+ <tr>
+ <td>delete</td>
+ <td>Synchronize deleted data</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema</td>
+ <td>database</td>
+ <td>Synchronize database creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>table</td>
+ <td>Synchronize table creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>TTL</td>
+ <td>Synchronize the data retention time</td>
+ </tr>
+ <tr>
+ <td>auth</td>
+ <td>-</td>
+ <td>Synchronize user permissions and access control</td>
+ </tr>
+</tbody>
+</table>
### 1.2 Functional Limitations and Notes
@@ -503,17 +543,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source parameter
-| **Parameter** | **Description**
| **Value Range** |
**Required** | **Default Value** |
-| :----------------------- |
:----------------------------------------------------------- |
:----------------------------------------------------------- | :----------- |
:---------------------------------------------------------- |
-| source | iotdb-source
| String: iotdb-source | Yes
| - |
-| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
-| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. | Boolean: true / false [...]
-| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
-| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched. | String:
Database name or database regular expression pattern string, which can match
uncreated or non - existent dat [...]
-| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched. | String: Data table
name or data table regular expression pattern string, which can be uncreated or
non - existent tables. | No [...]
-| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| forwarding-pipe-requests | Specifies whether to forward data that was
synchronized via the pipe to external clusters. Typically used for setting up
**active-active clusters**. In active-active cluster mode, this parameter
should be set to `false` to prevent **infinite circular synchronization**. |
Boolean: true / false | No |
true |
+| **Parameter** | **Description**
[...]
+| :-----------------------
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| source | iotdb-source
[...]
+| inclusion | Used to specify the range of data to be
synchronized in the data synchronization task, including data,schema and auth
[...]
+| inclusion.exclusion | Used to exclude specific operations from the
range specified by inclusion, reducing the amount of data synchronized
[...]
+| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
+| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. [...]
+| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
+| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched.
[...]
+| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched.
[...]
+| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| mode.double-living | Whether to enable full dual-active mode. When
enabled, the system will ignore the `-sql_dialect` connection method to capture
all tree-table model data and not forward data synced from another pipe (to
avoid circular synchronization).
[...]
> 💎 **Note:** The difference between the values of true and false for the
> data extraction mode `mode.streaming`
>
diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
index 099a8a20..1d2f5096 100644
--- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
+++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
@@ -35,7 +35,47 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the
source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in
the `sink` section of the SQL statement.
-By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved.
+By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved. Currently, data
synchronization supports the synchronization of the following information, and
you can select the synchronization scope when creating a synchronization task
(the default is data.insert, which means synchronizing newly written data):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>Synchronization Scope</th>
+ <th>Synchronization Content </th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>All scopes</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(Data)</td>
+ <td>insert</td>
+ <td>Synchronize newly written data</td>
+ </tr>
+ <tr>
+ <td>delete</td>
+ <td>Synchronize deleted data</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema</td>
+ <td>database</td>
+ <td>Synchronize database creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>table</td>
+ <td>Synchronize table creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>TTL</td>
+ <td>Synchronize the data retention time</td>
+ </tr>
+ <tr>
+ <td>auth</td>
+ <td>-</td>
+ <td>Synchronize user permissions and access control</td>
+ </tr>
+</tbody>
+</table>
### 1.2 Functional Limitations and Notes
@@ -468,17 +508,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source parameter
-| **Parameter** | **Description**
| **Value Range** |
**Required** | **Default Value** |
-| :----------------------- |
:----------------------------------------------------------- |
:----------------------------------------------------------- | :----------- |
:---------------------------------------------------------- |
-| source | iotdb-source
| String: iotdb-source | Yes
| - |
-| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
-| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. | Boolean: true / false [...]
-| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
-| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched. | String:
Database name or database regular expression pattern string, which can match
uncreated or non - existent dat [...]
-| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched. | String: Data table
name or data table regular expression pattern string, which can be uncreated or
non - existent tables. | No [...]
-| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| forwarding-pipe-requests | Specifies whether to forward data that was
synchronized via the pipe to external clusters. Typically used for setting up
**active-active clusters**. In active-active cluster mode, this parameter
should be set to `false` to prevent **infinite circular synchronization**. |
Boolean: true / false | No |
true |
+| **Parameter** | **Description**
[...]
+| :-----------------------
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| source | iotdb-source
[...]
+| inclusion | Used to specify the range of data to be
synchronized in the data synchronization task, including data, schema and auth
[...]
+| inclusion.exclusion | Used to exclude specific operations from the
range specified by inclusion, reducing the amount of data synchronized
[...]
+| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
+| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. [...]
+| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
+| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched.
[...]
+| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched.
[...]
+| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| mode.double-living | Whether to enable full dual-active mode. When
enabled, the system will ignore the `-sql_dialect` connection method to capture
all tree-table model data and not forward data synced from another pipe (to
avoid circular synchronization).
[...]
> 💎 **Note:** The difference between the values of true and false for the
> data extraction mode `mode.streaming`
>
diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
index e8383920..4a7a6ff1 100644
--- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
+++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
@@ -34,7 +34,47 @@ A data synchronization task consists of three stages:
- Process Stage: This stage is used to process the data extracted from the
source IoTDB, defined in the `processor` section of the SQL statement.
- Sink Stage: This stage is used to send data to the target IoTDB, defined in
the `sink` section of the SQL statement.
-By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved.
+By declaratively configuring these three parts in an SQL statement, flexible
data synchronization capabilities can be achieved.Currently, data
synchronization supports the synchronization of the following information, and
you can select the synchronization scope when creating a synchronization task
(the default is data.insert, which means synchronizing newly written data):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>Synchronization Scope</th>
+ <th>Synchronization Content </th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>All scopes</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(Data)</td>
+ <td>insert</td>
+ <td>Synchronize newly written data</td>
+ </tr>
+ <tr>
+ <td>delete</td>
+ <td>Synchronize deleted data</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema</td>
+ <td>database</td>
+ <td>Synchronize database creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>table</td>
+ <td>Synchronize table creation, modification or deletion
operations</td>
+ </tr>
+ <tr>
+ <td>TTL</td>
+ <td>Synchronize the data retention time</td>
+ </tr>
+ <tr>
+ <td>auth</td>
+ <td>-</td>
+ <td>Synchronize user permissions and access control</td>
+ </tr>
+</tbody>
+</table>
### 1.2 Functional Limitations and Notes
@@ -503,17 +543,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source parameter
-| **Parameter** | **Description**
| **Value Range** |
**Required** | **Default Value** |
-| :----------------------- |
:----------------------------------------------------------- |
:----------------------------------------------------------- | :----------- |
:---------------------------------------------------------- |
-| source | iotdb-source
| String: iotdb-source | Yes
| - |
-| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
-| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. | Boolean: true / false [...]
-| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
-| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched. | String:
Database name or database regular expression pattern string, which can match
uncreated or non - existent dat [...]
-| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched. | String: Data table
name or data table regular expression pattern string, which can be uncreated or
non - existent tables. | No [...]
-| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe. | Long:
[Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format
timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE,
Long.MAX_VALUE](Unix bare timestamp) |
-| forwarding-pipe-requests | Specifies whether to forward data that was
synchronized via the pipe to external clusters. Typically used for setting up
**active-active clusters**. In active-active cluster mode, this parameter
should be set to `false` to prevent **infinite circular synchronization**. |
Boolean: true / false | No |
true |
+| **Parameter** | **Description**
[...]
+| :-----------------------
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| source | iotdb-source
[...]
+| inclusion | Used to specify the range of data to be
synchronized in the data synchronization task, including data,schema and auth
[...]
+| inclusion.exclusion | Used to exclude specific operations from the
range specified by inclusion, reducing the amount of data synchronized
[...]
+| mode.streaming | This parameter specifies the source of
time-series data capture. It applies to scenarios where `mode.streaming` is set
to `false`, determining the capture source for `data.insert` in `inclusion`.
Two capture strategies are available: - **true**: Dynamically selects the
capture type. The system adapts to downstream processing speed, choosing
between capturing each write request or only capturing TsFile file sealing
requests. When downstream processing is fast [...]
+| mode.strict | Determines whether to strictly filter data when
using the `time`, `path`, `database-name`, or `table-name` parameters: -
**true**: Strict filtering. The system will strictly filter captured data
according to the specified conditions, ensuring that only matching data is
selected. - **false**: Non-strict filtering. Some extra data may be included
during the selection process to optimize performance and reduce CPU and I/O
consumption. [...]
+| mode.snapshot | This parameter determines the data capture mode,
affecting the `data` in `inclusion`. Two modes are available: - **true**:
Static data capture. A one-time data snapshot is taken when the pipe starts.
Once the snapshot data is fully consumed, the pipe automatically terminates
(executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture.
In addition to capturing snapshot data when the pipe starts, it continuously
captures subsequent data chang [...]
+| database-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the database name to
filter. It can be a specific database name or a Java-style regular expression
to match multiple databases. By default, all databases are matched.
[...]
+| table-name | When the user connects with `sql_dialect` set to
`table`, this parameter can be specified. Determines the scope of data
capture, affecting the `data` in `inclusion`. Specifies the table name to
filter. It can be a specific table name or a Java-style regular expression to
match multiple tables. By default, all tables are matched.
[...]
+| start-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **greater than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| end-time | Determines the scope of data capture, affecting
the `data` in `inclusion`. Data with an event time **less than or equal to**
this parameter will be selected for stream processing in the pipe.
[...]
+| mode.double-living | Whether to enable full dual-active mode. When
enabled, the system will ignore the `-sql_dialect` connection method to capture
all tree-table model data and not forward data synced from another pipe (to
avoid circular synchronization).
[...]
> 💎 **Note:** The difference between the values of true and false for the
> data extraction mode `mode.streaming`
>
diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
index abd3f7f6..deb843ee 100644
--- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
+++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
@@ -34,7 +34,47 @@
- 处理(Process)阶段:该部分用于处理从源 IoTDB 抽取出的数据,在 SQL 语句中的 processor 部分定义
- 发送(Sink)阶段:该部分用于向目标 IoTDB 发送数据,在 SQL 语句中的 sink 部分定义
-通过 SQL 语句声明式地配置 3 个部分的具体内容,可实现灵活的数据同步能力。
+通过 SQL 语句声明式地配置 3
个部分的具体内容,可实现灵活的数据同步能力。目前数据同步支持以下信息的同步,您可以在创建同步任务时对同步范围进行选择(默认选择
data.insert,即同步新写入的数据):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>同步范围</th>
+ <th>同步内容</th>
+ <th>说明</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>所有范围</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(数据)</td>
+ <td>insert(增量)</td>
+ <td>同步新写入的数据</td>
+ </tr>
+ <tr>
+ <td>delete(删除)</td>
+ <td>同步被删除的数据</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema(元数据)</td>
+ <td>database(数据库)</td>
+ <td>同步数据库的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>table(表)</td>
+ <td>同步表的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>TTL(数据到期时间)</td>
+ <td>同步数据的存活时间</td>
+ </tr>
+ <tr>
+ <td>auth(权限)</td>
+ <td>-</td>
+ <td>同步用户权限和访问控制</td>
+ </tr>
+ </tbody>
+</table>
### 1.2 功能限制及说明
@@ -460,17 +500,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source 参数
-| **参数** | **描述**
| **value 取值范围** | **是否必填** |
**默认取值** |
-| ------------------------ |
------------------------------------------------------------ |
------------------------------------------------------------ | ------------ |
------------------------------- |
-| source | iotdb-source
| String: iotdb-source | 必填
| - |
-| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false | 否 |
true |
-| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。 | Boolean: true / false |
否 | true |
-| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 |
Boolean: true / false | 否 |
false |
-| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。 | String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库 | 否
| ".*" |
-| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。 | String:数据表名或数据表正则模式串,可以是未创建的、不存在的表 | 否
| ".*" |
-| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]
(unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE(unix
裸时间戳) |
-| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE](unix
裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE(unix 裸时间戳) |
-| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用,双活集群模式下该参数为
false,以此避免无限的环形同步。 | Boolean: true / false
| 否 | true |
+| **参数** | **描述**
| **value 取值范围**
| **是否必填** | **默认取值**
|
+| ------------------------
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
------------ | ------------------------------- |
+| source | iotdb-source
| String: iotdb-source
| 必填 | -
|
+| inclusion | 用于指定数据同步任务中需要同步范围,分为数据,元数据和权限
| String:all, data(insert,delete),
schema(database,table,ttl), auth | 选填 | data.insert |
+| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作,减少同步的数据量
| String:all, data(insert,delete),
schema(database,table,ttl), auth | 选填 | 空字符串 |
+| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false
| 否 | true |
+| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。
| Boolean: true / false
| 否 | true |
+| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。
| Boolean: true / false
| 否 | false
|
+| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。
|
String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库
| 否 | ".*" |
+| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。
|
String:数据表名或数据表正则模式串,可以是未创建的、不存在的表
| 否 | ".*" |
+| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE] (unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MIN_VALUE(unix 裸时间戳) |
+| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE](unix 裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MAX_VALUE(unix 裸时间戳) |
+| mode.double-living |
是否开启全量双活模式,开启后将忽略`-sql_dialect`连接方式,树表模型数据均会被捕获,且不会转发由另一pipe同步而来的数据。
| Boolean: true / false |
否 | false |
> 💎 **说明:数据抽取模式 mode.streaming 取值 true 和 false 的差异**
> - **true(推荐)**:该取值下,任务将对数据进行实时处理、发送,其特点是高时效、低吞吐
diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
index 7a9f52df..58e05842 100644
--- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
+++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md
@@ -34,7 +34,47 @@
- 处理(Process)阶段:该部分用于处理从源 IoTDB 抽取出的数据,在 SQL 语句中的 processor 部分定义
- 发送(Sink)阶段:该部分用于向目标 IoTDB 发送数据,在 SQL 语句中的 sink 部分定义
-通过 SQL 语句声明式地配置 3 个部分的具体内容,可实现灵活的数据同步能力。
+通过 SQL 语句声明式地配置 3
个部分的具体内容,可实现灵活的数据同步能力。目前数据同步支持以下信息的同步,您可以在创建同步任务时对同步范围进行选择(默认选择
data.insert,即同步新写入的数据):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>同步范围</th>
+ <th>同步内容</th>
+ <th>说明</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>所有范围</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(数据)</td>
+ <td>insert(增量)</td>
+ <td>同步新写入的数据</td>
+ </tr>
+ <tr>
+ <td>delete(删除)</td>
+ <td>同步被删除的数据</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema(元数据)</td>
+ <td>database(数据库)</td>
+ <td>同步数据库的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>table(表)</td>
+ <td>同步表的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>TTL(数据到期时间)</td>
+ <td>同步数据的存活时间</td>
+ </tr>
+ <tr>
+ <td>auth(权限)</td>
+ <td>-</td>
+ <td>同步用户权限和访问控制</td>
+ </tr>
+ </tbody>
+</table>
### 1.2 功能限制及说明
@@ -510,17 +550,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source 参数
-| **参数** | **描述**
| **value 取值范围** | **是否必填** |
**默认取值** |
-| ------------------------ |
------------------------------------------------------------ |
------------------------------------------------------------ | ------------ |
------------------------------- |
-| source | iotdb-source
| String: iotdb-source | 必填
| - |
-| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false | 否 |
true |
-| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。 | Boolean: true / false |
否 | true |
-| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 |
Boolean: true / false | 否 |
false |
-| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。 | String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库 | 否
| ".*" |
-| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。 | String:数据表名或数据表正则模式串,可以是未创建的、不存在的表 | 否
| ".*" |
-| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]
(unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE(unix
裸时间戳) |
-| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE](unix
裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE(unix 裸时间戳) |
-| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用,双活集群模式下该参数为
false,以此避免无限的环形同步。 | Boolean: true / false
| 否 | true |
+| **参数** | **描述**
| **value 取值范围**
| **是否必填** | **默认取值**
|
+| ------------------------
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
------------ | ------------------------------- |
+| source | iotdb-source
| String: iotdb-source
| 必填 | -
|
+| inclusion | 用于指定数据同步任务中需要同步范围,分为数据,元数据和权限
| String:all, data(insert,delete),
schema(database,table,ttl),auth | 选填 | data.insert |
+| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作,减少同步的数据量
| String:all, data(insert,delete),
schema(database,table,ttl), auth | 选填 | 空字符串 |
+| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false
| 否 | true |
+| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。
| Boolean: true / false
| 否 | true |
+| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。
| Boolean: true / false
| 否 | false
|
+| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。
|
String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库
| 否 | ".*" |
+| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。
|
String:数据表名或数据表正则模式串,可以是未创建的、不存在的表
| 否 | ".*" |
+| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE] (unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MIN_VALUE(unix 裸时间戳) |
+| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE](unix 裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MAX_VALUE(unix 裸时间戳) |
+| mode.double-living |
是否开启全量双活模式,开启后将忽略`-sql_dialect`连接方式,树表模型数据均会被捕获,且不会转发由另一pipe同步而来的数据。
| Boolean: true / false |
否 | false |
> 💎 **说明:数据抽取模式 mode.streaming 取值 true 和 false 的差异**
> - **true(推荐)**:该取值下,任务将对数据进行实时处理、发送,其特点是高时效、低吞吐
diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
index abd3f7f6..deb843ee 100644
--- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
+++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md
@@ -34,7 +34,47 @@
- 处理(Process)阶段:该部分用于处理从源 IoTDB 抽取出的数据,在 SQL 语句中的 processor 部分定义
- 发送(Sink)阶段:该部分用于向目标 IoTDB 发送数据,在 SQL 语句中的 sink 部分定义
-通过 SQL 语句声明式地配置 3 个部分的具体内容,可实现灵活的数据同步能力。
+通过 SQL 语句声明式地配置 3
个部分的具体内容,可实现灵活的数据同步能力。目前数据同步支持以下信息的同步,您可以在创建同步任务时对同步范围进行选择(默认选择
data.insert,即同步新写入的数据):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>同步范围</th>
+ <th>同步内容</th>
+ <th>说明</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>所有范围</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(数据)</td>
+ <td>insert(增量)</td>
+ <td>同步新写入的数据</td>
+ </tr>
+ <tr>
+ <td>delete(删除)</td>
+ <td>同步被删除的数据</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema(元数据)</td>
+ <td>database(数据库)</td>
+ <td>同步数据库的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>table(表)</td>
+ <td>同步表的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>TTL(数据到期时间)</td>
+ <td>同步数据的存活时间</td>
+ </tr>
+ <tr>
+ <td>auth(权限)</td>
+ <td>-</td>
+ <td>同步用户权限和访问控制</td>
+ </tr>
+ </tbody>
+</table>
### 1.2 功能限制及说明
@@ -460,17 +500,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source 参数
-| **参数** | **描述**
| **value 取值范围** | **是否必填** |
**默认取值** |
-| ------------------------ |
------------------------------------------------------------ |
------------------------------------------------------------ | ------------ |
------------------------------- |
-| source | iotdb-source
| String: iotdb-source | 必填
| - |
-| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false | 否 |
true |
-| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。 | Boolean: true / false |
否 | true |
-| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 |
Boolean: true / false | 否 |
false |
-| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。 | String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库 | 否
| ".*" |
-| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。 | String:数据表名或数据表正则模式串,可以是未创建的、不存在的表 | 否
| ".*" |
-| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]
(unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE(unix
裸时间戳) |
-| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE](unix
裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE(unix 裸时间戳) |
-| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用,双活集群模式下该参数为
false,以此避免无限的环形同步。 | Boolean: true / false
| 否 | true |
+| **参数** | **描述**
| **value 取值范围**
| **是否必填** | **默认取值**
|
+| ------------------------
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
------------ | ------------------------------- |
+| source | iotdb-source
| String: iotdb-source
| 必填 | -
|
+| inclusion | 用于指定数据同步任务中需要同步范围,分为数据,元数据和权限
| String:all, data(insert,delete),
schema(database,table,ttl), auth | 选填 | data.insert |
+| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作,减少同步的数据量
| String:all, data(insert,delete),
schema(database,table,ttl), auth | 选填 | 空字符串 |
+| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false
| 否 | true |
+| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。
| Boolean: true / false
| 否 | true |
+| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。
| Boolean: true / false
| 否 | false
|
+| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。
|
String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库
| 否 | ".*" |
+| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。
|
String:数据表名或数据表正则模式串,可以是未创建的、不存在的表
| 否 | ".*" |
+| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE] (unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MIN_VALUE(unix 裸时间戳) |
+| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE](unix 裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MAX_VALUE(unix 裸时间戳) |
+| mode.double-living |
是否开启全量双活模式,开启后将忽略`-sql_dialect`连接方式,树表模型数据均会被捕获,且不会转发由另一pipe同步而来的数据。
| Boolean: true / false |
否 | false |
> 💎 **说明:数据抽取模式 mode.streaming 取值 true 和 false 的差异**
> - **true(推荐)**:该取值下,任务将对数据进行实时处理、发送,其特点是高时效、低吞吐
diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
index 7a9f52df..58e05842 100644
--- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
+++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md
@@ -34,7 +34,47 @@
- 处理(Process)阶段:该部分用于处理从源 IoTDB 抽取出的数据,在 SQL 语句中的 processor 部分定义
- 发送(Sink)阶段:该部分用于向目标 IoTDB 发送数据,在 SQL 语句中的 sink 部分定义
-通过 SQL 语句声明式地配置 3 个部分的具体内容,可实现灵活的数据同步能力。
+通过 SQL 语句声明式地配置 3
个部分的具体内容,可实现灵活的数据同步能力。目前数据同步支持以下信息的同步,您可以在创建同步任务时对同步范围进行选择(默认选择
data.insert,即同步新写入的数据):
+
+<table style="text-align: left;">
+ <tbody>
+ <tr> <th>同步范围</th>
+ <th>同步内容</th>
+ <th>说明</th>
+ </tr>
+ <tr>
+ <td colspan="2">all</td>
+ <td>所有范围</td>
+ </tr>
+ <tr>
+ <td rowspan="2">data(数据)</td>
+ <td>insert(增量)</td>
+ <td>同步新写入的数据</td>
+ </tr>
+ <tr>
+ <td>delete(删除)</td>
+ <td>同步被删除的数据</td>
+ </tr>
+ <tr>
+ <td rowspan="3">schema(元数据)</td>
+ <td>database(数据库)</td>
+ <td>同步数据库的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>table(表)</td>
+ <td>同步表的创建、修改或删除操作</td>
+ </tr>
+ <tr>
+ <td>TTL(数据到期时间)</td>
+ <td>同步数据的存活时间</td>
+ </tr>
+ <tr>
+ <td>auth(权限)</td>
+ <td>-</td>
+ <td>同步用户权限和访问控制</td>
+ </tr>
+ </tbody>
+</table>
### 1.2 功能限制及说明
@@ -510,17 +550,19 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1
### source 参数
-| **参数** | **描述**
| **value 取值范围** | **是否必填** |
**默认取值** |
-| ------------------------ |
------------------------------------------------------------ |
------------------------------------------------------------ | ------------ |
------------------------------- |
-| source | iotdb-source
| String: iotdb-source | 必填
| - |
-| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false | 否 |
true |
-| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。 | Boolean: true / false |
否 | true |
-| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。 |
Boolean: true / false | 否 |
false |
-| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。 | String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库 | 否
| ".*" |
-| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。 | String:数据表名或数据表正则模式串,可以是未创建的、不存在的表 | 否
| ".*" |
-| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE]
(unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MIN_VALUE(unix
裸时间戳) |
-| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。 | Long: [Long.MIN_VALUE, Long.MAX_VALUE](unix
裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 | Long.MAX_VALUE(unix 裸时间戳) |
-| forwarding-pipe-requests | 是否转发由 pipe 数据同步而来的集群外的数据。一般供搭建双活集群时使用,双活集群模式下该参数为
false,以此避免无限的环形同步。 | Boolean: true / false
| 否 | true |
+| **参数** | **描述**
| **value 取值范围**
| **是否必填** | **默认取值**
|
+| ------------------------
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
------------ | ------------------------------- |
+| source | iotdb-source
| String: iotdb-source
| 必填 | -
|
+| inclusion | 用于指定数据同步任务中需要同步范围,分为数据,元数据和权限
| String:all, data(insert,delete),
schema(database,table,ttl),auth | 选填 | data.insert |
+| inclusion.exclusion | 用于从 inclusion 指定的同步范围内排除特定的操作,减少同步的数据量
| String:all, data(insert,delete),
schema(database,table,ttl), auth | 选填 | 空字符串 |
+| mode.streaming | 此参数指定时序数据写入的捕获来源。适用于 `mode.streaming`为 `false`
模式下的场景,决定`inclusion`中`data.insert`数据的捕获来源。提供两种捕获策略:true:
动态选择捕获的类型。系统将根据下游处理速度,自适应地选择是捕获每个写入请求还是仅捕获 TsFile
文件的封口请求。当下游处理速度快时,优先捕获写入请求以减少延迟;当处理速度慢时,仅捕获文件封口请求以避免处理堆积。这种模式适用于大多数场景,能够实现处理延迟和吞吐量的最优平衡。false:固定按批捕获方式。仅捕获
TsFile 文件的封口请求,适用于资源受限的应用场景,以降低系统负载。注意,pipe 启动时捕获的快照数据只会以文件的方式供下游处理。 |
Boolean: true / false
| 否 | true |
+| mode.strict | 在使用 time / path / database-name / table-name
参数过滤数据时,是否需要严格按照条件筛选:`true`:
严格筛选。系统将完全按照给定条件过滤筛选被捕获的数据,确保只有符合条件的数据被选中。`false`:非严格筛选。系统在筛选被捕获的数据时可能会包含一些额外的数据,适用于性能敏感的场景,可降低
CPU 和 IO 消耗。
| Boolean: true / false
| 否 | true |
+| mode.snapshot |
此参数决定时序数据的捕获方式,影响`inclusion`中的`data`数据。提供两种模式:`true`:静态数据捕获。启动 pipe
时,会进行一次性的数据快照捕获。当快照数据被完全消费后,**pipe 将自动终止(DROP PIPE SQL
会自动执行)**。`false`:动态数据捕获。除了在 pipe 启动时捕获快照数据外,还会持续捕获后续的数据变更。pipe 将持续运行以处理动态数据流。
| Boolean: true / false
| 否 | false
|
+| database-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的数据库的名称。它可以是具体的数据库名,也可以是 Java
风格正则表达式来匹配多个数据库。默认情况下,匹配所有的库。
|
String:数据库名或数据库正则模式串,可以匹配未创建的、不存在的库
| 否 | ".*" |
+| table-name | 当用户连接指定的 sql_dialect 为 table
时可以指定。此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。表示要过滤的表的名称。它可以是具体的表名,也可以是 Java
风格正则表达式来匹配多个表。默认情况下,匹配所有的表。
|
String:数据表名或数据表正则模式串,可以是未创建的、不存在的表
| 否 | ".*" |
+| start-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 大于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE] (unix 裸时间戳)或 String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MIN_VALUE(unix 裸时间戳) |
+| end-time | 此参数决定时序数据的捕获范围,影响`inclusion`中的`data`数据。当数据的 event
time 小于等于该参数时,数据会被筛选出来进入流处理 pipe。
| Long: [Long.MIN_VALUE,
Long.MAX_VALUE](unix 裸时间戳)或String:IoTDB 支持的 ISO 格式时间戳 | 否 |
Long.MAX_VALUE(unix 裸时间戳) |
+| mode.double-living |
是否开启全量双活模式,开启后将忽略`-sql_dialect`连接方式,树表模型数据均会被捕获,且不会转发由另一pipe同步而来的数据。
| Boolean: true / false |
否 | false |
> 💎 **说明:数据抽取模式 mode.streaming 取值 true 和 false 的差异**
> - **true(推荐)**:该取值下,任务将对数据进行实时处理、发送,其特点是高时效、低吞吐