This is an automated email from the ASF dual-hosted git repository.
JNSimba pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 9347f8d46ae [Doc] Align streaming CDC documentation across locales
(#4008)
9347f8d46ae is described below
commit 9347f8d46ae87e02810e67fe9cbe07ab8fe1da92
Author: wudi <[email protected]>
AuthorDate: Thu Jul 23 18:04:18 2026 +0800
[Doc] Align streaming CDC documentation across locales (#4008)
## Summary
- align the current English Streaming Job CDC documentation with its
Chinese counterparts
- remove `skip_snapshot_backfill` from Auto Table Creation Sync, the
overview's exactly-once description, and the multi-table CDC statement
reference
- retain `skip_snapshot_backfill` for SQL Mapping Sync and CDC Stream
TVF, where it remains supported
- align PostgreSQL publication behavior and CDC job property constraints
across locales
## Validation
- `git diff --check`
- `yarn docs:lint:changed`
- `yarn docusaurus build --locale en`
- verified matching heading structures and parameter keys across the ten
current English/Chinese page pairs
## Version scope
The aligned parameter additions are documented as available from Doris
4.1.0, so versioned 4.x pages are intentionally unchanged.
---
.../continuous-load-mysql-database.md | 86 +++++++-------
.../streaming-job/continuous-load-mysql-table.md | 83 +++++++++----
.../streaming-job/continuous-load-overview.md | 20 ++--
.../continuous-load-postgresql-database.md | 79 ++++++++-----
.../continuous-load-postgresql-table.md | 90 +++++++++-----
.../prerequisites/amazon-aurora-postgresql.md | 35 ++++--
.../prerequisites/amazon-rds-postgresql.md | 34 ++++--
.../table-valued-functions/cdc-stream.md | 129 +++++++++++++--------
docs/sql-manual/sql-statements/job/ALTER-JOB.md | 86 +++++++++-----
.../sql-statements/job/CREATE-STREAMING-JOB.md | 90 ++++++++++----
.../continuous-load-mysql-database.md | 1 -
.../streaming-job/continuous-load-overview.md | 2 +-
.../continuous-load-postgresql-database.md | 1 -
.../sql-statements/job/CREATE-STREAMING-JOB.md | 3 +-
14 files changed, 487 insertions(+), 252 deletions(-)
diff --git
a/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
b/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
index 0b52d267d4f..804195ea5b4 100644
---
a/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
+++
b/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
@@ -3,7 +3,7 @@
"title": "MySQL CDC with Auto Table Creation",
"language": "en",
"sidebar_label": "Auto Table Creation Sync",
- "description": "Use a Streaming Job to continuously sync full and
incremental data from a MySQL database to Doris, with tables created
automatically on first sync.",
+ "description": "Learn how Doris Streaming Jobs sync MySQL tables with
automatic table creation, filtering, offsets, snapshots, SSL, and data quality
controls.",
"keywords": [
"MySQL Auto Table Creation Sync",
"MySQL whole-database sync",
@@ -20,15 +20,15 @@
<!-- Knowledge type: Procedure + Configuration parameters -->
<!-- Applicable scenario: Mirror sync of a whole MySQL database to Doris -->
-Auto Table Creation Sync is implemented through the native `FROM MYSQL (...)
TO DATABASE (...)` DDL. **The target is a Doris database container, and Doris
auto-creates the downstream tables.** You can use `include_tables` to control
whether to sync one table, several tables, or all tables. On the first sync,
Doris automatically creates downstream primary key tables and keeps the primary
keys consistent with the upstream. This is suitable for mirror replication
scenarios where no SQL proc [...]
+Auto Table Creation Sync is implemented through the native `FROM MYSQL (...)
TO DATABASE (...)` DDL. The target is a Doris database, and Doris automatically
creates the corresponding downstream tables based on the upstream table
schemas. You can use `include_tables` to control whether to sync one table,
several tables, or all tables. On the first sync, Doris automatically creates
downstream primary key tables and keeps the primary keys consistent with the
upstream. This is suitable for m [...]
-By integrating [Flink CDC](https://github.com/apache/flink-cdc) capabilities,
Doris reads change logs from MySQL and continuously writes the full and
incremental data of a group of tables into Doris through Stream Load. If you
need to perform column mapping, filtering, or data transformation during sync,
refer to [MySQL CDC with SQL Mapping](./continuous-load-mysql-table.md).
+By integrating [Flink CDC](https://github.com/apache/flink-cdc) capabilities,
Doris reads change logs from MySQL and continuously writes the full and
incremental data of a group of tables into Doris through Stream Load. Auto
Table Creation Sync supports assigning a target table name to an individual
source table or excluding non-key columns. If you need SQL expressions, row
filtering, or data transformation, refer to [MySQL CDC with SQL
Mapping](./continuous-load-mysql-table.md).
### Applicable Scenarios
- You need to mirror a group of tables (or an entire database) from MySQL to
Doris.
- You want the downstream table schema and primary keys to be created
automatically based on the upstream, without manually creating tables.
-- No column mapping, filtering, or data transformation is required during sync.
+- You only need simple target table renaming or column pruning, without SQL
expressions, row filtering, or data transformation.
- You need to support both initial full initialization and continuous
reception of incremental changes.
### Capabilities and Limitations
@@ -36,10 +36,10 @@ By integrating [Flink
CDC](https://github.com/apache/flink-cdc) capabilities, Do
| Item | Description
|
| --------------------- |
-------------------------------------------------------------------- |
| Consistency semantics | Currently only at-least-once semantics are
guaranteed |
-| Table type | Only primary key tables (Unique Key) are supported
for sync |
+| Table type | Only upstream tables with primary keys can be
synced; automatically created downstream tables use the Unique Key model |
| Permission | Load permission is required; Create permission is
also required when the downstream table does not exist |
| Auto table creation | Creation is skipped if the target table already
exists; you can customize the table schema as needed |
-| Data processing | Column mapping, filtering, and transformation are
not supported; use SQL Mapping Sync if needed |
+| Data processing | Supports target table renaming and exclusion of
non-key columns; does not support SQL expressions, row filtering, or data
transformation |
## Prerequisites
@@ -133,20 +133,37 @@ For more general operations (pausing, resuming, deleting,
viewing tasks, and so
<!-- Knowledge type: Configuration parameters -->
-The MySQL source side (`FROM MYSQL`) supports the following parameters:
-
-| Parameter | Default | Description
|
-| -------------------- | ------- |
--------------------------------------------------------------------------------------------------------------------
|
-| jdbc_url | - | MySQL JDBC connection string
|
-| driver_url | - | Path to the JDBC driver jar. Supports file
name, local absolute path, and HTTP address. See [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md) for details |
-| driver_class | - | JDBC driver class name
|
-| user | - | Database user name
|
-| password | - | Database password
|
-| database | - | Database name
|
-| include_tables | - | Names of tables to sync, separated by
commas. If left empty, all tables are synced by default |
-| offset | latest | `latest`: incremental sync only; `initial`:
full + incremental sync |
-| snapshot_split_size | 8096 | Size of a split (in rows). During full
sync, a table is divided into multiple splits for sync |
-| snapshot_parallelism | 1 | Parallelism during the full-sync phase,
that is, the maximum number of splits a single Task can schedule at a time |
+The MySQL source (`FROM MYSQL`) supports the following parameters. Connection
information, the database name, and driver information are required.
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| `jdbc_url` | Yes | - | MySQL JDBC connection string. |
+| `driver_url` | Yes | - | Path to the JDBC driver jar. Supports a file name,
local absolute path, or HTTP URL. For details, see [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md). |
+| `driver_class` | Yes | - | JDBC driver class name, for example,
`com.mysql.cj.jdbc.Driver`. |
+| `user` | Yes | - | Database user name. |
+| `password` | Yes | - | Database password. |
+| `database` | Yes | - | MySQL database name. |
+| `include_tables` | No | - | Names of the tables to sync, separated by
commas. If not set, all tables in the database are synced. |
+| `exclude_tables` | No | - | Names of the tables not to sync, separated by
commas. This parameter takes effect only when `include_tables` is not set. |
+| `table.<table_name>.target_table` | No | Source table name | Supported since
version 4.1.0. Sets the Doris target table name for a source table.
`<table_name>` is the source table name. |
+| `table.<table_name>.exclude_columns` | No | - | Supported since version
4.1.0. Specifies source columns not to sync, separated by commas. The columns
must exist and cannot include primary key columns. |
+| `offset` | No | `latest` | Startup offset. `initial`: full and incremental
sync; `earliest`: start from the earliest available Binlog offset; `latest`:
sync only changes after the job starts. You can also specify an exact JSON
offset, such as `{"file":"binlog.000001","pos":"154"}` or
`{"gtids":"<gtid_set>"}`. Since version 4.1.0, `snapshot` is also supported for
full sync only. |
+| `snapshot_split_size` | No | `8096` | Split size in rows. During full sync,
a table is divided into multiple splits. Must be a positive integer. |
+| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that
is, the maximum number of splits scheduled by a Task at one time. Must be a
positive integer. |
+| `server_id` | No | Automatically generated | Supported since version 4.1.0.
Server ID of the MySQL CDC reader. Supports a single value, such as `5400`, or
a closed range, such as `5400-5408`. The range width must be greater than or
equal to `snapshot_parallelism`. |
+| `ssl_mode` | No | `disable` | Supported since version 4.1.0. SSL mode. Valid
values are `disable`, `require`, and `verify-ca`. |
+| `ssl_rootcert` | Conditionally required | - | Supported since version 4.1.0.
CA certificate file in the format `FILE:<file_name>`. Required when `ssl_mode`
is `verify-ca`. Upload the file first using [CREATE
FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |
+
+If both `include_tables` and `exclude_tables` are set, `include_tables` takes
precedence. The following `FROM MYSQL` parameter fragment syncs `orders` and
`customers`, writes `orders` to `ods_orders`, excludes the non-key column
`internal_note`, and enables CA verification:
+
+```sql
+"include_tables" = "orders,customers",
+"table.orders.target_table" = "ods_orders",
+"table.orders.exclude_columns" = "internal_note",
+"server_id" = "5400-5403",
+"ssl_mode" = "verify-ca",
+"ssl_rootcert" = "FILE:ca.pem"
+```
## Reference Manual
@@ -179,17 +196,19 @@ Description of each module:
| source_properties | Parameters related to the MySQL source |
| target_properties | Parameters related to the Doris target database |
+`job_properties` supports `max_interval` and `compute_group`. For details, see
[Common Job Load Configuration
Parameters](./continuous-load-overview.md#job-common-load-configuration-parameters).
Auto Table Creation mode does not use `session.*`.
+
### Doris Target Database Configuration Parameters
<!-- Knowledge type: Configuration parameters -->
-`TO DATABASE` supports the following parameters:
+`TO DATABASE` supports the following optional parameters:
-| Parameter | Default | Description
|
-| ------------------------- | ------- |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| table.create.properties.* | - | Specify table properties when creating
a table, such as `replication_num`
|
-| load.strict_mode | - | Whether to enable strict mode.
Disabled by default
|
-| load.max_filter_ratio | - | The maximum filter ratio allowed
within the sampling window. Must be greater than or equal to 0 and less than or
equal to 1. The default value is 0, meaning zero tolerance. The sampling window
is `max_interval * 10`. If the ratio of error rows to total rows within the
sampling window exceeds `load.max_filter_ratio`, the routine job is paused, and
manual intervention is required to check the data quality issue. |
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `table.create.properties.*` | - | Adds table properties when Doris creates a
table, for example, `table.create.properties.replication_num`. |
+| `load.strict_mode` | `false` | Whether to enable strict mode for Stream Load
writes. Valid values are `true` and `false`. |
+| `load.max_filter_ratio` | `0` | Maximum allowed filter ratio in the sampling
window, in the range `[0, 1]`. `0` means that erroneous rows cannot be
filtered. The sampling window is `max_interval * 10` seconds. The job is paused
when the ratio of erroneous rows to total rows in the window exceeds this
value. |
## FAQ
@@ -210,24 +229,13 @@ Set the `offset` parameter to `latest`. The job will skip
the full-sync phase an
**Q4: How do I choose between Auto Table Creation Sync and SQL Mapping Sync?**
- For mirror replication, auto table creation, and keeping the table schema
consistent with the upstream: use Auto Table Creation Sync.
-- For column mapping, filtering, or data transformation: use [MySQL CDC with
SQL Mapping](./continuous-load-mysql-table.md).
+- If you only need to rename target tables or exclude non-key columns: use
Auto Table Creation Sync.
+- If you need SQL expressions, row filtering, or data transformation: use
[MySQL CDC with SQL Mapping](./continuous-load-mysql-table.md).
**Q5: What should I do if table creation fails in a single-BE deployment?**
Explicitly specify a replica count of 1 in `TO DATABASE`:
`"table.create.properties.replication_num" = "1"`.
-## Troubleshooting
-
-<!-- Knowledge type: Troubleshooting -->
-
-| Symptom | Possible cause
| Diagnosis and resolution
|
-| ---------------------------------------- |
----------------------------------------------------------- |
-------------------------------------------------------------------------------------------------------------------
|
-| Job creation fails with permission error | The current user lacks Load or
Create permission | Grant Load permission to the executing account;
Create permission is also required for auto table creation |
-| Slow full sync | `snapshot_parallelism` is too
small, or splits are too large | Increase `snapshot_parallelism` appropriately;
adjust `snapshot_split_size` based on the number of rows in the table |
-| Job is paused due to data quality issues | The error row ratio exceeds
`load.max_filter_ratio` | Check the `ErrorMsg` field to locate dirty
data; adjust `load.strict_mode` or `load.max_filter_ratio` |
-| Replica count error during auto table creation | Single-BE deployment but
the default replica count is 3 | Set `table.create.properties.replication_num =
1` |
-| Incremental data does not reach Doris | MySQL binlog is not enabled, or
the account has no binlog permission | Enable binlog on the MySQL side and
grant `REPLICATION SLAVE` and `REPLICATION CLIENT` permissions to the sync
account |
-
## Related Documents
- [Continuous Load Overview](./continuous-load-overview.md)
diff --git
a/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md
b/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md
index 797aead44ca..c1c05a3c360 100644
---
a/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md
+++
b/docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md
@@ -3,7 +3,7 @@
"title": "MySQL CDC with SQL Mapping",
"language": "en",
"sidebar_label": "SQL Mapping Sync",
- "description": "How to continuously sync a single MySQL table to Doris
with SQL Mapping (Job + CDC Stream TVF), supporting column mapping and data
transformation.",
+ "description": "Use Doris Streaming Jobs and CDC Stream TVF to sync a
MySQL table with SQL mapping, full and incremental offsets, SSL, and delete
handling.",
"keywords": [
"MySQL sync to Doris",
"SQL Mapping Sync",
@@ -20,7 +20,9 @@
<!-- Knowledge type: Operation steps / Configuration parameters -->
<!-- Applicable scenario: Real-time sync of a single MySQL table to Doris with
column mapping or data transformation -->
-SQL Mapping Sync is implemented through Job + [CDC Stream
TVF](../../../../sql-manual/sql-functions/table-valued-functions/cdc-stream.md),
targeting an **existing** Doris table (`INSERT INTO tbl SELECT * FROM
cdc_stream(...)`). With the expressive power of Doris SQL, you can perform
column mapping, filtering, and data transformation in the sync pipeline, and
exactly-once semantics is guaranteed. This is suitable for real-time sync
scenarios that require data processing.
+SQL Mapping Sync is implemented through Job + [CDC Stream
TVF](../../../../sql-manual/sql-functions/table-valued-functions/cdc-stream.md),
targeting an **existing** Doris table (`INSERT INTO tbl SELECT * FROM
cdc_stream(...)`). With the expressive power of Doris SQL, you can perform
column mapping, filtering, and data transformation in the sync pipeline. This
is suitable for real-time sync scenarios that require data processing.
+
+> SQL Mapping Sync is supported since version 4.1.0.
By integrating the read capability of [Flink
CDC](https://github.com/apache/flink-cdc), Doris reads the change log (Binlog)
from MySQL and completes **full + incremental** sync from source table to
target table. If you want Doris to automatically create downstream tables and
sync a group of tables on a per-database basis, see [MySQL CDC with Auto Table
Creation](./continuous-load-mysql-database.md).
@@ -35,9 +37,9 @@ By integrating the read capability of [Flink
CDC](https://github.com/apache/flin
| Item | Description |
| ------ | ---- |
| Doris version | 4.1.0 and above |
-| Table type | Currently only **primary key tables** are supported as the
target table |
+| Table type | The upstream source table must have a primary key; the Doris
target table must use the Unique Key model |
| User privileges | `Load` privilege is required |
-| MySQL configuration | Binlog must be enabled on the MySQL side. See the
[Configuration Guide](./continuous-load-overview.md) |
+| MySQL configuration | Binlog must be enabled on the MySQL side. See the
[Configuration
Guide](./continuous-load-overview.md#supported-data-sources-and-sync-modes) |
| Semantic guarantee | Supports exactly-once semantics |
## Quick Start
@@ -80,29 +82,60 @@ For more general operations (pause, resume, delete, view
tasks, and so on), see
### Data Source Parameters
-The MySQL data source parameters supported by CDC Stream TVF are as follows:
+CDC Stream TVF supports the following MySQL data source parameters.
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| `type` | Yes | - | Data source type. Set to `mysql`. |
+| `jdbc_url` | Yes | - | MySQL JDBC connection string. |
+| `driver_url` | Yes | - | Path to the JDBC driver jar. Supports a file name,
local absolute path, or HTTP URL. For details, see [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md). |
+| `driver_class` | Yes | - | JDBC driver class name, for example,
`com.mysql.cj.jdbc.Driver`. |
+| `user` | Yes | - | Database user name. |
+| `password` | Yes | - | Database password. |
+| `database` | Yes | - | MySQL database name. |
+| `table` | Yes | - | Name of the table to sync. Each SQL Mapping job supports
one source table. |
+| `offset` | Yes | - | Startup offset. `initial`: full and incremental sync;
`snapshot`: full sync only; `earliest`: start from the earliest available
Binlog offset; `latest`: sync only changes after the job starts. You can also
specify an exact JSON offset, such as `{"file":"binlog.000001","pos":"154"}` or
`{"gtids":"<gtid_set>"}`. |
+| `snapshot_split_size` | No | `8096` | Split size in rows. During full sync,
the table is divided into multiple splits. Must be a positive integer. |
+| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that
is, the maximum number of splits scheduled by a Task at one time. Must be a
positive integer. |
+| `skip_snapshot_backfill` | No | `false` | Whether to skip Binlog backfill
during the snapshot. When set to `true`, at-least-once semantics are used. |
+| `server_id` | No | Automatically generated | Server ID of the MySQL CDC
reader. Supports a single value, such as `5400`, or a closed range, such as
`5400-5408`. The range width must be greater than or equal to
`snapshot_parallelism`. |
+| `ssl_mode` | No | `disable` | SSL mode. Valid values are `disable`,
`require`, and `verify-ca`. |
+| `ssl_rootcert` | Conditionally required | - | CA certificate file in the
format `FILE:<file_name>`. Required when `ssl_mode` is `verify-ca`. Upload the
file first using [CREATE
FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |
+| `include_delete_sign` | No | `false` | Whether the TVF additionally outputs
the `__DORIS_DELETE_SIGN__` column. Set to `true` to sync upstream DELETE
operations as deletes in a Doris primary key table. |
+
+When `include_delete_sign` is enabled, the target table must be a
Merge-on-Write Unique Key table, and `__DORIS_DELETE_SIGN__` must be explicitly
mapped in both the INSERT target column list and the SELECT list:
-| Parameter | Default | Description
|
-| --------------------- | ------- |
-----------------------------------------------------------------------------------------------------------------------------------------------
|
-| type | - | Data source type. Set to `mysql`
|
-| jdbc_url | - | MySQL JDBC connection string
|
-| driver_url | - | Path to the JDBC driver jar. Supports file
name, local absolute path, and HTTP URL. See [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md) for details |
-| driver_class | - | JDBC driver class name
|
-| user | - | Database username
|
-| password | - | Database password
|
-| database | - | Database name
|
-| table | - | Name of the table to sync
|
-| offset | latest | `latest`: incremental sync only;
`initial`: full + incremental sync
|
-| snapshot_split_size | 8096 | Size of a split (in rows). During full
sync, the table is divided into multiple splits for syncing
|
-| snapshot_parallelism | 1 | Parallelism during the full sync stage,
that is, the maximum number of splits scheduled per Task
|
+```sql
+CREATE JOB mysql_cdc_with_delete
+ON STREAMING
+DO
+INSERT INTO db1.target_table (id, value, __DORIS_DELETE_SIGN__)
+SELECT id, value, __DORIS_DELETE_SIGN__
+FROM cdc_stream(
+ "type" = "mysql",
+ "jdbc_url" = "jdbc:mysql://127.0.0.1:3306",
+ "driver_url" = "mysql-connector-java-8.0.25.jar",
+ "driver_class" = "com.mysql.cj.jdbc.Driver",
+ "user" = "root",
+ "password" = "123456",
+ "database" = "source_db",
+ "table" = "source_table",
+ "offset" = "initial",
+ "include_delete_sign" = "true"
+);
+```
+
+### Job Configuration Parameters
-### Import Configuration Parameters
+Set the following parameters through `CREATE JOB ... PROPERTIES (...)`:
-| Parameter | Default | Description
|
-| --------- | ------- |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| session.* | None | All session variables can be configured under
`job_properties`. For import variables, see [Insert Into
Select](../../../../data-operate/import/import-way/insert-into-manual.md#import-configuration-parameters)
|
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `max_interval` | `10` | Idle scheduling interval in seconds when no new
upstream data is available. Must be an integer greater than or equal to 1. |
+| `compute_group` | Current session or user default compute group | Supported
only in compute-storage decoupled mode. Specifies the compute group in which
the job runs. The user must have the USAGE privilege on the compute group. |
+| `session.<variable_name>` | Default value of the corresponding session
variable | Sets a session variable for the INSERT task. For load variables, see
[Insert Into
Select](../../../../data-operate/import/import-way/insert-into-manual.md#import-configuration-parameters).
|
-For more general parameters (such as `max_interval`), see the [Continuous
Import Overview](./continuous-load-overview.md#common-parameters).
+For the Job Property `offset` used to reset a CDC position through `ALTER JOB`
and its complete restrictions, see [Common Job Load Configuration
Parameters](./continuous-load-overview.md#job-common-load-configuration-parameters).
The initial position for a new job must be set in `cdc_stream(...)`.
## FAQ
@@ -113,7 +146,7 @@ For more general parameters (such as `max_interval`), see
the [Continuous Import
**Q2: Are non-primary-key tables supported as the target table?**
-Currently, **only primary key tables** are supported as the target table.
+The upstream source table must have a primary key, and the Doris target table
currently must use the Unique Key model.
**Q3: How do I sync only incremental data without the historical full data?**
@@ -126,6 +159,8 @@ You can adjust the following two parameters to improve
throughput during the ful
- `snapshot_split_size`: Increase the number of rows per split.
- `snapshot_parallelism`: Increase the parallelism of splits scheduled per
Task.
+When adjusting `snapshot_parallelism`, if a `server_id` range is also
configured, make sure that the number of IDs in the range is not less than the
parallelism.
+
## Related Documents
- [Continuous Import Overview](./continuous-load-overview.md)
diff --git
a/docs/data-operate/import/import-way/streaming-job/continuous-load-overview.md
b/docs/data-operate/import/import-way/streaming-job/continuous-load-overview.md
index b366baaa4de..b83fb61c8b8 100644
---
a/docs/data-operate/import/import-way/streaming-job/continuous-load-overview.md
+++
b/docs/data-operate/import/import-way/streaming-job/continuous-load-overview.md
@@ -3,7 +3,7 @@
"title": "Continuous Load Overview",
"language": "en",
"sidebar_label": "Overview",
- "description": "Learn about Doris Streaming Job continuous load: supported
data sources, SQL Mapping vs Auto Table Creation selection, job state machine,
and common operations.",
+ "description": "Explore Doris Streaming Job data sources, sync modes,
consistency semantics, job states, common parameters, and operational
commands.",
"keywords": [
"Doris continuous load",
"Streaming Job",
@@ -62,10 +62,10 @@ SQL Mapping Sync and Auto Table Creation Sync are two
continuous load methods wi
| Target level | An existing Doris table
| A Doris database container |
| Sync scope | A single table
| One to multiple tables to the entire database (controlled by
`include_tables`) |
| Automatic table creation | Tables must be pre-created
| Primary key tables are created automatically on first sync |
-| SQL flexibility | Supports column mapping, filtering, and transformation
(SELECT clause) | Copies as-is, does not support ETL |
+| SQL flexibility | Supports column mapping, filtering, and transformation
(SELECT clause) | Supports target table renaming and exclusion of non-key
columns; does not support SQL expressions, row filtering, or data
transformation |
| Semantic guarantee | exactly-once
| at-least-once |
| Required privileges | Load
| Load + Create (when creating tables automatically) |
-| Typical scenarios | Real-time sync that requires column pruning, field
renaming, type conversion, or conditional filtering | Mirror replication of an
entire database or a group of tables, where downstream table schemas should
automatically follow the upstream |
+| Typical scenarios | Real-time sync that requires SQL expressions, type
conversion, or conditional filtering | Mirror replication of an entire database
or a group of tables, allowing simple target table renaming or column pruning |
### Selection Recommendations
@@ -197,9 +197,15 @@ DROP JOB WHERE jobName = <job_name>;
### Job Common Load Configuration Parameters
-| Parameter | Default | Description
|
-| ------------ | ------- |
------------------------------------------------------ |
-| max_interval | 10 | Idle scheduling interval in seconds when the
upstream has no new data. Only an integer (number of seconds) is accepted, e.g.
`10`; a unit suffix such as `10s` is not supported. Must be >= 1. |
+Configure the following parameters through `CREATE JOB ... PROPERTIES (...)`.
Unless otherwise specified, they can be used by both SQL Mapping Sync and Auto
Table Creation Sync for MySQL and PostgreSQL.
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `max_interval` | `10` | Idle scheduling interval in seconds when no new
upstream data is available. Only integers greater than or equal to 1 are
accepted; unit suffixes such as `10s` are not supported. |
+| `compute_group` | Current session or user default compute group | Supported
only in compute-storage decoupled mode. Specifies the compute group in which
the job runs. An explicitly configured value cannot be empty, and the user must
have the USAGE privilege on the compute group. Job creation fails if this
parameter is not set and neither the current session nor the user has an
available default compute group. |
+| `session.<variable_name>` | Default value of the corresponding session
variable | Supported only in TVF mode. Sets a session variable for the Job's
INSERT task, for example, `session.insert_max_filter_ratio`. The variable name
and value must be valid Doris session variables. |
+
+When used as a Job Property, `offset` does not set the initial position of a
MySQL or PostgreSQL job. When creating a job, set the initial position in the
source parameters of `FROM MYSQL`, `FROM POSTGRES`, or `cdc_stream(...)`. After
pausing a CDC job, you can reset its position through `ALTER JOB ... PROPERTIES
("offset" = '<json_offset>')`. MySQL uses
`{"file":"binlog.000001","pos":"154"}`, and PostgreSQL uses
`{"lsn":"12345678"}`. `ALTER JOB` accepts only exact JSON offsets.
## Limitations
@@ -227,7 +233,7 @@ DDL sync applies **only to the [Auto Table Creation Sync
method](#capability-com
**Solution 1:** Add the `allowPublicKeyRetrieval=true` parameter to the JDBC
URL:
-```
+```text
jdbc:mysql://127.0.0.1:3306?allowPublicKeyRetrieval=true
```
diff --git
a/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
b/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
index 6e597b9394d..b205d473056 100644
---
a/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
+++
b/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
@@ -3,7 +3,7 @@
"title": "PostgreSQL CDC with Auto Table Creation",
"language": "en",
"sidebar_label": "Auto Table Creation Sync",
- "description": "Use a Streaming Job to continuously sync full and
incremental data from an entire PostgreSQL database to Doris, with automatic
table creation on first sync.",
+ "description": "Learn how Doris Streaming Jobs sync PostgreSQL tables with
automatic table creation, filtering, replication resources, SSL, and data
quality controls.",
"keywords": [
"PostgreSQL sync",
"PostgreSQL CDC",
@@ -22,21 +22,21 @@
<!-- Knowledge type: Operating procedure -->
<!-- Applicable scenario: Continuously sync an entire PostgreSQL database to
Doris -->
-Auto Table Creation Sync is implemented through the native `FROM POSTGRES
(...) TO DATABASE (...)` DDL. **The target is a Doris database container, and
Doris auto-creates the downstream tables.** You can use `include_tables` to
control whether one table, multiple tables, or all tables are synced. On the
first sync, Doris automatically creates the downstream primary-key tables and
keeps their primary keys consistent with the upstream. This approach is
suitable for mirror-replication scena [...]
+Auto Table Creation Sync is implemented through the native `FROM POSTGRES
(...) TO DATABASE (...)` DDL. The target is a Doris database, and Doris
automatically creates the corresponding downstream tables based on the upstream
table schemas. You can use `include_tables` to control whether one table,
multiple tables, or all tables are synced. On the first sync, Doris
automatically creates the downstream primary-key tables and keeps their primary
keys consistent with the upstream. This appr [...]
-By integrating [Flink CDC](https://github.com/apache/flink-cdc) capabilities,
Doris reads change logs from PostgreSQL and continuously writes the full +
incremental data of a group of tables into Doris through Stream Load. If you
need to perform column mapping, filtering, or data transformation during
synchronization, see [PostgreSQL CDC with SQL
Mapping](./continuous-load-postgresql-table.md).
+By integrating [Flink CDC](https://github.com/apache/flink-cdc) capabilities,
Doris reads change logs from PostgreSQL and continuously writes the full +
incremental data of a group of tables into Doris through Stream Load. Auto
Table Creation Sync supports assigning a target table name to an individual
source table or excluding non-key columns. If you need SQL expressions, row
filtering, or data transformation, see [PostgreSQL CDC with SQL
Mapping](./continuous-load-postgresql-table.md).
### Use Cases
- Mirror-replicating an entire PostgreSQL database or multiple tables to
Doris
- Keeping the downstream table schema consistent with the upstream without
manual table creation
-- No need for column pruning, filtering, or transformation in the sync
pipeline
+- Only simple target table renaming or column pruning is required, without
SQL expressions, row filtering, or data transformation
- Need both full initialization and continuous capture of incremental changes
### Notes
1. Currently only at-least-once semantics is guaranteed.
-2. Currently only primary-key table sync is supported.
+2. Only upstream tables with primary keys can be synced; automatically
created downstream tables use the Unique Key model.
3. The Load privilege is required. If the downstream table does not exist,
the Create privilege is also required.
4. During automatic table creation, if the target table already exists it is
skipped, so users can customize tables for different scenarios.
@@ -83,31 +83,52 @@ For more general operations (pause, resume, delete, view
tasks, etc.), see [Cont
### Source Parameters (PostgreSQL Side)
-PostgreSQL source parameters configure the JDBC connection, sync scope, and
full-snapshot slicing behavior.
-
-| Parameter | Default | Description
|
-| -------------------- | ------- |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| jdbc_url | - | PostgreSQL JDBC connection string
|
-| driver_url | - | Path to the JDBC driver jar. Supports file
name, local absolute path, and HTTP URL. See [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md) for details |
-| driver_class | - | JDBC driver class name
|
-| user | - | Database user name
|
-| password | - | Database password
|
-| database | - | Database name
|
-| schema | - | Schema name
|
-| include_tables | - | Tables to sync, separated by commas. If
left empty, all tables are synced by default
|
-| offset | latest | latest: incremental-only sync; initial:
full + incremental sync
|
-| snapshot_split_size | 8096 | Size of a split (in rows). During full
sync, a table is divided into multiple splits for synchronization
|
-| snapshot_parallelism | 1 | Parallelism during the full-sync phase,
that is, the maximum number of splits scheduled in a single Task
|
+PostgreSQL source parameters configure the JDBC connection, sync scope, and
full-snapshot slicing behavior. Connection information, driver information,
`database`, and `schema` are required.
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| `jdbc_url` | Yes | - | PostgreSQL JDBC connection string. |
+| `driver_url` | Yes | - | Path to the JDBC driver jar. Supports a file name,
local absolute path, or HTTP URL. For details, see [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md). |
+| `driver_class` | Yes | - | JDBC driver class name, for example,
`org.postgresql.Driver`. |
+| `user` | Yes | - | Database user name. |
+| `password` | Yes | - | Database password. |
+| `database` | Yes | - | PostgreSQL database name. It must match the database
name in the JDBC URL and cannot exceed 63 bytes after UTF-8 encoding. |
+| `schema` | Yes | - | Schema name. |
+| `include_tables` | No | - | Names of the tables to sync, separated by
commas. If not set, all tables in the Schema are synced. |
+| `exclude_tables` | No | - | Names of the tables not to sync, separated by
commas. This parameter takes effect only when `include_tables` is not set. |
+| `table.<table_name>.target_table` | No | Source table name | Supported since
version 4.1.0. Sets the Doris target table name for a source table.
`<table_name>` is the source table name. |
+| `table.<table_name>.exclude_columns` | No | - | Supported since version
4.1.0. Specifies source columns not to sync, separated by commas. The columns
must exist and cannot include primary key columns. |
+| `offset` | No | `latest` | Startup offset. `initial`: full and incremental
sync; `latest`: sync only changes after the job starts. You can also specify an
exact JSON offset, such as `{"lsn":"12345678"}`. Since version 4.1.0,
`snapshot` is also supported for full sync only. PostgreSQL does not support
`earliest`. |
+| `snapshot_split_size` | No | `8096` | Split size in rows. During full sync,
a table is divided into multiple splits. Must be a positive integer. |
+| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that
is, the maximum number of splits scheduled by a Task at one time. Must be a
positive integer. |
+| `slot_name` | No | `doris_cdc_<job_id>` | Supported since version 4.1.0.
Logical replication slot name. The name can contain only lowercase letters,
digits, and underscores, cannot start with a digit, and is limited to 63
characters. A custom slot must be created in advance and is not deleted by
Doris. |
+| `publication_name` | No | `doris_pub_<job_id>` | Supported since version
4.1.0. Publication name. The naming rules are the same as for `slot_name`. A
custom publication must be created in advance, include all synced tables, and
is not deleted by Doris. |
+| `ssl_mode` | No | `disable` | Supported since version 4.1.0. SSL mode. Valid
values are `disable`, `require`, and `verify-ca`. |
+| `ssl_rootcert` | Conditionally required | - | Supported since version 4.1.0.
CA certificate file in the format `FILE:<file_name>`. Required when `ssl_mode`
is `verify-ca`. Upload the file first using [CREATE
FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |
+
+If both `include_tables` and `exclude_tables` are set, `include_tables` takes
precedence. The following `FROM POSTGRES` parameter fragment syncs `orders` and
`customers`, writes `orders` to `ods_orders`, excludes the non-key column
`internal_note`, and uses pre-created logical replication resources with CA
verification:
+
+```sql
+"include_tables" = "orders,customers",
+"table.orders.target_table" = "ods_orders",
+"table.orders.exclude_columns" = "internal_note",
+"slot_name" = "orders_slot",
+"publication_name" = "orders_publication",
+"ssl_mode" = "verify-ca",
+"ssl_rootcert" = "FILE:ca.pem"
+```
+
+Custom replication slots and publications are managed by the user and are
retained when the Streaming Job is deleted. If `slot_name` or
`publication_name` is omitted, Doris creates the corresponding resource for the
job and cleans up resources it created when the job is deleted.
### Doris Target Database Parameters
-Target-side parameters control the properties of automatically created tables
and the Stream Load write strategy.
+Target-side parameters control the properties of automatically created tables
and the Stream Load write strategy. All of the following parameters are
optional.
-| Parameter | Default | Description
|
-| ------------------------- | ------- |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| table.create.properties.* | - | Supports specifying table properties
when creating tables, such as replication_num
|
-| load.strict_mode | - | Whether to enable strict mode.
Disabled by default
|
-| load.max_filter_ratio | - | The maximum allowed filter ratio
within the sampling window. The value must be between 0 and 1 inclusive. The
default value is 0, meaning zero tolerance. The sampling window is max_interval
* 10. If, within the sampling window, the ratio of error rows to total rows
exceeds max_filter_ratio, the routine job is paused, and manual intervention is
required to check data quality issues. |
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `table.create.properties.*` | - | Adds table properties when Doris creates a
table, for example, `table.create.properties.replication_num`. |
+| `load.strict_mode` | `false` | Whether to enable strict mode for Stream Load
writes. Valid values are `true` and `false`. |
+| `load.max_filter_ratio` | `0` | Maximum allowed filter ratio in the sampling
window, in the range `[0, 1]`. `0` means that erroneous rows cannot be
filtered. The sampling window is `max_interval * 10` seconds. The job is paused
when the ratio of erroneous rows to total rows in the window exceeds this
value. |
## Reference
@@ -140,14 +161,16 @@ The modules are described below:
| source_properties | PostgreSQL source-related parameters |
| target_properties | Doris target database-related parameters |
+`job_properties` supports `max_interval` and `compute_group`. For details, see
[Common Job Load Configuration
Parameters](./continuous-load-overview.md#job-common-load-configuration-parameters).
Auto Table Creation mode does not use `session.*`.
+
## FAQ
<!-- Knowledge type: Frequently asked questions -->
**Q1: How to choose between Auto Table Creation Sync and SQL Mapping Sync?**
-- When you need mirror replication, automatic table creation, and no column
pruning or transformation, use Auto Table Creation Sync.
-- When you need to perform column mapping, filtering, or data transformation
in the sync pipeline, use [PostgreSQL CDC with SQL
Mapping](./continuous-load-postgresql-table.md).
+- When you need mirror replication, automatic table creation, or only target
table renaming and exclusion of non-key columns, use Auto Table Creation Sync.
+- When you need SQL expressions, row filtering, or data transformation in
the sync pipeline, use [PostgreSQL CDC with SQL
Mapping](./continuous-load-postgresql-table.md).
**Q2: Are non-primary-key tables supported?**
diff --git
a/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-table.md
b/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-table.md
index a9bc4d8042b..affae062237 100644
---
a/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-table.md
+++
b/docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-table.md
@@ -20,7 +20,9 @@
<!-- Knowledge type: Procedure -->
<!-- Applicable scenario: PostgreSQL real-time data sync / single-table CDC
ingestion with SQL mapping -->
-SQL Mapping Sync is implemented through Job + [CDC Stream
TVF](../../../../sql-manual/sql-functions/table-valued-functions/cdc-stream.md).
The target is an existing Doris table (`INSERT INTO tbl SELECT * FROM
cdc_stream(...)`). Doris SQL provides the expressiveness to support column
mapping, filtering, and data transformation, while guaranteeing exactly-once
semantics. This approach is suitable for real-time sync scenarios where data
needs to be processed before writing.
+SQL Mapping Sync is implemented through Job + [CDC Stream
TVF](../../../../sql-manual/sql-functions/table-valued-functions/cdc-stream.md).
The target is an existing Doris table (`INSERT INTO tbl SELECT * FROM
cdc_stream(...)`). Doris SQL provides the expressiveness to support column
mapping, filtering, and data transformation. This approach is suitable for
real-time sync scenarios where data needs to be processed before writing.
+
+> SQL Mapping Sync is supported since version 4.1.0.
By integrating the read capability of [Flink
CDC](https://github.com/apache/flink-cdc), Doris reads change logs (WAL) from
PostgreSQL to perform full + incremental sync from the source table to the
target table. If you want Doris to automatically create downstream tables and
sync a group of tables on a per-database basis, see [PostgreSQL CDC with Auto
Table Creation](./continuous-load-postgresql-database.md).
@@ -34,17 +36,17 @@ By integrating the read capability of [Flink
CDC](https://github.com/apache/flin
Before creating a job, confirm the following:
-1. Logical replication is enabled on the PostgreSQL side. See the
[Configuration Guide](./continuous-load-overview.md) for details.
+1. Logical replication is enabled on the PostgreSQL side. See the
[Configuration
Guide](./continuous-load-overview.md#supported-data-sources-and-sync-modes) for
details.
2. The current user has the `Load` privilege.
3. The source table is a primary key table (only primary key tables are
currently supported for sync).
-4. The target table has been created in Doris in advance, and its schema is
compatible with the mapping SQL.
+4. The target Unique Key table has been created in Doris in advance, and its
schema is compatible with the mapping SQL.
### Notes
- Exactly-once semantics is supported.
- Only primary key table sync is currently supported.
- The `Load` privilege is required.
-- Logical replication must be enabled on the PostgreSQL side. See the
[Configuration Guide](./continuous-load-overview.md).
+- Logical replication must be enabled on the PostgreSQL side. See the
[Configuration
Guide](./continuous-load-overview.md#supported-data-sources-and-sync-modes).
## Quick Start
@@ -91,27 +93,59 @@ For other common operations (pause, resume, delete, view
tasks, and so on), see
Data source parameters are configured through the `cdc_stream(...)` TVF. They
describe the PostgreSQL table to be synced and the read behavior.
-| Parameter | Default | Description
|
-| -------------------- | ------- |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| type | - | Data source type. Set to `postgres`.
|
-| jdbc_url | - | PostgreSQL JDBC connection string.
|
-| driver_url | - | Path to the JDBC driver jar. Supports a
file name, a local absolute path, or an HTTP URL. See the [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md) for details.
|
-| driver_class | - | JDBC driver class name.
|
-| user | - | Database username.
|
-| password | - | Database password.
|
-| database | - | Database name.
|
-| schema | - | Schema name.
|
-| table | - | Name of the table to sync.
|
-| offset | latest | `latest`: incremental sync only. `initial`:
full + incremental sync.
|
-| snapshot_split_size | 8096 | Split size in rows. During full sync, the
table is divided into multiple splits for syncing.
|
-| snapshot_parallelism | 1 | Parallelism for the full sync stage, that
is, the maximum number of splits scheduled per Task.
|
-
-### Load Configuration Parameters
-
-Load configuration parameters are set through the job's `job_properties`. They
control load behavior and session variables.
-
-| Parameter | Default | Description
|
-| --------- | ------- |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| session.* | None | All session variables can be configured in
`job_properties`. For load-related variables, see [Insert Into
Select](../../../../data-operate/import/import-way/insert-into-manual.md#import-configuration-parameters).
|
-
-For more common parameters (such as `max_interval`), see the [Common
Parameters](./continuous-load-overview.md#common-parameters) section in the
Continuous Load Overview.
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| `type` | Yes | - | Data source type. Set to `postgres`. |
+| `jdbc_url` | Yes | - | PostgreSQL JDBC connection string. |
+| `driver_url` | Yes | - | Path to the JDBC driver jar. Supports a file name,
local absolute path, or HTTP URL. For details, see [JDBC Catalog
Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md). |
+| `driver_class` | Yes | - | JDBC driver class name, for example,
`org.postgresql.Driver`. |
+| `user` | Yes | - | Database user name. |
+| `password` | Yes | - | Database password. |
+| `database` | No | Database name in the JDBC URL | PostgreSQL database name.
When explicitly set, it overrides the database name in the JDBC URL. It cannot
exceed 63 bytes after UTF-8 encoding. |
+| `schema` | Yes | - | Schema name. |
+| `table` | Yes | - | Name of the table to sync. Each SQL Mapping job supports
one source table. |
+| `offset` | Yes | - | Startup offset. `initial`: full and incremental sync;
`snapshot`: full sync only; `latest`: sync only changes after the job starts.
You can also specify an exact JSON offset, such as `{"lsn":"12345678"}`.
PostgreSQL does not support `earliest`. |
+| `snapshot_split_size` | No | `8096` | Split size in rows. During full sync,
the table is divided into multiple splits. Must be a positive integer. |
+| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that
is, the maximum number of splits scheduled by a Task at one time. Must be a
positive integer. |
+| `skip_snapshot_backfill` | No | `false` | Whether to skip WAL backfill
during the snapshot. When set to `true`, at-least-once semantics are used. |
+| `slot_name` | No | `doris_cdc_<job_id>` | Logical replication slot name. The
name can contain only lowercase letters, digits, and underscores, cannot start
with a digit, and is limited to 63 characters. A custom slot must be created in
advance and is not deleted by Doris. |
+| `publication_name` | No | `doris_pub_<job_id>` | Publication name. The
naming rules are the same as for `slot_name`. A custom publication must be
created in advance, include the current source table, and is not deleted by
Doris. |
+| `ssl_mode` | No | `disable` | SSL mode. Valid values are `disable`,
`require`, and `verify-ca`. |
+| `ssl_rootcert` | Conditionally required | - | CA certificate file in the
format `FILE:<file_name>`. Required when `ssl_mode` is `verify-ca`. Upload the
file first using [CREATE
FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |
+| `include_delete_sign` | No | `false` | Whether the TVF additionally outputs
the `__DORIS_DELETE_SIGN__` column. Set to `true` to sync upstream DELETE
operations as deletes in a Doris primary key table. |
+
+When `include_delete_sign` is enabled, the target table must be a
Merge-on-Write Unique Key table, and `__DORIS_DELETE_SIGN__` must be explicitly
mapped in both the INSERT target column list and the SELECT list:
+
+```sql
+CREATE JOB pg_cdc_with_delete
+ON STREAMING
+DO
+INSERT INTO db1.target_table (id, value, __DORIS_DELETE_SIGN__)
+SELECT id, value, __DORIS_DELETE_SIGN__
+FROM cdc_stream(
+ "type" = "postgres",
+ "jdbc_url" = "jdbc:postgresql://127.0.0.1:5432/postgres",
+ "driver_url" = "postgresql-42.5.1.jar",
+ "driver_class" = "org.postgresql.Driver",
+ "user" = "postgres",
+ "password" = "postgres",
+ "schema" = "public",
+ "table" = "source_table",
+ "offset" = "initial",
+ "include_delete_sign" = "true"
+);
+```
+
+When you customize `slot_name` or `publication_name`, prepare the
corresponding resource before creating the Job. Doris retains user-provided
resources when the Job is deleted.
+
+### Job Configuration Parameters
+
+Set the following parameters through `CREATE JOB ... PROPERTIES (...)`:
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `max_interval` | `10` | Idle scheduling interval in seconds when no new
upstream data is available. Must be an integer greater than or equal to 1. |
+| `compute_group` | Current session or user default compute group | Supported
only in compute-storage decoupled mode. Specifies the compute group in which
the job runs. The user must have the USAGE privilege on the compute group. |
+| `session.<variable_name>` | Default value of the corresponding session
variable | Sets a session variable for the INSERT task. For load variables, see
[Insert Into
Select](../../../../data-operate/import/import-way/insert-into-manual.md#import-configuration-parameters).
|
+
+For the Job Property `offset` used to reset a CDC position through `ALTER JOB`
and its complete restrictions, see [Common Job Load Configuration
Parameters](./continuous-load-overview.md#job-common-load-configuration-parameters).
The initial position for a new job must be set in `cdc_stream(...)`.
diff --git
a/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-aurora-postgresql.md
b/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-aurora-postgresql.md
index 25f49e4bb50..97634392e37 100644
---
a/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-aurora-postgresql.md
+++
b/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-aurora-postgresql.md
@@ -10,6 +10,7 @@
"Doris streaming ingestion",
"Publication",
"dbz_publication",
+ "publication_name",
"CDC prerequisites"
]
}
@@ -39,7 +40,7 @@ The full configuration workflow consists of 5 steps:
2. Configure the cluster parameter group
3. Apply the cluster parameter group and restart the instance
4. Create a Doris sync user and grant privileges
-5. Create a Publication
+5. Configure the Publication based on the Doris version
If Step 1 shows that logical replication is already enabled, you can skip
directly to **Step 4** to create the sync user.
@@ -137,23 +138,32 @@ Privilege description:
| `ALTER DEFAULT PRIVILEGES ... GRANT SELECT` | Automatically grants read
privileges on tables created later in this schema |
| `rds_replication` | The role required to perform logical replication in
Aurora PostgreSQL |
-## Step 5: Create the Publication
+## Step 5: Configure the Publication
<!-- Knowledge type: Procedure -->
-**Purpose:** Create the Publication that Doris will subscribe to.
+### Doris 4.0
-Execute the following SQL:
+A Publication named `dbz_publication` must be created in advance with `FOR ALL
TABLES`:
```sql
CREATE PUBLICATION dbz_publication FOR ALL TABLES;
```
-:::caution
-Currently, Doris only supports a Publication named `dbz_publication`, and it
must be `FOR ALL TABLES`. Custom Publication names or specifying only a subset
of tables are not yet supported.
-:::
+If the sync user has superuser privileges, such as the `rds_superuser` role,
Doris can automatically create `dbz_publication` when the task starts, and you
do not need to execute this statement manually. Doris 4.0 does not support
custom Publication names or Publications that contain only a subset of tables.
+
+### Doris 4.1.0 and later
+
+By default, you do not need to create a Publication manually. When
`publication_name` is not set, Doris uses the sync account to create a
Publication named `doris_pub_<job_id>` and cleans it up when the job is
deleted. The sync account therefore needs permission to create Publications.
+
+If `publication_name` is explicitly set when creating the Streaming Job,
create a Publication with the same name in advance. A custom Publication must
include all source tables synchronized by the job and must be maintained and
cleaned up by the user. For example:
+
+```sql
+CREATE PUBLICATION doris_pub_custom
+FOR TABLE public.orders, public.customers;
+```
-> **Note:** If the sync user has superuser privileges (such as the
`rds_superuser` role), Doris creates the Publication automatically, and you do
not need to perform this step manually.
+Set `"publication_name" = "doris_pub_custom"` when creating the job. The name
can contain only lowercase letters, digits, and underscores, cannot start with
a digit, and is limited to 63 characters.
## FAQ
@@ -169,11 +179,11 @@ No. Doris streaming ingestion requires Aurora PostgreSQL
14 or later.
**Q3: Can I use a Publication with a custom name?**
-No. Currently, Doris only supports a Publication named `dbz_publication` with
`FOR ALL TABLES`.
+Doris 4.0 does not support custom names and only supports `dbz_publication FOR
ALL TABLES`. Since version 4.1.0, you can specify a custom name through
`publication_name`, but you must create the Publication before creating the job
and ensure that it includes all synchronized tables.
**Q4: Must I use a superuser account for synchronization?**
-No. It is recommended to use a regular user `doris_sync` and grant privileges
as described in this document. Only when the account has superuser privileges
(such as `rds_superuser`) does Doris create the Publication automatically.
+No. In Doris 4.0, a regular sync user requires `dbz_publication` to be created
manually, while Doris can create it automatically when the user has superuser
privileges. Since version 4.1.0, when `publication_name` is omitted, Doris
automatically creates and manages the default Publication. A Publication must
be created in advance only when a custom Publication is used.
## Troubleshooting
@@ -185,5 +195,6 @@ No. It is recommended to use a regular user `doris_sync`
and grant privileges as
| `SHOW rds.logical_replication;` still returns `off` | The new parameter
group is not bound to the cluster, or the writer instance has not been
restarted | Verify that **DB cluster parameter group** has been switched to the
new parameter group, and restart the writer instance |
| User creation or privilege grant fails | The current login account has
insufficient privileges | Sign in with an account that has `rds_superuser` or
equivalent privileges, then perform the operation |
| `GRANT rds_replication` returns an error | The current Aurora version is too
low, or the cluster is not Aurora PostgreSQL | Upgrade to Aurora PostgreSQL 14
or later |
-| Permission error when creating the Publication | The `doris_sync` user lacks
the privilege to create Publications | Use a superuser account to manually
execute `CREATE PUBLICATION`, or temporarily grant the required privileges to
the sync account |
-| Doris reports `dbz_publication` not found when starting synchronization |
The Publication has not been created, and the sync account lacks superuser
privileges, so it cannot be created automatically | Manually execute the
`CREATE PUBLICATION dbz_publication FOR ALL TABLES;` from Step 5 |
+| Doris 4.0 reports that `dbz_publication` does not exist | `dbz_publication`
has not been created, and the sync account lacks superuser privileges | Execute
`CREATE PUBLICATION dbz_publication FOR ALL TABLES;`, or use an account with
the required privileges |
+| Doris 4.1.0 or later fails to create the default Publication | The sync
account lacks permission to create Publications | Grant the required privileges
to the sync account, or have an administrator create a custom Publication in
advance and configure `publication_name` |
+| Doris 4.1.0 or later reports that a custom Publication does not exist |
`publication_name` is configured, but a Publication with the same name has not
been created | Create the Publication as described in [Step
5](#step-5-configure-the-publication), or remove the `publication_name`
configuration |
diff --git
a/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-rds-postgresql.md
b/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-rds-postgresql.md
index 5b79cedbf1e..b5539bc814d 100644
---
a/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-rds-postgresql.md
+++
b/docs/data-operate/import/import-way/streaming-job/prerequisites/amazon-rds-postgresql.md
@@ -11,6 +11,7 @@
"rds_replication",
"Publication",
"dbz_publication",
+ "publication_name",
"CDC sync"
]
}
@@ -42,7 +43,7 @@ The overall configuration flow is as follows:
2. Create and configure a custom parameter group (if not enabled)
3. Apply the parameter group to the instance and restart it
4. Create the Doris sync user and grant privileges
-5. Create the Publication for Doris to subscribe to
+5. Configure the Publication based on the Doris version
---
@@ -154,23 +155,32 @@ Privilege overview:
---
-## Step 5: Create the Publication
+## Step 5: Configure the Publication
<!-- Knowledge type: Procedure / Configuration parameters -->
-**Purpose**: Create the Publication that Doris subscribes to.
+### Doris 4.0
-Run the following SQL:
+A Publication named `dbz_publication` must be created in advance with `FOR ALL
TABLES`:
```sql
CREATE PUBLICATION dbz_publication FOR ALL TABLES;
```
-:::caution
-Currently, Doris only supports a Publication named `dbz_publication`, and it
must be created with `FOR ALL TABLES`. Custom Publication names or specifying
only a subset of tables are not supported yet.
-:::
+If the sync user has superuser privileges, such as the `rds_superuser` role,
Doris can automatically create `dbz_publication` when the task starts, and you
do not need to execute this statement manually. Doris 4.0 does not support
custom Publication names or Publications that contain only a subset of tables.
+
+### Doris 4.1.0 and later
+
+By default, you do not need to create a Publication manually. When
`publication_name` is not set, Doris uses the sync account to create a
Publication named `doris_pub_<job_id>` and cleans it up when the job is
deleted. The sync account therefore needs permission to create Publications.
+
+If `publication_name` is explicitly set when creating the Streaming Job,
create a Publication with the same name in advance. A custom Publication must
include all source tables synchronized by the job and must be maintained and
cleaned up by the user. For example:
+
+```sql
+CREATE PUBLICATION doris_pub_custom
+FOR TABLE public.orders, public.customers;
+```
-> **Note**: If the sync user has superuser privileges (such as the
`rds_superuser` role), Doris will automatically create this Publication when
the task starts, and you do not need to perform this step manually.
+Set `"publication_name" = "doris_pub_custom"` when creating the job. The name
can contain only lowercase letters, digits, and underscores, cannot start with
a digit, and is limited to 63 characters.
---
@@ -184,7 +194,7 @@ No. `on` means logical replication is already in effect,
and you can skip direct
**Q2: Can I use a Publication with a different name?**
-No. Doris currently only supports a Publication named `dbz_publication`, and
it must be created with `FOR ALL TABLES`.
+Doris 4.0 does not support custom names and only supports `dbz_publication FOR
ALL TABLES`. Since version 4.1.0, you can specify a custom name through
`publication_name`, but you must create the Publication before creating the job
and ensure that it includes all synchronized tables.
**Q3: Is it required to use the `rds_replication` role?**
@@ -192,7 +202,7 @@ Yes. For security reasons, RDS PostgreSQL does not allow
using the standard Post
**Q4: Is it mandatory to manually create the Publication?**
-Not necessarily. If the sync user has superuser privileges such as
`rds_superuser`, Doris will automatically create `dbz_publication`, and you can
skip [Step 5](#step-5-create-the-publication).
+In Doris 4.0, a regular sync user must create `dbz_publication` manually,
while Doris can create it automatically when the user has superuser privileges.
Since version 4.1.0, when `publication_name` is omitted, Doris automatically
creates and manages the default Publication. Only a custom Publication must be
created in advance. For details, see [Step
5](#step-5-configure-the-publication).
---
@@ -205,5 +215,7 @@ Not necessarily. If the sync user has superuser privileges
such as `rds_superuse
| `SHOW rds.logical_replication;` still returns `off` | The parameter group is
not bound to the instance, or the instance has not been restarted | Check
whether the instance uses the new parameter group, and restart it |
| Permission error when creating the user | The current account does not have
sufficient privileges | Use an administrator account with the privilege to
create users |
| `GRANT rds_replication` returns an error | The current account does not have
permission to grant this role | Use an `rds_superuser` account or another
account with the grant privilege |
-| Doris task reports that the Publication does not exist | `dbz_publication`
has not been created, or privileges are insufficient | Manually run [Step
5](#step-5-create-the-publication), or grant superuser privileges to the sync
user |
+| Doris 4.0 reports that `dbz_publication` does not exist | `dbz_publication`
has not been created, and the sync account lacks superuser privileges | Execute
`CREATE PUBLICATION dbz_publication FOR ALL TABLES;`, or use an account with
the required privileges |
+| Doris 4.1.0 or later fails to create the default Publication | The sync
account lacks permission to create Publications | Grant the required privileges
to the sync account, or have an administrator create a custom Publication in
advance and configure `publication_name` |
+| Doris 4.1.0 or later reports that a custom Publication does not exist |
`publication_name` is configured, but a Publication with the same name has not
been created | Create the Publication as described in [Step
5](#step-5-configure-the-publication), or remove `publication_name` to use the
default Publication managed by Doris |
| Parameter changes do not take effect | **Apply immediately** was not
selected, or the instance was not restarted | Re-execute [Step
3](#step-3-apply-the-parameter-group-and-restart) |
diff --git a/docs/sql-manual/sql-functions/table-valued-functions/cdc-stream.md
b/docs/sql-manual/sql-functions/table-valued-functions/cdc-stream.md
index 41741eb945c..15bed7d8ca9 100644
--- a/docs/sql-manual/sql-functions/table-valued-functions/cdc-stream.md
+++ b/docs/sql-manual/sql-functions/table-valued-functions/cdc-stream.md
@@ -2,7 +2,7 @@
{
"title": "CDC_STREAM | Table Valued Functions",
"language": "en",
- "description": "The CDC Stream table-valued-function (TVF) enables users
to read change data from relational databases (such as MySQL, PostgreSQL) via
CDC, supporting full and incremental synchronization.",
+ "description": "Use CDC Stream TVF to read MySQL and PostgreSQL changes
for incremental queries or full and incremental Streaming Job synchronization.",
"sidebar_label": "CDC_STREAM"
}
---
@@ -11,9 +11,25 @@
## Description
-The CDC Stream table-valued-function (TVF) enables users to read change data
from relational databases (such as MySQL, PostgreSQL) via CDC. By integrating
[Flink CDC](https://github.com/apache/flink-cdc) reading capabilities, it
supports full and incremental data synchronization.
+The CDC Stream table-valued function (TVF) reads change data from relational
databases such as MySQL and PostgreSQL through CDC. By integrating [Flink
CDC](https://github.com/apache/flink-cdc) reading capabilities, the function
continuously reads database change logs (Binlog/WAL) and provides the results
to a Streaming Job INSERT statement for writing into Doris.
-It is typically used with `CREATE JOB ON STREAMING` to achieve continuous
single-table data synchronization with SQL mapping. For detailed usage, see
[MySQL CDC with SQL
Mapping](../../../data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md)
and [PostgreSQL CDC with SQL
Mapping](../../../data-operate/import/import-way/streaming-job/continuous-load-postgresql-table.md).
+> This function is supported since version 4.1.0.
+
+It is typically used with `CREATE JOB ON STREAMING` to perform continuous full
and incremental synchronization of a single table with SQL mapping. For
detailed usage, see [MySQL CDC with SQL
Mapping](../../../data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md)
and [PostgreSQL CDC with SQL
Mapping](../../../data-operate/import/import-way/streaming-job/continuous-load-postgresql-table.md).
+
+:::note
+When used alone, CDC Stream TVF supports only incremental data synchronization
and does not support reading a full snapshot. When used with [CREATE STREAMING
JOB](../../sql-statements/job/CREATE-STREAMING-JOB.md), it supports full and
incremental synchronization.
+:::
+
+## Usage Notes
+
+1. When used alone, CDC Stream TVF supports only incremental synchronization;
when used with `CREATE JOB ON STREAMING`, it supports full and incremental
synchronization.
+2. It is typically used with `CREATE JOB ON STREAMING` and is not recommended
for use in regular queries.
+3. When `skip_snapshot_backfill` is set to `true`, at-least-once semantics are
used.
+4. When using MySQL, Binlog must be enabled (`binlog_format=ROW`). If a
`server_id` range is explicitly set, the number of IDs in the range cannot be
less than `snapshot_parallelism`.
+5. When using PostgreSQL, logical replication must be enabled
(`wal_level=logical`). User-specified replication slots and publications must
be created in advance and cleaned up by the user.
+6. When `ssl_mode=verify-ca`, `ssl_rootcert` must also be set.
+7. When `include_delete_sign=true`, the target table must be a Merge-on-Write
Unique Key table, and `__DORIS_DELETE_SIGN__` must be explicitly mapped in both
the INSERT target column list and the SELECT list.
## Syntax
@@ -25,40 +41,41 @@ cdc_stream(
"driver_class" = "<driver_class>",
"user" = "<user>",
"password" = "<password>",
- "database" = "<database>",
- "table" = "<table>"
+ "table" = "<table>",
+ "offset" = "<offset>"
+ [, "database" = "<database>"]
+ [, "schema" = "<schema>"]
[, "<optional_property_key>" = "<optional_property_value>" [, ...] ]
)
```
-## Required Parameters
-
-| Parameter | Description
|
-|----------------|--------------------------------------------------------------|
-| `type` | Data source type, currently supports `mysql` and `postgres`
|
-| `jdbc_url` | JDBC connection string, e.g. `jdbc:mysql://127.0.0.1:3306`
or `jdbc:postgresql://127.0.0.1:5432/postgres` |
-| `driver_url` | JDBC driver jar path
|
-| `driver_class` | JDBC driver class name. `com.mysql.cj.jdbc.Driver` for
MySQL, `org.postgresql.Driver` for PostgreSQL |
-| `user` | Database username
|
-| `password` | Database password
|
-| `database` | Database name
|
-| `table` | Table name to synchronize
|
-
-## Optional Parameters
-
-| Parameter | Default | Description
|
-|------------------------|---------|--------------------------------------------------------------|
-| `schema` | - | Schema name, required for PostgreSQL
|
-| `offset` | latest | `latest`: incremental only; `initial`:
full + incremental; also supports `snapshot`, `earliest` (MySQL only), or a
JSON binlog/LSN position |
-| `snapshot_split_size` | 8096 | Split size (in rows). During full sync,
the table is divided into multiple splits |
-| `snapshot_parallelism` | 1 | Parallelism during full sync phase, i.e.,
max splits per task |
-
-## Notes
-
-1. CDC Stream TVF is typically used with `CREATE JOB ON STREAMING` and is not
recommended for use in regular queries.
-2. When using the MySQL type, Binlog must be enabled on MySQL
(`binlog_format=ROW`).
-3. When using the PostgreSQL type, logical replication must be enabled
(`wal_level=logical`).
-4. Supports exactly-once semantics.
+## Parameters
+
+| Parameter | Applicable sources | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `type` | MySQL, PostgreSQL | Yes | - | Data source type. Set to `mysql` for
MySQL or `postgres` for PostgreSQL. |
+| `jdbc_url` | MySQL, PostgreSQL | Yes | - | JDBC connection string, for
example, `jdbc:mysql://127.0.0.1:3306` or
`jdbc:postgresql://127.0.0.1:5432/postgres`. |
+| `driver_url` | MySQL, PostgreSQL | Yes | - | Path to the JDBC driver jar.
Supports a file name, local absolute path, or HTTP URL. |
+| `driver_class` | MySQL, PostgreSQL | Yes | - | JDBC driver class name. Use
`com.mysql.cj.jdbc.Driver` for MySQL or `org.postgresql.Driver` for PostgreSQL.
|
+| `user` | MySQL, PostgreSQL | Yes | - | Database user name. |
+| `password` | MySQL, PostgreSQL | Yes | - | Database password. |
+| `database` | MySQL, PostgreSQL | Required for MySQL | Database name in the
PostgreSQL JDBC URL | Database name. For PostgreSQL, an explicitly configured
value overrides the database name in the JDBC URL and cannot exceed 63 bytes
after UTF-8 encoding. |
+| `schema` | PostgreSQL | Yes | - | PostgreSQL Schema name. |
+| `table` | MySQL, PostgreSQL | Yes | - | Name of the table to sync. Each CDC
Stream TVF supports one source table. |
+| `offset` | MySQL, PostgreSQL | Yes | - | Startup offset. `initial`: full and
incremental sync; `snapshot`: full sync only; `latest`: sync only changes after
startup. MySQL also supports `earliest`. Exact JSON offset examples: MySQL uses
`{"file":"binlog.000001","pos":"154"}` or `{"gtids":"<gtid_set>"}`, and
PostgreSQL uses `{"lsn":"12345678"}`. |
+| `snapshot_split_size` | MySQL, PostgreSQL | No | `8096` | Split size in
rows. During full sync, the table is divided into multiple splits. Must be a
positive integer. |
+| `snapshot_parallelism` | MySQL, PostgreSQL | No | `1` | Parallelism of the
full-sync phase, that is, the maximum number of splits scheduled by a Task at
one time. Must be a positive integer. |
+| `skip_snapshot_backfill` | MySQL, PostgreSQL | No | `false` | Whether to
skip incremental backfill during the snapshot. When set to `true`,
at-least-once semantics are used. |
+| `ssl_mode` | MySQL, PostgreSQL | No | `disable` | SSL mode. Valid values are
`disable`, `require`, and `verify-ca`. |
+| `ssl_rootcert` | MySQL, PostgreSQL | Conditionally required | - | CA
certificate file in the format `FILE:<file_name>`. Required when `ssl_mode` is
`verify-ca`. Upload the file first using [CREATE
FILE](../../sql-statements/security/CREATE-FILE.md). |
+| `server_id` | MySQL | No | Automatically generated | Server ID of the MySQL
CDC reader. Supports a single value, such as `5400`, or a closed range, such as
`5400-5408`. The range width must be greater than or equal to
`snapshot_parallelism`. |
+| `slot_name` | PostgreSQL | No | `doris_cdc_<job_id>` | Logical replication
slot name. The name can contain only lowercase letters, digits, and
underscores, cannot start with a digit, and is limited to 63 characters. A
custom slot must be created in advance and is not deleted by Doris. |
+| `publication_name` | PostgreSQL | No | `doris_pub_<job_id>` | Publication
name. The naming rules are the same as for `slot_name`. A custom publication
must be created in advance, include the source table, and is not deleted by
Doris. |
+| `include_delete_sign` | MySQL, PostgreSQL | No | `false` | Whether to output
an additional `__DORIS_DELETE_SIGN__` column. Set to `true` to sync upstream
DELETE operations as deletes in a Doris primary key table. |
+
+## Return Value
+
+Returns the columns of the source table. Column names and types are derived
from the source table Schema, and NULL source values are returned using the
corresponding column types. When `include_delete_sign=true`, the result
includes an additional `TINYINT` column named `__DORIS_DELETE_SIGN__`; regular
records have a value of `0`, and delete records have a value of `1`. If the
data source cannot be reached, the source table cannot be found, or a parameter
is invalid, function analysis or [...]
## Examples
@@ -79,7 +96,11 @@ cdc_stream(
"database" = "source_db",
"table" = "source_table",
"offset" = "initial"
- )
+ );
+ ```
+
+ ```text
+ The Job is created successfully; the source table's initial data and
subsequent Binlog changes are continuously written to db1.target_table.
```
- Continuously synchronize a single table from PostgreSQL
@@ -100,17 +121,22 @@ cdc_stream(
"schema" = "public",
"table" = "source_table",
"offset" = "initial"
- )
+ );
```
-- Incremental sync only (skip full snapshot)
+ ```text
+ The Job is created successfully; the source table's initial data and
subsequent WAL changes are continuously written to db1.target_table.
+ ```
+
+- Column mapping and data transformation
```sql
- CREATE JOB mysql_incremental_job
+ CREATE JOB mysql_cdc_transform_job
ON STREAMING
DO
- INSERT INTO db1.target_table
- SELECT * FROM cdc_stream(
+ INSERT INTO db1.target_table (id, name, age)
+ SELECT id, name, cast(age as INT) as age
+ FROM cdc_stream(
"type" = "mysql",
"jdbc_url" = "jdbc:mysql://127.0.0.1:3306",
"driver_url" = "mysql-connector-java-8.0.25.jar",
@@ -119,18 +145,22 @@ cdc_stream(
"password" = "123456",
"database" = "source_db",
"table" = "source_table",
- "offset" = "latest"
- )
+ "offset" = "initial"
+ );
```
-- Column mapping and data transformation
+ ```text
+ The Job is created successfully; after age is converted to INT, the selected
columns are continuously written to db1.target_table.
+ ```
+
+- Synchronize upstream DELETE operations
```sql
- CREATE JOB mysql_cdc_transform_job
+ CREATE JOB mysql_cdc_delete_job
ON STREAMING
DO
- INSERT INTO db1.target_table (id, name, age)
- SELECT id, name, cast(age as INT) as age
+ INSERT INTO db1.target_table (id, name, __DORIS_DELETE_SIGN__)
+ SELECT id, name, __DORIS_DELETE_SIGN__
FROM cdc_stream(
"type" = "mysql",
"jdbc_url" = "jdbc:mysql://127.0.0.1:3306",
@@ -140,6 +170,11 @@ cdc_stream(
"password" = "123456",
"database" = "source_db",
"table" = "source_table",
- "offset" = "initial"
- )
+ "offset" = "initial",
+ "include_delete_sign" = "true"
+ );
+ ```
+
+ ```text
+ The Job is created successfully; regular records carry delete sign 0, and
DELETE records carry delete sign 1.
```
diff --git a/docs/sql-manual/sql-statements/job/ALTER-JOB.md
b/docs/sql-manual/sql-statements/job/ALTER-JOB.md
index 37be679f57f..20b1e368618 100644
--- a/docs/sql-manual/sql-statements/job/ALTER-JOB.md
+++ b/docs/sql-manual/sql-statements/job/ALTER-JOB.md
@@ -2,35 +2,49 @@
{
"title": "ALTER JOB",
"language": "en",
- "description": "A user can modify a job. Only jobs in the PAUSE state can
be modified, and only Streaming type jobs can be modified."
+ "description": "Use ALTER JOB to update paused Streaming Jobs, change
runtime properties or SQL, and reset MySQL or PostgreSQL CDC progress to an
exact JSON offset."
}
---
## Description
-A user can modify a job. Only jobs in the PAUSE state can be modified, and
only Streaming type jobs can be modified.
+Modifies the runtime properties, execution SQL, or Auto Table Creation Sync
configuration of a Streaming Job. Only Streaming Jobs in the `PAUSED` state can
be modified, and each statement must provide at least one actual change.
## Syntax
-```SQL
-Alter Job <job_name>
-[job_properties]
-DO <Insert_Command>
+```sql
+ALTER JOB <job_name>
+[
+ PROPERTIES (<job_property>[, ...])
+]
+[
+ <Insert_Command>
+ | FROM <source_type> (<source_property>[, ...])
+ TO DATABASE <target_db> (<target_property>[, ...])
+]
```
## Required Parameters
**1. `<job_name>`**
-> Modify the job name of the job
+> The name of the Job to modify.
## Optional parameters
-**1. `<job_properties>`**
-> Modify the job's attributes.
+**1. `<job_property>`**
+> Modifies Job properties such as `max_interval`, `compute_group`, or
`session.*`, or resets a CDC position through a JSON `offset`.
-**1. `<Insert_Command>`**
-> Modify the SQL executed by the job.
+**2. `<Insert_Command>`**
+> Modifies the INSERT SQL executed by a TVF-mode job. Do not use the `DO`
keyword in `ALTER JOB`.
+**3. `<source_property>` and `<target_property>`**
+> Modify source-side or target-side properties of Auto Table Creation Sync.
The data source type and target database cannot be changed.
+
+:::note
+Auto Table Creation Sync does not allow changes to `jdbc_url`, `database`,
`schema`, `include_tables`, `exclude_tables`, the source-side `offset`,
`snapshot_split_size`, `snapshot_parallelism`, per-table mapping parameters,
`slot_name`, or `publication_name`. To reset a CDC position, use the Job
Property `offset`.
+
+CDC Stream TVF mode does not allow changes to the target table, TVF type,
`type`, `jdbc_url`, `database`, `schema`, `table`, `snapshot_split_size`,
`snapshot_parallelism`, `slot_name`, or `publication_name`. You can modify the
INSERT SQL to rotate credentials, drivers, or other modifiable parameters.
+:::
## Access Control Requirements
@@ -44,16 +58,16 @@ The user executing this SQL command must have at least the
following privileges:
- Modify the session variable of my_job
- ```SQL
- Alter Job my_job
+ ```sql
+ ALTER JOB my_job
PROPERTIES(
"session.insert_max_filter_ratio"="0.5"
- )
+ );
```
- Modify the SQL statement for my_job
- ```SQL
- Alter Job my_job
+ ```sql
+ ALTER JOB my_job
INSERT INTO db1.tbl1
SELECT * FROM S3
(
@@ -68,8 +82,8 @@ The user executing this SQL command must have at least the
following privileges:
- Simultaneously modify the Properties and SQL statements of my_job.
- ```SQL
- Alter Job my_job
+ ```sql
+ ALTER JOB my_job
PROPERTIES(
"session.insert_max_filter_ratio"="0.5"
)
@@ -81,14 +95,34 @@ The user executing this SQL command must have at least the
following privileges:
"s3.region" = "<s3_region>",
"s3.endpoint" = "<s3_endpoint>",
"format" = "<format>"
- )
+ );
```
-- Modify the synchronization progress of my_job
+- Modify the synchronization progress of the S3 job my_job
- ```sql
- Alter JOB my_job
- PROPERTIES(
- 'offset' = '{"fileName":"regression/load/data/example_0.csv"}'
- )
- ```
\ No newline at end of file
+```sql
+ALTER JOB my_job
+PROPERTIES(
+ 'offset' = '{"fileName":"regression/load/data/example_0.csv"}'
+);
+```
+
+- Reset a MySQL CDC job to an exact Binlog offset
+
+```sql
+ALTER JOB mysql_cdc_job
+PROPERTIES (
+ "offset" = '{"file":"binlog.000001","pos":"154"}'
+);
+```
+
+- Reset a PostgreSQL CDC job to an exact LSN
+
+```sql
+ALTER JOB pg_cdc_job
+PROPERTIES (
+ "offset" = '{"lsn":"12345678"}'
+);
+```
+
+A CDC position can be modified only while the job is in the `PAUSED` state.
The `offset` property accepts only an exact JSON offset in the formats shown
above; it does not accept `initial`, `snapshot`, `earliest`, or `latest`.
diff --git a/docs/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
b/docs/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
index dc5b097ccdf..dea6ec37666 100644
--- a/docs/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
+++ b/docs/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
@@ -2,7 +2,7 @@
{
"title": "CREATE STREAMING JOB",
"language": "en",
- "description": "Doris Streaming Job is a continuous import task based on
the Job approach. After the Job is submitted, Doris continuously runs the
import job, querying TVF or upstream data sources in real time and writing the
data into Doris tables."
+ "description": "Use CREATE STREAMING JOB to continuously ingest data from
TVFs or synchronize MySQL and PostgreSQL tables into Doris."
}
---
@@ -67,41 +67,57 @@ TO DATABASE <target_db> (
**4. `<source_property>`** (Multi-table CDC Mode)
-| Parameter | Default | Description
|
-| -------------- | ------- |
------------------------------------------------------------ |
-| jdbc_url | - | JDBC connection string (MySQL/PostgreSQL)
|
-| driver_url | - | JDBC driver jar path
|
-| driver_class | - | JDBC driver class name
|
-| user | - | Database username
|
-| password | - | Database password
|
-| database | - | Database name
|
-| schema | - | Schema name (PostgreSQL)
|
-| include_tables | - | Tables to synchronize, comma separated. If not
specified, all tables will be synchronized by default. |
-| offset | latest | `latest`: incremental only (default); `initial`:
full + incremental sync |
-| snapshot_split_size | 8096 | Split size (in rows). During full sync, the
table is divided into multiple splits. |
-| snapshot_parallelism | 1 | Parallelism during full sync phase, i.e., max
splits per task. |
+| Parameter | Applicable sources | Default | Description |
+| --- | --- | --- | --- |
+| `jdbc_url` | MySQL, PostgreSQL | - | Required. JDBC connection string. |
+| `driver_url` | MySQL, PostgreSQL | - | Required. Path to the JDBC driver
jar. |
+| `driver_class` | MySQL, PostgreSQL | - | Required. JDBC driver class name. |
+| `user` | MySQL, PostgreSQL | - | Required. Database user name. |
+| `password` | MySQL, PostgreSQL | - | Required. Database password. |
+| `database` | MySQL, PostgreSQL | - | Required. Upstream database name. For
PostgreSQL, it must match the database name in the JDBC URL. |
+| `schema` | PostgreSQL | - | Required. PostgreSQL Schema name. |
+| `include_tables` | MySQL, PostgreSQL | - | Names of tables to sync,
separated by commas. If not set, all tables in the database or Schema are
synced. |
+| `exclude_tables` | MySQL, PostgreSQL | - | Names of tables not to sync,
separated by commas. This parameter takes effect only when `include_tables` is
not set. |
+| `table.<table_name>.target_table` | MySQL, PostgreSQL | Source table name |
Sets the Doris target table name for a source table. |
+| `table.<table_name>.exclude_columns` | MySQL, PostgreSQL | - | Excludes
non-key columns from a source table, separated by commas. |
+| `offset` | MySQL, PostgreSQL | `latest` | Startup offset. Supports
`initial`, `snapshot`, `latest`, and exact JSON offsets. MySQL also supports
`earliest`. |
+| `snapshot_split_size` | MySQL, PostgreSQL | `8096` | Full-snapshot split
size in rows. Must be a positive integer. |
+| `snapshot_parallelism` | MySQL, PostgreSQL | `1` | Maximum number of splits
scheduled by a Task at one time during the full-sync phase. Must be a positive
integer. |
+| `ssl_mode` | MySQL, PostgreSQL | `disable` | SSL mode. Valid values are
`disable`, `require`, and `verify-ca`. |
+| `ssl_rootcert` | MySQL, PostgreSQL | - | CA certificate file in the format
`FILE:<file_name>`. Required when `ssl_mode=verify-ca`. |
+| `server_id` | MySQL | Automatically generated | Server ID of the MySQL CDC
reader. Supports a single value, such as `5400`, or a closed range, such as
`5400-5408`. The range width must be greater than or equal to
`snapshot_parallelism`. |
+| `slot_name` | PostgreSQL | `doris_cdc_<job_id>` | Logical replication slot
name. A custom slot must be created in advance and managed by the user. |
+| `publication_name` | PostgreSQL | `doris_pub_<job_id>` | Publication name. A
custom publication must be created in advance, include all synchronized tables,
and be managed by the user. |
+
+> Version note: `table.<table_name>.target_table`,
`table.<table_name>.exclude_columns`, `offset=snapshot`, `ssl_mode`,
`ssl_rootcert`, `server_id`, `slot_name`, and `publication_name` are supported
since version 4.1.0.
+
+If both `include_tables` and `exclude_tables` are set, `include_tables` takes
precedence. `slot_name` and `publication_name` can contain only lowercase
letters, digits, and underscores, cannot start with a digit, and are limited to
63 characters. For complete source-specific differences, JSON offset formats,
and SSL certificate configuration, see [MySQL CDC with Auto Table
Creation](../../../data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md)
and [PostgreSQL C [...]
**5. `<target_db>`** (Multi-table CDC Mode)
> Doris target database name to import into.
**6. `<target_property>`** (Multi-table CDC Mode)
-| Parameter | Default | Description
|
-| ------------------------------- | ------- |
------------------------------------------------------------ |
-| table.create.properties.* | - | Table properties when creating,
e.g. replication_num |
-| load.strict_mode | - | Whether to enable strict mode.
Disabled by default. |
-| load.max_filter_ratio | - | The maximum allowed filtering
ratio within a sampling window. Must be between 0 and 1 (inclusive). The
default value is 0, indicating zero tolerance. The sampling window equals
max_interval * 10. If, within this window, the ratio of erroneous rows to total
rows exceeds max_filter_ratio, the scheduled job will be paused and requires
manual intervention to address data quality issues. |
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `table.create.properties.*` | - | Adds table properties when Doris creates a
table, for example, `table.create.properties.replication_num`. |
+| `load.strict_mode` | `false` | Whether to enable strict mode for Stream Load
writes. Valid values are `true` and `false`. |
+| `load.max_filter_ratio` | `0` | Maximum allowed filter ratio in the sampling
window, in the range `[0, 1]`. The sampling window is `max_interval * 10`
seconds. The job is paused when the error-row ratio in the window exceeds this
value. |
## Optional Parameters
**1. `<job_property>`**
-| Parameter | Default | Description
|
-| ------------------ | ------- |
------------------------------------------------------------ |
-| session.* | - | Supports configuring all session variables in
job_properties (TVF Mode only) |
-| s3.max_batch_files | 256 | Triggers an import write when the accumulated
file count reaches this value (S3 TVF only) |
-| s3.max_batch_bytes | 10737418240 | Triggers an import write when the
accumulated data size (in bytes) reaches this value. Only a plain integer byte
count is accepted, e.g. `10737418240`; a unit suffix such as `10G` is not
supported. Valid range is 100MB–10GB, i.e. 104857600–10737418240. (S3 TVF only)
|
-| max_interval | 10 | Idle scheduling interval in seconds when
there are no new files or data upstream. Only an integer (number of seconds) is
accepted, e.g. `10`; a unit suffix such as `10s` is not supported. Must be >= 1
|
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `max_interval` | `10` | Idle scheduling interval in seconds when no new
upstream file or data is available. Must be an integer greater than or equal to
1. |
+| `compute_group` | Current session or user default compute group | Supported
only in compute-storage decoupled mode. Specifies the compute group in which
the job runs. The user must have the USAGE privilege on the compute group. |
+| `session.<variable_name>` | Default value of the corresponding session
variable | Supported only in TVF mode. Sets a valid Doris session variable for
the INSERT task. |
+| `s3.max_batch_files` | `256` | Triggers a load when the accumulated file
count reaches this value. Used only by S3 TVF. |
+| `s3.max_batch_bytes` | `10737418240` | Triggers a load when the accumulated
data size reaches this value. Used only by S3 TVF. The unit is bytes, and the
valid range is 104857600 to 10737418240. |
+| `offset` | - | Sets the source starting position when creating an S3
Streaming Job. The initial MySQL/PostgreSQL position must be set in source
parameters; only an exact JSON offset can reset CDC progress through `ALTER
JOB`. |
+
+For all CDC Stream TVF parameters, including `type`, `table`, and
`include_delete_sign`, see [CDC
Stream](../../sql-functions/table-valued-functions/cdc-stream.md).
## Privilege Control
@@ -115,6 +131,30 @@ The user executing this SQL command must have at least the
following privileges:
- TASK only retains the latest 100 records.
+### PROPERTIES Usage
+
+Use `PROPERTIES` to configure Job runtime parameters, such as changing the
scheduling interval or setting session variables:
+
+```sql
+CREATE JOB my_job
+PROPERTIES (
+ "max_interval" = "30",
+ "session.insert_max_filter_ratio" = "0.5"
+)
+ON STREAMING
+DO
+INSERT INTO db1.tbl1
+SELECT * FROM S3(
+ "uri" = "s3://bucket/s3/demo/*.csv",
+ "format" = "csv",
+ "column_separator" = ",",
+ "s3.endpoint" = "https://s3.ap-southeast-1.amazonaws.com";,
+ "s3.region" = "ap-southeast-1",
+ "s3.access_key" = "",
+ "s3.secret_key" = ""
+);
+```
+
## Examples
### TVF Mode
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
index 9beeece9333..d01b21274c4 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md
@@ -150,7 +150,6 @@ MySQL 源端（`FROM MYSQL`）支持以下参数。连接信息、数据库名
| `offset` | 否 | `latest` | 启动位点。`initial`：全量 + 增量同步；`earliest`：从当前可用的最早
Binlog 位点开始；`latest`：仅同步作业启动后的增量；也可设置 JSON 精确位点，例如
`{"file":"binlog.000001","pos":"154"}` 或 `{"gtids":"<gtid_set>"}`。自 4.1.0 版本起支持
`snapshot`，表示仅全量同步。 |
| `snapshot_split_size` | 否 | `8096` | split 的大小（行数）。全量同步时，表会被切分成多个
split。必须为正整数。 |
| `snapshot_parallelism` | 否 | `1` | 全量阶段的并行度，即单次 Task 最多调度的 split 数量。必须为正整数。 |
-| `skip_snapshot_backfill` | 否 | `true` | 自 4.1.0 版本起支持。是否跳过全量快照阶段的 Binlog
回填。自动建表同步为 at-least-once，默认跳过，快照期间的变更会在后续增量阶段重放。 |
| `server_id` | 否 | 自动生成 | 自 4.1.0 版本起支持。MySQL CDC reader 的 server ID。支持单值（如
`5400`）或闭区间（如 `5400-5408`）；区间宽度必须大于等于 `snapshot_parallelism`。 |
| `ssl_mode` | 否 | `disable` | 自 4.1.0 版本起支持。SSL 模式，可选值为
`disable`、`require`、`verify-ca`。 |
| `ssl_rootcert` | 条件必填 | - | 自 4.1.0 版本起支持。CA 证书文件，格式为
`FILE:<file_name>`。`ssl_mode` 为 `verify-ca` 时必填；文件需先通过 [CREATE
FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md) 上传。 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-overview.md
index f05b1d93221..78dc7aa9d4f 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-overview.md
@@ -63,7 +63,7 @@ SQL 映射同步和自动建表同步是两种**实现机制完全不同**的持
| 同步范围 | 单张表 |
一张到多张到整库（由 `include_tables` 控制） |
| 自动建表 | 需预建 |
首次同步自动创建主键表 |
| SQL 灵活表达 | 支持列映射、过滤、转换（SELECT 子句） |
支持目标表重命名和排除非主键列；不支持 SQL 表达式、行过滤或数据转换 |
-| 语义保证 | exactly-once（`skip_snapshot_backfill=false`） |
at-least-once |
+| 语义保证 | exactly-once
| at-least-once |
| 所需权限 | Load
| Load + Create（自动建表时） |
| 典型适用场景 | 需要 SQL 表达式、类型转换或条件过滤的实时同步 |
整库或一组表的镜像复制，允许简单的表重命名或列裁剪 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
index 4a3bb7d1c72..8115d15a3d1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md
@@ -101,7 +101,6 @@ PostgreSQL 源端参数用于配置 JDBC 连接、同步范围以及全量切片
| `offset` | 否 | `latest` | 启动位点。`initial`：全量 + 增量同步；`latest`：仅同步作业启动后的增量；也可设置
JSON 精确位点，例如 `{"lsn":"12345678"}`。自 4.1.0 版本起支持 `snapshot`，表示仅全量同步。PostgreSQL
不支持 `earliest`。 |
| `snapshot_split_size` | 否 | `8096` | split 的大小（行数）。全量同步时，表会被切分成多个
split。必须为正整数。 |
| `snapshot_parallelism` | 否 | `1` | 全量阶段的并行度，即单次 Task 最多调度的 split 数量。必须为正整数。 |
-| `skip_snapshot_backfill` | 否 | `true` | 自 4.1.0 版本起支持。是否跳过全量快照阶段的 WAL
回填。自动建表同步为 at-least-once，默认跳过，快照期间的变更会在后续增量阶段重放。 |
| `slot_name` | 否 | `doris_cdc_<job_id>` | 自 4.1.0
版本起支持。逻辑复制槽名称。名称只能包含小写字母、数字和下划线，不能以数字开头，最长 63 个字符。自定义复制槽必须预先创建，且不会由 Doris 删除。 |
| `publication_name` | 否 | `doris_pub_<job_id>` | 自 4.1.0 版本起支持。发布名称，命名规则与
`slot_name` 相同。自定义发布必须预先创建并包含所有同步表，且不会由 Doris 删除。 |
| `ssl_mode` | 否 | `disable` | 自 4.1.0 版本起支持。SSL 模式，可选值为
`disable`、`require`、`verify-ca`。 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
index 0a9472cb790..7b29c532a70 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md
@@ -83,14 +83,13 @@ TO DATABASE <target_db> (
| `offset` | MySQL、PostgreSQL | `latest` | 启动位点。支持
`initial`、`snapshot`、`latest` 和 JSON 精确位点；MySQL 还支持 `earliest`。 |
| `snapshot_split_size` | MySQL、PostgreSQL | `8096` | 全量切片的大小（行数），必须为正整数。 |
| `snapshot_parallelism` | MySQL、PostgreSQL | `1` | 全量阶段单次 Task 最多调度的 split
数量，必须为正整数。 |
-| `skip_snapshot_backfill` | MySQL、PostgreSQL | `true` |
是否跳过全量快照期间的增量回填。自动建表同步默认跳过，提供 at-least-once 语义。 |
| `ssl_mode` | MySQL、PostgreSQL | `disable` | SSL 模式，可选值为
`disable`、`require`、`verify-ca`。 |
| `ssl_rootcert` | MySQL、PostgreSQL | - | CA 证书文件，格式为
`FILE:<file_name>`；`ssl_mode=verify-ca` 时必填。 |
| `server_id` | MySQL | 自动生成 | MySQL CDC reader 的 server ID，支持单值（如
`5400`）或闭区间（如 `5400-5408`）；区间宽度必须大于等于 `snapshot_parallelism`。 |
| `slot_name` | PostgreSQL | `doris_cdc_<job_id>` |
逻辑复制槽名称。自定义复制槽必须预先创建，由用户管理。 |
| `publication_name` | PostgreSQL | `doris_pub_<job_id>` |
发布名称。自定义发布必须预先创建、覆盖所有同步表，并由用户管理。 |
->
版本说明：`table.<table_name>.target_table`、`table.<table_name>.exclude_columns`、`offset=snapshot`、`skip_snapshot_backfill`、`ssl_mode`、`ssl_rootcert`、`server_id`、`slot_name`
和 `publication_name` 自 4.1.0 版本起支持。
+>
版本说明：`table.<table_name>.target_table`、`table.<table_name>.exclude_columns`、`offset=snapshot`、`ssl_mode`、`ssl_rootcert`、`server_id`、`slot_name`
和 `publication_name` 自 4.1.0 版本起支持。
同时设置 `include_tables` 和 `exclude_tables` 时，以 `include_tables` 为准。`slot_name` 和
`publication_name` 只能包含小写字母、数字和下划线，不能以数字开头，最长 63 个字符。完整的数据源差异、JSON 位点格式和 SSL
证书配置见 [MySQL
自动建表同步](../../../data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md)和
[PostgreSQL
自动建表同步](../../../data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md)。
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
