This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13193-8964879f41468951f4cca86705fa1b642bae2b00 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 6d4e92ee67d5c3fa0b2a89f911adea79ef40a7c4 Author: Jast <[email protected]> AuthorDate: Tue Nov 17 15:48:59 2026 +0000 [Docs] Fix JDBC type mappings, connector feature claims and examples verified against source code (#13193) Co-authored-by: zhangshenghang <[email protected]> Co-authored-by: jast <[email protected]> --- docs/en/connectors/connector-faq.md | 8 ++-- docs/en/connectors/sink/Assert.md | 6 +-- docs/en/connectors/sink/DB2.md | 2 +- docs/en/connectors/sink/DuckDB.md | 7 ++-- docs/en/connectors/sink/Mysql.md | 5 ++- docs/en/connectors/sink/OceanBase.md | 45 +++++++++++++--------- docs/en/connectors/sink/Oracle.md | 3 +- docs/en/connectors/sink/Pulsar.md | 15 ++++---- docs/en/connectors/source/DB2.md | 6 +-- docs/en/connectors/source/DuckDB.md | 2 +- docs/en/connectors/source/Firebase.md | 2 +- docs/en/connectors/source/Http.md | 2 +- docs/en/connectors/source/IoTDB.md | 5 ++- docs/en/connectors/source/MongoDB.md | 7 +++- docs/en/connectors/source/OceanBase.md | 2 +- docs/en/connectors/source/PostgreSQL.md | 2 +- docs/en/connectors/source/Typesense.md | 5 ++- docs/en/engines/zeta/engine-jar-storage-mode.md | 2 +- docs/en/getting-started/docker/docker.md | 20 +++++----- docs/en/introduction/concepts/schema-feature.md | 4 +- docs/zh/connectors/connector-faq.md | 8 ++-- docs/zh/connectors/sink/Assert.md | 6 +-- docs/zh/connectors/sink/DB2.md | 2 +- docs/zh/connectors/sink/DuckDB.md | 7 ++-- docs/zh/connectors/sink/Greenplum.md | 34 ---------------- docs/zh/connectors/sink/Mysql.md | 5 ++- docs/zh/connectors/sink/OceanBase.md | 45 +++++++++++++--------- docs/zh/connectors/sink/Oracle.md | 3 +- docs/zh/connectors/sink/Pulsar.md | 15 ++++---- docs/zh/connectors/source/DB2.md | 6 +-- docs/zh/connectors/source/DuckDB.md | 2 +- docs/zh/connectors/source/Firebase.md | 2 +- docs/zh/connectors/source/Http.md | 2 +- docs/zh/connectors/source/IoTDB.md | 5 ++- docs/zh/connectors/source/MongoDB.md | 5 ++- docs/zh/connectors/source/OceanBase.md | 2 +- docs/zh/connectors/source/PostgreSQL.md | 2 +- docs/zh/connectors/source/Typesense.md | 5 ++- docs/zh/engines/zeta/engine-jar-storage-mode.md | 2 +- docs/zh/getting-started/docker/docker.md | 19 ++++----- .../locally/quick-start-seatunnel-engine.md | 2 +- docs/zh/introduction/concepts/schema-feature.md | 4 +- 42 files changed, 168 insertions(+), 165 deletions(-) diff --git a/docs/en/connectors/connector-faq.md b/docs/en/connectors/connector-faq.md index d37b000808..5d463df3c6 100644 --- a/docs/en/connectors/connector-faq.md +++ b/docs/en/connectors/connector-faq.md @@ -64,8 +64,6 @@ Change Data Capture connectors read real-time change events (INSERT / UPDATE / D 1. **Connector-specific issues** → go directly to the connector's page and scroll to its **FAQ** section. 2. **Type and schema questions** (e.g., JSON schema, Debezium schema, Oracle CLOB, MySQL TINYINT, ORC/Parquet mapping) → see [Type and Schema FAQ](./type-schema-faq.md). 3. **File and object storage questions** (e.g., MinIO/S3, small files, save modes, multi-table file paths) → see [File and Object Storage FAQ](./file-object-storage-faq.md). -4. **Cross-connector topics** (e.g., "does SeaTunnel support CDC?", "what is `schema_save_mode`?") → see the [General FAQ](../faq.md). -5. **Still stuck?** → search the [GitHub Issues](https://github.com/apache/seatunnel/issues) or reach out via the [mailing list](https://lists.apache.org/[email protected]). -2. **Cross-connector sink write topics** (e.g., "`generate_sink_sql` vs `query`", "`schema_save_mode`", "`data_save_mode`", or `enable_upsert`) → see [Sink Write Modes and Save Modes](./common-options/sink-write-modes.md). -3. **Other cross-connector topics** (e.g., "does SeaTunnel support CDC?") → see the [General FAQ](../faq.md). -4. **Still stuck?** → search the [GitHub Issues](https://github.com/apache/seatunnel/issues) or reach out via the [mailing list](https://lists.apache.org/[email protected]). +4. **Cross-connector sink write topics** (e.g., "`generate_sink_sql` vs `query`", "`schema_save_mode`", "`data_save_mode`", or `enable_upsert`) → see [Sink Write Modes and Save Modes](./common-options/sink-write-modes.md). +5. **Other cross-connector topics** (e.g., "does SeaTunnel support CDC?") → see the [General FAQ](../faq.md). +6. **Still stuck?** → search the [GitHub Issues](https://github.com/apache/seatunnel/issues) or reach out via the [mailing list](https://lists.apache.org/[email protected]). diff --git a/docs/en/connectors/sink/Assert.md b/docs/en/connectors/sink/Assert.md index 37cf2dc599..83b6bd4df4 100644 --- a/docs/en/connectors/sink/Assert.md +++ b/docs/en/connectors/sink/Assert.md @@ -29,15 +29,15 @@ Assert is a sink connector used to validate pipeline output. It checks row count |------------------------------------------------------------------------------------------------|-------------------------------------------------|----------|---------| | rules | ConfigMap | yes | - | | rules.field_rules | ConfigList | no | - | -| rules.field_rules.field_name | string\|ConfigMap | yes | - | -| rules.field_rules.field_type | string | no | - | +| rules.field_rules.field_name | string | yes | - | +| rules.field_rules.field_type | string\|ConfigMap | no | - | | rules.field_rules.field_value | ConfigList | no | - | | rules.field_rules.field_value.rule_type | string | no | - | | rules.field_rules.field_value.rule_value | numeric | no | - | | rules.field_rules.field_value.equals_to | boolean\|numeric\|string\|ConfigList\|ConfigMap | no | - | | rules.row_rules | ConfigList | no | - | | rules.row_rules.rule_type | string | no | - | -| rules.row_rules.rule_value | string | no | - | +| rules.row_rules.rule_value | numeric | no | - | | rules.catalog_table_rule | ConfigMap | no | - | | rules.catalog_table_rule.primary_key_rule | ConfigMap | no | - | | rules.catalog_table_rule.primary_key_rule.primary_key_name | string | no | - | diff --git a/docs/en/connectors/sink/DB2.md b/docs/en/connectors/sink/DB2.md index 5290fe9a92..726886aa28 100644 --- a/docs/en/connectors/sink/DB2.md +++ b/docs/en/connectors/sink/DB2.md @@ -46,7 +46,7 @@ Write data to DB2 through JDBC. Supports batch and streaming jobs, concurrent wr | DB2 Data Type | SeaTunnel Data Type | |------------------------------------------------------------------------------------------------------|---------------------| | BOOLEAN | BOOLEAN | -| SMALLINT | SHORT | +| SMALLINT | SMALLINT | | INT<br/>INTEGER<br/> | INT | | BIGINT | BIGINT | | DECIMAL<br/>DEC<br/>NUMERIC<br/>NUM | DECIMAL((Get the designated column's specified column size),<br/>(Gets the designated column's number of digits to right of the decimal point.))) | diff --git a/docs/en/connectors/sink/DuckDB.md b/docs/en/connectors/sink/DuckDB.md index b525871ab0..663eb3b16b 100644 --- a/docs/en/connectors/sink/DuckDB.md +++ b/docs/en/connectors/sink/DuckDB.md @@ -35,10 +35,10 @@ works against a local database file path (`jdbc:duckdb:/path/to/database.db`) or - [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md) - [x] [cdc](../../introduction/concepts/connector-v2-features.md) +- [x] [timer flush](../../introduction/concepts/connector-v2-features.md) > The generic JDBC sink implements exactly-once through XA transactions. The > DuckDB JDBC driver does not > provide an XA datasource; do not set `is_exactly_once = true` for DuckDB. -- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md) ## Supported DataSource Info @@ -101,7 +101,8 @@ DuckDB `TIME` values preserve microsecond precision when read or written through ### Tips -> If partition_column is not set, it will run in single concurrency, and if partition_column is set, it will be executed in parallel according to the concurrency of tasks. +> Sink concurrency follows the job or pipeline parallelism. `partition_column` is a JDBC +> source option for splitting reads; it does not control how the sink distributes writes. ## Task Example @@ -116,7 +117,7 @@ env { source { FakeSource { parallelism = 1 - row_num = 1000 + row.num = 1000 schema = { fields { id = "int" diff --git a/docs/en/connectors/sink/Mysql.md b/docs/en/connectors/sink/Mysql.md index 0633e88dfe..eac49386c6 100644 --- a/docs/en/connectors/sink/Mysql.md +++ b/docs/en/connectors/sink/Mysql.md @@ -65,7 +65,7 @@ semantics (using XA transaction guarantee). | TIME | TIME | | DATETIME | TIMESTAMP | | TIMESTAMP | TIMESTAMP_TZ | -| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINAR<br/>BIT(n)<br/>GEOMETRY | BYTES | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n)<br/>GEOMETRY | BYTES | | UNKNOWN | Not supported yet | ## Sink Options @@ -101,7 +101,8 @@ semantics (using XA transaction guarantee). ### Tips -> If partition_column is not set, it will run in single concurrency, and if partition_column is set, it will be executed in parallel according to the concurrency of tasks. +> Sink concurrency follows the job or pipeline parallelism. `partition_column` is a JDBC +> source option for splitting reads; it does not control how the sink distributes writes. ## Task Example diff --git a/docs/en/connectors/sink/OceanBase.md b/docs/en/connectors/sink/OceanBase.md index 3efb8cd4c0..2d8f357b88 100644 --- a/docs/en/connectors/sink/OceanBase.md +++ b/docs/en/connectors/sink/OceanBase.md @@ -40,8 +40,10 @@ Write data to OceanBase through JDBC. The sink supports batch and streaming jobs | Mysql Data type | SeaTunnel Data type | |-----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| -| BIT(1)<br/>INT UNSIGNED | BOOLEAN | -| TINYINT<br/>TINYINT UNSIGNED<br/>SMALLINT<br/>SMALLINT UNSIGNED<br/>MEDIUMINT<br/>MEDIUMINT UNSIGNED<br/>INT<br/>INTEGER<br/>YEAR | INT | +| BIT(1)<br/>TINYINT(1) | BOOLEAN | +| TINYINT | TINYINT | +| TINYINT UNSIGNED<br/>SMALLINT | SMALLINT | +| SMALLINT UNSIGNED<br/>MEDIUMINT<br/>MEDIUMINT UNSIGNED<br/>INT<br/>INTEGER<br/>YEAR | INT | | INT UNSIGNED<br/>INTEGER UNSIGNED<br/>BIGINT | BIGINT | | BIGINT UNSIGNED | DECIMAL(20,0) | | DECIMAL(x,y)(Get the designated column's specified column size.<38) | DECIMAL(x,y) | @@ -49,27 +51,34 @@ Write data to OceanBase through JDBC. The sink supports batch and streaming jobs | DECIMAL UNSIGNED | DECIMAL((Get the designated column's specified column size)+1,<br/>(Gets the designated column's number of digits to right of the decimal point.))) | | FLOAT<br/>FLOAT UNSIGNED | FLOAT | | DOUBLE<br/>DOUBLE UNSIGNED | DOUBLE | -| CHAR<br/>VARCHAR<br/>TINYTEXT<br/>MEDIUMTEXT<br/>TEXT<br/>LONGTEXT<br/>JSON | STRING | +| CHAR<br/>VARCHAR<br/>TINYTEXT<br/>MEDIUMTEXT<br/>TEXT<br/>LONGTEXT<br/>JSON<br/>ENUM | STRING | | DATE | DATE | | TIME | TIME | -| DATETIME<br/>TIMESTAMP | TIMESTAMP | -| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n) | BYTES | -| GEOMETRY<br/>UNKNOWN | Not supported yet | +| DATETIME | TIMESTAMP | +| TIMESTAMP | TIMESTAMP_TZ | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n)<br/>GEOMETRY | BYTES | +| UNKNOWN | Not supported yet | ### Oracle Mode -| Oracle Data type | SeaTunnel Data type | -|-----------------------------------------------------------|---------------------| -| Number(p), p <= 9 | INT | -| Number(p), p <= 18 | BIGINT | -| Number(p), p > 18 | DECIMAL(38,18) | -| REAL<br/> BINARY_FLOAT | FLOAT | -| BINARY_DOUBLE | DOUBLE | -| CHAR<br/>NCHAR<br/>NVARCHAR2<br/>NCLOB<br/>CLOB<br/>ROWID | STRING | -| DATE | DATE | -| TIMESTAMP<br/>TIMESTAMP WITH LOCAL TIME ZONE | TIMESTAMP | -| BLOB<br/>RAW<br/>LONG RAW<br/>BFILE | BYTES | -| UNKNOWN | Not supported yet | +| Oracle Data type | SeaTunnel Data type | +|-----------------------------------------------------------------------------------------------------|---------------------| +| Integer | DECIMAL(38,0) | +| Number(p), p <= 9 | INT | +| Number(p), p <= 18 | BIGINT | +| Number(p), 18 < p < 38 | DECIMAL(p,0) | +| Number(p), p >= 38 | DECIMAL(38,0) | +| Number(p,s), s > 0 | DECIMAL(p, min(s, 18)) | +| Number(p,s), s < 0 | Normalized to `DECIMAL(p - s, 0)` (or narrowed types when decimal narrowing applies). | +| Float | DECIMAL(38,18) | +| REAL<br/> BINARY_FLOAT | FLOAT | +| BINARY_DOUBLE | DOUBLE | +| CHAR<br/>NCHAR<br/>VARCHAR<br/>VARCHAR2<br/>NVARCHAR2<br/>NCLOB<br/>CLOB<br/>LONG<br/>XML<br/>ROWID | STRING | +| DATE | TIMESTAMP | +| TIMESTAMP | TIMESTAMP | +| TIMESTAMP WITH TIME ZONE<br/>TIMESTAMP WITH LOCAL TIME ZONE | TIMESTAMP_TZ | +| BLOB<br/>RAW<br/>LONG RAW<br/>BFILE | BYTES | +| UNKNOWN | Not supported yet | ## Sink Options diff --git a/docs/en/connectors/sink/Oracle.md b/docs/en/connectors/sink/Oracle.md index 2923fb483f..5023bd5a4a 100644 --- a/docs/en/connectors/sink/Oracle.md +++ b/docs/en/connectors/sink/Oracle.md @@ -99,7 +99,8 @@ semantics (using XA transaction guarantee). ### Tips -> If partition_column is not set, it will run in single concurrency, and if partition_column is set, it will be executed in parallel according to the concurrency of tasks. +> Sink concurrency follows the job or pipeline parallelism. `partition_column` is a JDBC +> source option for splitting reads; it does not control how the sink distributes writes. ## Task Example diff --git a/docs/en/connectors/sink/Pulsar.md b/docs/en/connectors/sink/Pulsar.md index c86b6b2f84..b3a86cf3e9 100644 --- a/docs/en/connectors/sink/Pulsar.md +++ b/docs/en/connectors/sink/Pulsar.md @@ -40,7 +40,7 @@ The Pulsar sink writes SeaTunnel rows to Apache Pulsar topics. It can write to o | avro_row_nullable | Boolean | No | false | Effective when `format = "avro"`. Set `true` when a ROW field (including a ROW nested in another ROW, an ARRAY or a MAP) may be `null`, so the ROW is written as a nullable Avro union instead of a bare record; the reader must enable the same option. See [Avro format](../formats/avro.md#avro_row_nullable). | | semantics | Enum | No | AT_LEAST_ONCE | Write consistency. Valid values: `NON`, `AT_LEAST_ONCE`, `EXACTLY_ONCE`. | | transaction_timeout | Int | No | 600 | Pulsar transaction timeout in seconds. Used by `EXACTLY_ONCE`. | -| pulsar.config | Map | No | - | Extra producer properties passed to the Pulsar producer client. | +| pulsar.config | Map | No | - | Extra key/value properties attached to the Pulsar producer as producer metadata. They are not producer client settings. | | message.routing.mode | Enum | No | RoundRobinPartition | Routing mode for partitioned topics. Valid values: `SinglePartition`, `RoundRobinPartition`. | | partition_key_fields | Array | No | - | Fields used to build the Pulsar message key. | | multi_table_sink_replica | Int | No | 1 | Writer replica count for multi-table writes. | @@ -101,7 +101,9 @@ This option is only used when `semantics = "EXACTLY_ONCE"`. If a transaction is ### pulsar.config [Map] -Extra Pulsar producer properties. These properties are passed to the Pulsar producer client. +Extra key/value properties attached to the Pulsar producer through `ProducerBuilder.properties(...)`. Pulsar keeps them as application-defined producer metadata, for example to identify the producer in topic stats. + +These entries are not applied as producer client configuration, so producer settings such as `sendTimeoutMs`, `batchingMaxMessages` or compression cannot be tuned through this option. ### message.routing.mode [Enum] @@ -156,9 +158,6 @@ sink { client.service-url = "pulsar://localhost:6650" admin.service-url = "http://localhost:8080" format = json - pulsar.config = { - sendTimeoutMs = 30000 - } } } ``` @@ -247,7 +246,7 @@ sink { ### Write With Pulsar Producer Properties -Pass extra producer properties via `pulsar.config`. This is forwarded to the Pulsar producer client and can be used for tuning (timeouts, batching, compression, etc.). +Use `pulsar.config` to attach key/value metadata to the producer. The entries are passed to `ProducerBuilder.properties(...)`; they do not change producer client settings such as timeouts, batching or compression. ```hocon sink { @@ -257,8 +256,8 @@ sink { admin.service-url = "http://localhost:8080" format = json pulsar.config = { - sendTimeoutMs = 30000 - batchingMaxMessages = 1000 + application = "seatunnel" + owner = "data-platform" } } } diff --git a/docs/en/connectors/source/DB2.md b/docs/en/connectors/source/DB2.md index a5c3da9f7d..4774b6c08d 100644 --- a/docs/en/connectors/source/DB2.md +++ b/docs/en/connectors/source/DB2.md @@ -51,9 +51,9 @@ Read data from DB2 through JDBC. DB2 requires the IBM `db2jcc` driver; SeaTunnel | DB2 Data Type | SeaTunnel Data Type | |------------------------------------------------------------------------------------------------------|---------------------| | BOOLEAN | BOOLEAN | -| SMALLINT | SHORT | -| INT<br/>INTEGER<br/> | INTEGER | -| BIGINT | LONG | +| SMALLINT | SMALLINT | +| INT<br/>INTEGER<br/> | INT | +| BIGINT | BIGINT | | DECIMAL<br/>DEC<br/>NUMERIC<br/>NUM | DECIMAL(precision, scale) | | REAL | FLOAT | | FLOAT<br/>DOUBLE<br/>DOUBLE PRECISION<br/>DECFLOAT | DOUBLE | diff --git a/docs/en/connectors/source/DuckDB.md b/docs/en/connectors/source/DuckDB.md index 8d425f2890..69e4a6a1a8 100644 --- a/docs/en/connectors/source/DuckDB.md +++ b/docs/en/connectors/source/DuckDB.md @@ -89,7 +89,7 @@ DuckDB `TIME` values preserve microsecond precision when read or written through | password | String | No | - | Connection instance password | | query | String | No | - | Query statement. Either `query`, `table_path` or `table_list` must be configured. | | connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete | -| partition_column | String | No | - | The column name for parallelism's partition, only support numeric type primary key, and only can config one column. | +| partition_column | String | No | - | The column used to split data for parallel reading. Only one column can be configured, and it must be one of the **Supported split data type** listed in [Parallel Reader](#parallel-reader) (string, number or date). | | partition_lower_bound | String | No | - | The partition_column min value for scan, if not set SeaTunnel will query database get min value. | | partition_upper_bound | String | No | - | The partition_column max value for scan, if not set SeaTunnel will query database get max value. | | partition_num | Int | No | 10 | The number of partition count, only support positive integer. default value is 10 | diff --git a/docs/en/connectors/source/Firebase.md b/docs/en/connectors/source/Firebase.md index 8500427fd3..90dd0e073d 100644 --- a/docs/en/connectors/source/Firebase.md +++ b/docs/en/connectors/source/Firebase.md @@ -13,7 +13,7 @@ The Firebase Source Connector allows reading data from Google Firebase Realtime - [x] Batch - [ ] Stream - [ ] Exactly-Once -- [x] Column Projection +- [ ] Column Projection --- diff --git a/docs/en/connectors/source/Http.md b/docs/en/connectors/source/Http.md index c716409916..ea27c0d190 100644 --- a/docs/en/connectors/source/Http.md +++ b/docs/en/connectors/source/Http.md @@ -218,7 +218,7 @@ source { sink { LocalFile { path = "/tmp/download" - file_format = "binary" + file_format_type = "binary" } } ``` diff --git a/docs/en/connectors/source/IoTDB.md b/docs/en/connectors/source/IoTDB.md index c159a07004..6958645cb7 100644 --- a/docs/en/connectors/source/IoTDB.md +++ b/docs/en/connectors/source/IoTDB.md @@ -48,10 +48,13 @@ The current source runs a bounded SQL query. It is suitable for batch reads and | FLOAT | FLOAT | | DOUBLE | DOUBLE | | TEXT | STRING | -| STRING | STRING | | time column | BIGINT | | time column | TIMESTAMP | +The source is built on the IoTDB `0.13.x` session client, which only knows the data types listed +above. Columns that use the `STRING`, `BLOB`, `DATE` or `TIMESTAMP` data types introduced in +IoTDB 1.3 are not supported; store string values as `TEXT` if they need to be read by this source. + ## Source Options Use either a root-level `sql` and `schema` for one table, or `tables_configs` for independently configured tables. The existing single-table configuration remains supported. diff --git a/docs/en/connectors/source/MongoDB.md b/docs/en/connectors/source/MongoDB.md index 82cd231ba9..683442e950 100644 --- a/docs/en/connectors/source/MongoDB.md +++ b/docs/en/connectors/source/MongoDB.md @@ -32,8 +32,11 @@ You can narrow what is read and what columns are returned without scanning the w - Use `flat.sync-string` to capture the whole document as one JSON `STRING` column when no fixed schema is needed. -The connector tracks split progress through checkpoints, so a restarted job resumes from the -last committed split instead of rescanning the whole collection. +Checkpoints only store splits that have not been fully read yet; a split does not record a read +position inside its range. When a job restarts from a checkpoint, the enumerator restores those +pending splits and then re-runs the split strategy over the whole collection, so documents that +were already read can be read again. Make the downstream write idempotent (for example, an upsert +on the primary key) if duplicates after a restore are not acceptable. ## Supported DataSource Info diff --git a/docs/en/connectors/source/OceanBase.md b/docs/en/connectors/source/OceanBase.md index 3d9ffb4ad3..df443f2273 100644 --- a/docs/en/connectors/source/OceanBase.md +++ b/docs/en/connectors/source/OceanBase.md @@ -57,7 +57,7 @@ Read OceanBase data through JDBC. OceanBase can run in MySQL-compatible mode or | TIME | TIME | | DATETIME | TIMESTAMP | | TIMESTAMP | TIMESTAMP_TZ | -| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINAR<br/>BIT(n)<br/>GEOMETRY | BYTES | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n)<br/>GEOMETRY | BYTES | ### Oracle Mode diff --git a/docs/en/connectors/source/PostgreSQL.md b/docs/en/connectors/source/PostgreSQL.md index 1628a23def..d9816a8b62 100644 --- a/docs/en/connectors/source/PostgreSQL.md +++ b/docs/en/connectors/source/PostgreSQL.md @@ -88,7 +88,7 @@ Read external data source data through JDBC. | password | String | No | - | Connection instance password [...] | query | String | No | - | Query statement. Required when neither `table_path` nor `table_list` is configured. [...] | connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete [...] -| partition_column | String | No | - | The column name for parallelism's partition, only support numeric type,Only support numeric type primary key, and only can config one column. [...] +| partition_column | String | No | - | The column used to split data for parallel reading. Only one column can be configured, and it must be one of the **Supported split data type** listed in [Parallel Reader](#parallel-reader) (string, number or date). [...] | partition_lower_bound | String | No | - | The partition_column min value for scan, if not set SeaTunnel will query database get min value. [...] | partition_upper_bound | String | No | - | The partition_column max value for scan, if not set SeaTunnel will query database get max value. [...] | partition_num | Int | No | 10 | The number of partition count, only support positive integer. default value is 10 [...] diff --git a/docs/en/connectors/source/Typesense.md b/docs/en/connectors/source/Typesense.md index e1b24a8d51..145a258754 100644 --- a/docs/en/connectors/source/Typesense.md +++ b/docs/en/connectors/source/Typesense.md @@ -22,9 +22,12 @@ The source is bounded. Each job reads every document that matches the configured - [ ] [Exactly-Once](../../introduction/concepts/connector-v2-features.md) - [ ] [cdc](../../introduction/concepts/connector-v2-features.md) - [x] [Schema](../../introduction/concepts/connector-v2-features.md) -- [x] [Parallelism](../../introduction/concepts/connector-v2-features.md) +- [ ] [Parallelism](../../introduction/concepts/connector-v2-features.md) - [ ] [User-Defined Splits Support](../../introduction/concepts/connector-v2-features.md) +The split enumerator creates exactly one split for the configured collection. A job can set +`parallelism` greater than `1`, but only one reader receives the split and the other readers stay idle. + ## Options | Name | Type | Required | Default | diff --git a/docs/en/engines/zeta/engine-jar-storage-mode.md b/docs/en/engines/zeta/engine-jar-storage-mode.md index 41ff99faaf..5239de8839 100644 --- a/docs/en/engines/zeta/engine-jar-storage-mode.md +++ b/docs/en/engines/zeta/engine-jar-storage-mode.md @@ -41,7 +41,7 @@ All options live under `seatunnel.engine.jar-storage` in `seatunnel.yaml`; a top |-------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------| | enable | false | Whether to enable uploading the connector jar package to the engine. The default enabled state is false. | | connector-jar-storage-mode | SHARED | Engine-side jar package storage mode selection. There are two optional modes, SHARED and ISOLATED. The default Jar package storage mode is SHARED. | -| connector-jar-storage-path | " " | User-defined jar package storage path. | +| connector-jar-storage-path | "" | User-defined jar package storage path. When it is empty, the engine stores the jar packages under the SeaTunnel home directory (`SEATUNNEL_HOME`). | | connector-jar-cleanup-task-interval | 3600s | Engine-side jar package cleaning scheduled task execution interval. | | connector-jar-expiry-time | 600s | Engine-side jar package storage expiration time. | diff --git a/docs/en/getting-started/docker/docker.md b/docs/en/getting-started/docker/docker.md index a57e14a657..80c4c89c6a 100644 --- a/docs/en/getting-started/docker/docker.md +++ b/docs/en/getting-started/docker/docker.md @@ -62,27 +62,29 @@ docker build -f src/main/docker/Dockerfile --build-arg VERSION=3.1.0-SNAPSHOT -t docker images | grep apache/seatunnel ``` -The Dockerfile is like this: +The Dockerfile (`seatunnel-dist/src/main/docker/Dockerfile`) is a two-stage build: the first stage unpacks the +binary package and adjusts the logging and Hazelcast configuration, and the final stage installs `python3` and copies +the prepared `/opt/seatunnel` directory: ```dockerfile -FROM eclipse-temurin:11-jdk +FROM eclipse-temurin:11-jdk as builder ARG VERSION -# Build from Source Code And Copy it into image -COPY ./target/apache-seatunnel-${VERSION}-bin.tar.gz /opt/ - -# Download From Internet -# Please Note this file only include fake/console connector, You'll need to download the other connectors manually -# wget -P /opt https://dlcdn.apache.org/seatunnel/${VERSION}/apache-seatunnel-${VERSION}-bin.tar.gz +COPY ./target/apache-seatunnel-${VERSION}-bin.tar.gz /opt/ RUN cd /opt && \ tar -zxvf apache-seatunnel-${VERSION}-bin.tar.gz && \ mv apache-seatunnel-${VERSION} seatunnel && \ rm apache-seatunnel-${VERSION}-bin.tar.gz && \ sed -i 's/#rootLogger.appenderRef.consoleStdout.ref/rootLogger.appenderRef.consoleStdout.ref/' seatunnel/config/log4j2.properties && \ sed -i 's/#rootLogger.appenderRef.consoleStderr.ref/rootLogger.appenderRef.consoleStderr.ref/' seatunnel/config/log4j2.properties && \ - sed -i 's/rootLogger.appenderRef.file.ref/#rootLogger.appenderRef.file.ref/' seatunnel/config/log4j2.properties && \ + sed -i 's/rootLogger.appenderRef.file.ref/#rootLogger.appenderRef.file.ref/' seatunnel/config/log4j2.properties && \ cp seatunnel/config/hazelcast-master.yaml seatunnel/config/hazelcast-worker.yaml +FROM eclipse-temurin:11-jdk +RUN apt-get update \ + && apt-get install -y --no-install-recommends python3 \ + && rm -rf /var/lib/apt/lists/* +COPY --from=builder /opt/seatunnel /opt/seatunnel WORKDIR /opt/seatunnel ``` diff --git a/docs/en/introduction/concepts/schema-feature.md b/docs/en/introduction/concepts/schema-feature.md index 12f4bd35c5..5b11c8a3d0 100644 --- a/docs/en/introduction/concepts/schema-feature.md +++ b/docs/en/introduction/concepts/schema-feature.md @@ -215,7 +215,7 @@ Constraint keys is a list of config used to define the constraint keys in schema constraintKeys = [ { constraintName = "id_index" - constraintType = KEY + constraintType = INDEX_KEY constraintColumns = [ { columnName = "id" @@ -229,7 +229,7 @@ constraintKeys = [ | Field | Required | Default Value | Description | |:------------------|:---------|:--------------|-------------------------------------------------------------------------------------------------------------------------------------------| | constraintName | Yes | - | The name of the constraintKey | -| constraintType | No | KEY | The type of the constraintKey | +| constraintType | Yes | - | The type of the constraintKey. Supported values are listed in [What constraintType supported at now](#what-constrainttype-supported-at-now). | | constraintColumns | Yes | - | The column list in the constraintKey. Each column must contain `columnName`. For `INDEX_KEY`/`UNIQUE_KEY`/`FOREIGN_KEY`, `sortType` is supported (ASC/DESC, default ASC). For `VECTOR_INDEX_KEY`, `indexName` is optional; `indexType` and `metricType` are optional at the schema parsing level but may be required by specific connectors that create vector indexes (e.g., Milvus). | #### What constraintType supported at now diff --git a/docs/zh/connectors/connector-faq.md b/docs/zh/connectors/connector-faq.md index e812bdda3f..b7eb23efbd 100644 --- a/docs/zh/connectors/connector-faq.md +++ b/docs/zh/connectors/connector-faq.md @@ -62,8 +62,6 @@ CDC(Change Data Capture)连接器从数据库事务日志中读取实时变 1. **连接器相关问题** → 直接进入对应连接器页面,滚动到 **FAQ** 章节。 2. **类型与 Schema 问题**(例如 JSON schema、Debezium schema、Oracle CLOB、MySQL TINYINT、ORC/Parquet 映射)→ 参阅[类型与 Schema 常见问题](./type-schema-faq.md)。 3. **文件与对象存储问题**(例如 MinIO/S3、小文件、save mode、多表文件路径)→ 参阅[文件与对象存储常见问题](./file-object-storage-faq.md)。 -4. **跨连接器主题**(例如「SeaTunnel 是否支持 CDC?」「`schema_save_mode` 是什么?」)→ 参阅[通用常见问题](../faq.md)。 -5. **仍未解决?** → 在 [GitHub Issues](https://github.com/apache/seatunnel/issues) 中搜索,或通过[邮件列表](https://lists.apache.org/[email protected])联系社区。 -2. **跨 Sink 写入主题**(例如「`generate_sink_sql` 和 `query` 怎么选?」「`schema_save_mode` 是什么?」「`data_save_mode` 是否支持?」「`enable_upsert` 什么时候生效?」)→ 参阅 [Sink 写入模式与 Save Mode](./common-options/sink-write-modes.md)。 -3. **其他跨连接器主题**(例如「SeaTunnel 是否支持 CDC?」)→ 参阅[通用常见问题](../faq.md)。 -4. **仍未解决?** → 在 [GitHub Issues](https://github.com/apache/seatunnel/issues) 中搜索,或通过[邮件列表](https://lists.apache.org/[email protected])联系社区。 +4. **跨 Sink 写入主题**(例如「`generate_sink_sql` 和 `query` 怎么选?」「`schema_save_mode` 是什么?」「`data_save_mode` 是否支持?」「`enable_upsert` 什么时候生效?」)→ 参阅 [Sink 写入模式与 Save Mode](./common-options/sink-write-modes.md)。 +5. **其他跨连接器主题**(例如「SeaTunnel 是否支持 CDC?」)→ 参阅[通用常见问题](../faq.md)。 +6. **仍未解决?** → 在 [GitHub Issues](https://github.com/apache/seatunnel/issues) 中搜索,或通过[邮件列表](https://lists.apache.org/[email protected])联系社区。 diff --git a/docs/zh/connectors/sink/Assert.md b/docs/zh/connectors/sink/Assert.md index aa8d21c214..de93447c9c 100644 --- a/docs/zh/connectors/sink/Assert.md +++ b/docs/zh/connectors/sink/Assert.md @@ -29,15 +29,15 @@ Assert 是一个用于校验任务输出结果的数据接收器。它可以按 |------------------------------------------------------------------------------------------------|-------------------------------------------------|----------|---------| | rules | ConfigMap | 是 | - | | rules.field_rules | ConfigList | 否 | - | -| rules.field_rules.field_name | string\|ConfigMap | 是 | - | -| rules.field_rules.field_type | string | 否 | - | +| rules.field_rules.field_name | string | 是 | - | +| rules.field_rules.field_type | string\|ConfigMap | 否 | - | | rules.field_rules.field_value | ConfigList | 否 | - | | rules.field_rules.field_value.rule_type | string | 否 | - | | rules.field_rules.field_value.rule_value | numeric | 否 | - | | rules.field_rules.field_value.equals_to | boolean\|numeric\|string\|ConfigList\|ConfigMap | 否 | - | | rules.row_rules | ConfigList | 否 | - | | rules.row_rules.rule_type | string | 否 | - | -| rules.row_rules.rule_value | string | 否 | - | +| rules.row_rules.rule_value | numeric | 否 | - | | rules.catalog_table_rule | ConfigMap | 否 | - | | rules.catalog_table_rule.primary_key_rule | ConfigMap | 否 | - | | rules.catalog_table_rule.primary_key_rule.primary_key_name | string | 否 | - | diff --git a/docs/zh/connectors/sink/DB2.md b/docs/zh/connectors/sink/DB2.md index b801f227a9..06abe7ca5a 100644 --- a/docs/zh/connectors/sink/DB2.md +++ b/docs/zh/connectors/sink/DB2.md @@ -46,7 +46,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | DB2 数据类型 | SeaTunnel 数据类型 | |------------------------------------------------------------------------------------------------------|--------------------| | BOOLEAN | BOOLEAN | -| SMALLINT | SHORT | +| SMALLINT | SMALLINT | | INT<br/>INTEGER | INT | | BIGINT | BIGINT | | DECIMAL<br/>DEC<br/>NUMERIC<br/>NUM | DECIMAL((获取指定列的指定列大小),<br/>(获取指定列小数点右边的位数。))) | diff --git a/docs/zh/connectors/sink/DuckDB.md b/docs/zh/connectors/sink/DuckDB.md index cbebd7fd92..ae111dda44 100644 --- a/docs/zh/connectors/sink/DuckDB.md +++ b/docs/zh/connectors/sink/DuckDB.md @@ -32,7 +32,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; - [ ] [精确一次](../../introduction/concepts/connector-v2-features.md) - [x] [CDC](../../introduction/concepts/connector-v2-features.md) -- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md) +- [x] [定时刷新](../../introduction/concepts/connector-v2-features.md) > 通用 JDBC Sink 通过 XA 事务实现精确一次;DuckDB JDBC 驱动没有 XA 数据源。DuckDB 作业不要设置 > `is_exactly_once = true`。 @@ -97,7 +97,8 @@ JDBC 连接器读取和写入 DuckDB `TIME` 时保留微秒精度。该类型表 ### 提示 -> 如果未设置 partition_column,它将以单一并发运行,如果设置了 partition_column,它将根据任务的并发度并行执行。 +> Sink 的写入并发度由作业或管道的并行度决定。`partition_column` 是 JDBC Source +> 用于拆分读取的选项,不控制 Sink 的写入分配。 ## 任务示例 @@ -113,7 +114,7 @@ env { source { FakeSource { parallelism = 1 - row_num = 1000 + row.num = 1000 schema = { fields { id = "int" diff --git a/docs/zh/connectors/sink/Greenplum.md b/docs/zh/connectors/sink/Greenplum.md index f943a350ae..a52ccb6daa 100644 --- a/docs/zh/connectors/sink/Greenplum.md +++ b/docs/zh/connectors/sink/Greenplum.md @@ -181,40 +181,6 @@ sink { } ``` -### CDC 数据流写入 - -```hocon -env { - parallelism = 1 - job.mode = "STREAMING" - checkpoint.interval = 10000 -} - -source { - MySQL-CDC { - username = "cdc_user" - password = "cdc_pass" - table-list = ["cdc_test.orders"] - base-url = "jdbc:mysql://localhost:3306/cdc_test" - startup.mode = "initial" - } -} - -sink { - Jdbc { - driver = "org.postgresql.Driver" - url = "jdbc:postgresql://localhost:5432/testdb" - username = "tester" - password = "pivotal" - generate_sink_sql = true - database = "testdb" - table = "orders" - primary_keys = ["id"] - enable_upsert = true - } -} -``` - ## 变更日志 <ChangeLog /> diff --git a/docs/zh/connectors/sink/Mysql.md b/docs/zh/connectors/sink/Mysql.md index 7e6eb7b996..b782f86f8d 100644 --- a/docs/zh/connectors/sink/Mysql.md +++ b/docs/zh/connectors/sink/Mysql.md @@ -65,7 +65,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | TIME | TIME | | DATETIME | TIMESTAMP | | TIMESTAMP | TIMESTAMP_TZ | -| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINAR<br/>BIT(n)<br/>GEOMETRY | BYTES | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n)<br/>GEOMETRY | BYTES | | UNKNOWN | Not supported yet | ## Sink 参数 @@ -101,7 +101,8 @@ import ChangeLog from '../changelog/connector-jdbc.md'; ### 提示 ->如果未设置partition_column,它将以单并发运行,如果设置了partition_column,它将根据任务的并发性并行执行。 +> Sink 的写入并发度由作业或管道的并行度决定。`partition_column` 是 JDBC Source +> 用于拆分读取的选项,不控制 Sink 的写入分配。 ## 任务示例 diff --git a/docs/zh/connectors/sink/OceanBase.md b/docs/zh/connectors/sink/OceanBase.md index 9252c746d4..391db36e5d 100644 --- a/docs/zh/connectors/sink/OceanBase.md +++ b/docs/zh/connectors/sink/OceanBase.md @@ -40,8 +40,10 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | MySQL 数据类型 | SeaTunnel 数据类型 | |-----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| -| BIT(1)<br/>INT UNSIGNED | BOOLEAN | -| TINYINT<br/>TINYINT UNSIGNED<br/>SMALLINT<br/>SMALLINT UNSIGNED<br/>MEDIUMINT<br/>MEDIUMINT UNSIGNED<br/>INT<br/>INTEGER<br/>YEAR | INT | +| BIT(1)<br/>TINYINT(1) | BOOLEAN | +| TINYINT | TINYINT | +| TINYINT UNSIGNED<br/>SMALLINT | SMALLINT | +| SMALLINT UNSIGNED<br/>MEDIUMINT<br/>MEDIUMINT UNSIGNED<br/>INT<br/>INTEGER<br/>YEAR | INT | | INT UNSIGNED<br/>INTEGER UNSIGNED<br/>BIGINT | BIGINT | | BIGINT UNSIGNED | DECIMAL(20,0) | | DECIMAL(x,y)(获取指定列的指定列大小<38) | DECIMAL(x,y) | @@ -49,27 +51,34 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | DECIMAL UNSIGNED | DECIMAL((获取指定列的指定列大小)+1,<br/>(获取指定列小数点右侧的位数。))) | | FLOAT<br/>FLOAT UNSIGNED | FLOAT | | DOUBLE<br/>DOUBLE UNSIGNED | DOUBLE | -| CHAR<br/>VARCHAR<br/>TINYTEXT<br/>MEDIUMTEXT<br/>TEXT<br/>LONGTEXT<br/>JSON | STRING | +| CHAR<br/>VARCHAR<br/>TINYTEXT<br/>MEDIUMTEXT<br/>TEXT<br/>LONGTEXT<br/>JSON<br/>ENUM | STRING | | DATE | DATE | | TIME | TIME | -| DATETIME<br/>TIMESTAMP | TIMESTAMP | -| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n) | BYTES | -| GEOMETRY<br/>UNKNOWN | 暂不支持 | +| DATETIME | TIMESTAMP | +| TIMESTAMP | TIMESTAMP_TZ | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n)<br/>GEOMETRY | BYTES | +| UNKNOWN | 暂不支持 | ### Oracle 模式 -| Oracle 数据类型 | SeaTunnel 数据类型 | -|-----------------------------------------------------------|---------------------| -| Number(p), p <= 9 | INT | -| Number(p), p <= 18 | BIGINT | -| Number(p), p > 18 | DECIMAL(38,18) | -| REAL<br/> BINARY_FLOAT | FLOAT | -| BINARY_DOUBLE | DOUBLE | -| CHAR<br/>NCHAR<br/>NVARCHAR2<br/>NCLOB<br/>CLOB<br/>ROWID | STRING | -| DATE | DATE | -| TIMESTAMP<br/>TIMESTAMP WITH LOCAL TIME ZONE | TIMESTAMP | -| BLOB<br/>RAW<br/>LONG RAW<br/>BFILE | BYTES | -| UNKNOWN | 暂不支持 | +| Oracle 数据类型 | SeaTunnel 数据类型 | +|---------------|------------------| +| Integer | DECIMAL(38,0) | +| Number(p), p <= 9 | INT | +| Number(p), p <= 18 | BIGINT | +| Number(p), 18 < p < 38 | DECIMAL(p,0) | +| Number(p), p >= 38 | DECIMAL(38,0) | +| Number(p,s), s > 0 | DECIMAL(p, min(s, 18)) | +| Number(p,s), s < 0 | 规范化为 `DECIMAL(p - s, 0)`(启用 decimal narrowing 时为收窄类型)。 | +| Float | DECIMAL(38,18) | +| REAL<br/> BINARY_FLOAT | FLOAT | +| BINARY_DOUBLE | DOUBLE | +| CHAR<br/>NCHAR<br/>VARCHAR<br/>VARCHAR2<br/>NVARCHAR2<br/>NCLOB<br/>CLOB<br/>LONG<br/>XML<br/>ROWID | STRING | +| DATE | TIMESTAMP | +| TIMESTAMP | TIMESTAMP | +| TIMESTAMP WITH TIME ZONE<br/>TIMESTAMP WITH LOCAL TIME ZONE | TIMESTAMP_TZ | +| BLOB<br/>RAW<br/>LONG RAW<br/>BFILE | BYTES | +| UNKNOWN | 暂不支持 | ## Sink 选项 diff --git a/docs/zh/connectors/sink/Oracle.md b/docs/zh/connectors/sink/Oracle.md index 58559c812c..6552fa2415 100644 --- a/docs/zh/connectors/sink/Oracle.md +++ b/docs/zh/connectors/sink/Oracle.md @@ -98,7 +98,8 @@ import ChangeLog from '../changelog/connector-jdbc.md'; ### 提示 ->如果未设置partition_column,它将以单并发运行,如果设置了partition_column,它将根据任务的并发数并行执行。 +> Sink 的写入并发度由作业或管道的并行度决定。`partition_column` 是 JDBC Source +> 用于拆分读取的选项,不控制 Sink 的写入分配。 ## 任务示例 diff --git a/docs/zh/connectors/sink/Pulsar.md b/docs/zh/connectors/sink/Pulsar.md index 8077991d88..108aa2412d 100644 --- a/docs/zh/connectors/sink/Pulsar.md +++ b/docs/zh/connectors/sink/Pulsar.md @@ -40,7 +40,7 @@ Pulsar Sink 用于将 SeaTunnel 数据写入 Apache Pulsar topic。它既可以 | avro_row_nullable | Boolean | 否 | false | 当 `format = "avro"` 时生效。当 ROW 字段(含嵌套在另一个 ROW、ARRAY 或 MAP 中的 ROW)可能为 `null` 时设为 `true`,此时 ROW 以可空 Avro union 而非裸 record 写入;读取端必须开启同一选项。详见 [Avro 格式](../formats/avro.md#avro_row_nullable)。 | | semantics | Enum | 否 | AT_LEAST_ONCE | 写入一致性语义。可选值:`NON`、`AT_LEAST_ONCE`、`EXACTLY_ONCE`。 | | transaction_timeout | Int | 否 | 600 | Pulsar 事务超时时间,单位为秒。用于 `EXACTLY_ONCE`。 | -| pulsar.config | Map | 否 | - | 传给 Pulsar 生产者客户端的额外参数。 | +| pulsar.config | Map | 否 | - | 作为 producer 元数据附加到 Pulsar producer 上的额外 key/value 属性,不是 producer 客户端配置。 | | message.routing.mode | Enum | 否 | RoundRobinPartition | 分区 topic 的消息路由模式。可选值:`SinglePartition`、`RoundRobinPartition`。 | | partition_key_fields | Array | 否 | - | 用于生成 Pulsar 消息 key 的字段。 | | multi_table_sink_replica | Int | 否 | 1 | 多表写入时的写入器副本数。 | @@ -100,7 +100,9 @@ Pulsar 事务超时时间,单位为秒。默认值为 `600`。 ### pulsar.config [Map] -Pulsar 生产者客户端的额外参数。这些参数会传给 Pulsar producer。 +通过 `ProducerBuilder.properties(...)` 附加到 Pulsar producer 上的额外 key/value 属性。Pulsar 会把它们作为应用自定义的 producer 元数据保存,例如用于在 topic 统计信息中识别 producer。 + +这些条目不会作为 producer 客户端配置生效,因此无法通过该参数调整 `sendTimeoutMs`、`batchingMaxMessages` 或压缩等 producer 参数。 ### message.routing.mode [Enum] @@ -155,9 +157,6 @@ sink { client.service-url = "pulsar://localhost:6650" admin.service-url = "http://localhost:8080" format = json - pulsar.config = { - sendTimeoutMs = 30000 - } } } ``` @@ -246,7 +245,7 @@ sink { ### 自定义 Pulsar Producer 属性 -通过 `pulsar.config` 传入额外的 producer 属性,这些配置会透传给 Pulsar producer 客户端,可以用来调整超时、批大小、压缩等参数。 +使用 `pulsar.config` 可以为 producer 附加 key/value 元数据。这些条目会传给 `ProducerBuilder.properties(...)`,不会改变超时、批量发送、压缩等 producer 客户端配置。 ```hocon sink { @@ -256,8 +255,8 @@ sink { admin.service-url = "http://localhost:8080" format = json pulsar.config = { - sendTimeoutMs = 30000 - batchingMaxMessages = 1000 + application = "seatunnel" + owner = "data-platform" } } } diff --git a/docs/zh/connectors/source/DB2.md b/docs/zh/connectors/source/DB2.md index 42fca4182d..b51551c167 100644 --- a/docs/zh/connectors/source/DB2.md +++ b/docs/zh/connectors/source/DB2.md @@ -51,9 +51,9 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | DB2 数据类型 | SeaTunnel 数据类型 | |------------------------------------------------------------------------------------------------------|--------------------| | BOOLEAN | BOOLEAN | -| SMALLINT | SHORT | -| INT<br/>INTEGER | INTEGER | -| BIGINT | LONG | +| SMALLINT | SMALLINT | +| INT<br/>INTEGER | INT | +| BIGINT | BIGINT | | DECIMAL<br/>DEC<br/>NUMERIC<br/>NUM | DECIMAL(precision, scale) | | REAL | FLOAT | | FLOAT<br/>DOUBLE<br/>DOUBLE PRECISION<br/>DECFLOAT | DOUBLE | diff --git a/docs/zh/connectors/source/DuckDB.md b/docs/zh/connectors/source/DuckDB.md index 9a50be1583..69e0e5686a 100644 --- a/docs/zh/connectors/source/DuckDB.md +++ b/docs/zh/connectors/source/DuckDB.md @@ -86,7 +86,7 @@ JDBC 连接器读取和写入 DuckDB `TIME` 时保留微秒精度。该类型表 | password | String | 否 | - | 连接实例密码 | | query | String | 否 | - | 查询语句。`query`、`table_path`、`table_list` 三者必须配置其一 | | connection_check_timeout_sec | Int | 否 | 30 | 等待用于验证连接的数据库操作完成的时间(以秒为单位) | -| partition_column | String | 否 | - | 并行度分区的列名,仅支持数字类型主键,并且只能配置一列。 | +| partition_column | String | 否 | - | 用于并行读取时拆分数据的列,只能配置一列,且必须是[并行读取器](#并行读取器)中列出的**支持的拆分数据类型**(字符串、数字或日期)。 | | partition_lower_bound | String | 否 | - | 扫描的 partition_column 最小值,如果未设置,SeaTunnel 将查询数据库获取最小值。 | | partition_upper_bound | String | 否 | - | 扫描的 partition_column 最大值,如果未设置,SeaTunnel 将查询数据库获取最大值。 | | partition_num | Int | 否 | 10 | 分区计数的数量,仅支持正整数。默认值为 10 | diff --git a/docs/zh/connectors/source/Firebase.md b/docs/zh/connectors/source/Firebase.md index 851cd3d92f..8ac17f8662 100644 --- a/docs/zh/connectors/source/Firebase.md +++ b/docs/zh/connectors/source/Firebase.md @@ -13,7 +13,7 @@ Firebase Source Connector 支持通过 REST API 从 Google Firebase Realtime Dat - [x] 批处理 (Batch) - [ ] 流处理 (Stream) - [ ] 精确一次 (Exactly-Once) -- [x] 列裁剪 (Column Projection) +- [ ] 列裁剪 (Column Projection) --- diff --git a/docs/zh/connectors/source/Http.md b/docs/zh/connectors/source/Http.md index 03adc5767a..1c8c8b2540 100644 --- a/docs/zh/connectors/source/Http.md +++ b/docs/zh/connectors/source/Http.md @@ -216,7 +216,7 @@ source { sink { LocalFile { path = "/tmp/download" - file_format = "binary" + file_format_type = "binary" } } ``` diff --git a/docs/zh/connectors/source/IoTDB.md b/docs/zh/connectors/source/IoTDB.md index 5876c97119..eff265901c 100644 --- a/docs/zh/connectors/source/IoTDB.md +++ b/docs/zh/connectors/source/IoTDB.md @@ -48,10 +48,13 @@ import ChangeLog from '../changelog/connector-iotdb.md'; | FLOAT | FLOAT | | DOUBLE | DOUBLE | | TEXT | STRING | -| STRING | STRING | | 时间列 | BIGINT | | 时间列 | TIMESTAMP | +该 Source 基于 IoTDB `0.13.x` session 客户端实现,只能识别上表列出的数据类型。不支持 IoTDB 1.3 +新增的 `STRING`、`BLOB`、`DATE`、`TIMESTAMP` 数据类型的列;如需通过该 Source 读取字符串数据,请使用 +`TEXT` 类型存储。 + ## Source 选项 单表读取使用根级别的 `sql` 和 `schema`;多表读取使用 `tables_configs`。原有单表配置保持兼容。 diff --git a/docs/zh/connectors/source/MongoDB.md b/docs/zh/connectors/source/MongoDB.md index 9ed2d1c255..8b1897080c 100644 --- a/docs/zh/connectors/source/MongoDB.md +++ b/docs/zh/connectors/source/MongoDB.md @@ -31,8 +31,9 @@ split,实现并行读取。 - 使用 `match.projection` 控制结果中返回的字段。 - 使用 `flat.sync-string` 把整篇文档作为一条 JSON `STRING` 列读入,跳过固定 schema 的定义。 -连接器会通过 checkpoint 跟踪 split 的读取进度,任务重启后会从上次提交的 split 继续读取, -而不需要重新扫描整张集合。 +checkpoint 只保存尚未读完的 split,split 本身不会记录范围内的读取位置。任务从 checkpoint 重启时, +enumerator 会先恢复这些待处理的 split,然后对整张集合重新执行一次切分策略,因此已经读过的文档可能会被 +再次读取。如果无法接受恢复后的重复数据,请让下游写入保持幂等(例如按主键 upsert)。 ## 支持的数据源信息 diff --git a/docs/zh/connectors/source/OceanBase.md b/docs/zh/connectors/source/OceanBase.md index f664063fd4..849661ea1f 100644 --- a/docs/zh/connectors/source/OceanBase.md +++ b/docs/zh/connectors/source/OceanBase.md @@ -57,7 +57,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | TIME | TIME | | DATETIME | TIMESTAMP | | TIMESTAMP | TIMESTAMP_TZ | -| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINAR<br/>BIT(n)<br/>GEOMETRY | BYTES | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINARY<br/>BIT(n)<br/>GEOMETRY | BYTES | ### Oracle 模式 diff --git a/docs/zh/connectors/source/PostgreSQL.md b/docs/zh/connectors/source/PostgreSQL.md index d06bc3aef7..6710fbee15 100644 --- a/docs/zh/connectors/source/PostgreSQL.md +++ b/docs/zh/connectors/source/PostgreSQL.md @@ -88,7 +88,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | password | String | 否 | - | 连接实例的密码 [...] | query | String | 否 | - | 查询语句。未配置 `table_path` 和 `table_list` 时必填。 [...] | connection_check_timeout_sec | Int | 否 | 30 | 用于验证连接的数据库操作完成的等待时间(秒) [...] -| partition_column | String | 否 | - | 用于并行化的分区列名,仅支持数字类型,<br/> 仅支持数字类型主键,并且只能配置一列。 [...] +| partition_column | String | 否 | - | 用于并行读取时拆分数据的列,只能配置一列,且必须是[并行读取器](#并行读取器)中列出的**支持的拆分数据类型**(字符串、数字或日期)。 [...] | partition_lower_bound | String | 否 | - | 扫描的 partition_column 的最小值,如果未设置,SeaTunnel 将查询数据库获取最小值。 [...] | partition_upper_bound | String | 否 | - | 扫描的 partition_column 的最大值,如果未设置,SeaTunnel 将查询数据库获取最大值。 [...] | partition_num | Int | 否 | 10 | 分区数量,仅支持正整数。默认值为 10 [...] diff --git a/docs/zh/connectors/source/Typesense.md b/docs/zh/connectors/source/Typesense.md index 1366be9787..b824fee513 100644 --- a/docs/zh/connectors/source/Typesense.md +++ b/docs/zh/connectors/source/Typesense.md @@ -22,9 +22,12 @@ Source 是有界读取:每次作业只会读取一次匹配 `query` 的全部 - [ ] [精确一次](../../introduction/concepts/connector-v2-features.md) - [ ] [cdc](../../introduction/concepts/connector-v2-features.md) - [x] [Schema](../../introduction/concepts/connector-v2-features.md) -- [x] [并行度](../../introduction/concepts/connector-v2-features.md) +- [ ] [并行度](../../introduction/concepts/connector-v2-features.md) - [ ] [支持用户定义的拆分](../../introduction/concepts/connector-v2-features.md) +split enumerator 只会为配置的 collection 创建一个 split。作业可以把 `parallelism` 设置为大于 `1`, +但只有一个 reader 会分到这个 split,其余 reader 处于空闲状态。 + ## 选项 | 名称 | 类型 | 必填 | 默认值 | diff --git a/docs/zh/engines/zeta/engine-jar-storage-mode.md b/docs/zh/engines/zeta/engine-jar-storage-mode.md index d8b74436ee..b2c6d032e0 100644 --- a/docs/zh/engines/zeta/engine-jar-storage-mode.md +++ b/docs/zh/engines/zeta/engine-jar-storage-mode.md @@ -42,7 +42,7 @@ COMMON_ PLUGIN_ JARS 指的是连接器所依赖的第三方 Jar 包, CONNECTO |-------------------------------------|--------|-------------------------------------------------------------------------| | enable | false | 是否启用上传连接器 Jar 包到引擎。默认启用状态为 false。 | | connector-jar-storage-mode | SHARED | 引擎端 Jar 包存储模式选择。有两个可选模式,SHARED(共享)和 ISOLATED(隔离)。默认的 Jar 包存储模式是 SHARED。 | -| connector-jar-storage-path | " " | 用户自定义的 Jar 包存储路径。 | +| connector-jar-storage-path | "" | 用户自定义的 Jar 包存储路径。为空时,引擎会把 Jar 包存储在 SeaTunnel 安装目录(`SEATUNNEL_HOME`)下。 | | connector-jar-cleanup-task-interval | 3600s | 引擎端 Jar 包清理定时任务执行间隔。 | | connector-jar-expiry-time | 600s | 引擎端 Jar 包存储过期时间。 | diff --git a/docs/zh/getting-started/docker/docker.md b/docs/zh/getting-started/docker/docker.md index 580d1c1f96..c02d0dc3db 100644 --- a/docs/zh/getting-started/docker/docker.md +++ b/docs/zh/getting-started/docker/docker.md @@ -62,27 +62,28 @@ docker build -f src/main/docker/Dockerfile --build-arg VERSION=3.1.0-SNAPSHOT -t docker images | grep apache/seatunnel ``` -Dockerfile文件内容为: +Dockerfile(`seatunnel-dist/src/main/docker/Dockerfile`)采用两阶段构建:第一阶段解压二进制包并调整日志与 Hazelcast +配置,最终阶段安装 `python3` 并复制准备好的 `/opt/seatunnel` 目录。文件内容为: ```dockerfile -FROM eclipse-temurin:11-jdk +FROM eclipse-temurin:11-jdk as builder ARG VERSION -# Build from Source Code And Copy it into image -COPY ./target/apache-seatunnel-${VERSION}-bin.tar.gz /opt/ - -# Download From Internet -# Please Note this file only include fake/console connector, You'll need to download the other connectors manually -# wget -P /opt https://dlcdn.apache.org/seatunnel/${VERSION}/apache-seatunnel-${VERSION}-bin.tar.gz +COPY ./target/apache-seatunnel-${VERSION}-bin.tar.gz /opt/ RUN cd /opt && \ tar -zxvf apache-seatunnel-${VERSION}-bin.tar.gz && \ mv apache-seatunnel-${VERSION} seatunnel && \ rm apache-seatunnel-${VERSION}-bin.tar.gz && \ sed -i 's/#rootLogger.appenderRef.consoleStdout.ref/rootLogger.appenderRef.consoleStdout.ref/' seatunnel/config/log4j2.properties && \ sed -i 's/#rootLogger.appenderRef.consoleStderr.ref/rootLogger.appenderRef.consoleStderr.ref/' seatunnel/config/log4j2.properties && \ - sed -i 's/rootLogger.appenderRef.file.ref/#rootLogger.appenderRef.file.ref/' seatunnel/config/log4j2.properties && \ + sed -i 's/rootLogger.appenderRef.file.ref/#rootLogger.appenderRef.file.ref/' seatunnel/config/log4j2.properties && \ cp seatunnel/config/hazelcast-master.yaml seatunnel/config/hazelcast-worker.yaml +FROM eclipse-temurin:11-jdk +RUN apt-get update \ + && apt-get install -y --no-install-recommends python3 \ + && rm -rf /var/lib/apt/lists/* +COPY --from=builder /opt/seatunnel /opt/seatunnel WORKDIR /opt/seatunnel ``` diff --git a/docs/zh/getting-started/locally/quick-start-seatunnel-engine.md b/docs/zh/getting-started/locally/quick-start-seatunnel-engine.md index 08c6d24dca..72e687d569 100644 --- a/docs/zh/getting-started/locally/quick-start-seatunnel-engine.md +++ b/docs/zh/getting-started/locally/quick-start-seatunnel-engine.md @@ -112,7 +112,7 @@ cd "apache-seatunnel-${version}" 在 Windows 上,请在 SeaTunnel 目录下运行对应的批处理入口: ```bat -cd apache-seatunnel-3.0.0 +cd apache-seatunnel-3.1.0 bin\seatunnel.cmd --config config\v2.batch.config.template -m local ``` diff --git a/docs/zh/introduction/concepts/schema-feature.md b/docs/zh/introduction/concepts/schema-feature.md index 27d0863806..400fe3c4b9 100644 --- a/docs/zh/introduction/concepts/schema-feature.md +++ b/docs/zh/introduction/concepts/schema-feature.md @@ -215,7 +215,7 @@ primaryKey { constraintKeys = [ { constraintName = "id_index" - constraintType = KEY + constraintType = INDEX_KEY constraintColumns = [ { columnName = "id" @@ -229,7 +229,7 @@ constraintKeys = [ | 字段 | 是否必须 | 默认值 | 描述 | |:------------------|:-----|:----|------------------------------------------------------------------------| | constraintName | 是 | - | 约束键的名称 | -| constraintType | 否 | KEY | 约束键的类型 | +| constraintType | 是 | - | 约束键的类型,可选值见[目前支持哪些约束类型](#目前支持哪些约束类型)。 | | constraintColumns | 是 | - | 约束键中的列。每列必须包含 `columnName`。当 `constraintType` 为 `INDEX_KEY`/`UNIQUE_KEY`/`FOREIGN_KEY` 时,支持 `sortType`(ASC/DESC,默认 ASC);当 `constraintType` 为 `VECTOR_INDEX_KEY` 时,`indexName` 为可选项;`indexType` 和 `metricType` 在 schema 解析层面为可选项,但需要创建向量索引的 connector(如 Milvus)可能要求必填。 | #### 目前支持哪些约束类型
