This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12951-95f9fd5e51175849b741fa4a0d7288c9b1c5c712 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit f3381ba41361cd39ec15dc40cd19239dd470e411 Author: Jast <[email protected]> AuthorDate: Tue Oct 27 11:57:59 2026 +0000 [Docs] Fix DuckDB, Kingbase, SmbFile, Rabbitmq and other connector doc inaccuracies verified against source code (#12951) Co-authored-by: zhangshenghang <[email protected]> --- docs/en/connectors/sink/DuckDB.md | 28 ++++++++++++--------- docs/en/connectors/sink/Hudi.md | 2 +- docs/en/connectors/sink/Kingbase.md | 9 ++++--- docs/en/connectors/sink/Rabbitmq.md | 12 +++------ docs/en/connectors/source/DuckDB.md | 21 ++++++++-------- docs/en/connectors/source/FakeSource.md | 2 +- docs/en/connectors/source/Hbase.md | 4 +-- docs/en/connectors/source/Kingbase.md | 9 ++++--- docs/en/connectors/source/SmbFile.md | 3 +++ docs/zh/connectors/sink/Aerospike.md | 40 ++++++++++++++++++++++++++++++ docs/zh/connectors/sink/ClickhouseFile.md | 2 +- docs/zh/connectors/sink/DuckDB.md | 10 +++++--- docs/zh/connectors/sink/Hudi.md | 2 +- docs/zh/connectors/sink/Kingbase.md | 9 ++++--- docs/zh/connectors/sink/Rabbitmq.md | 12 +++------ docs/zh/connectors/source/DuckDB.md | 21 ++++++++-------- docs/zh/connectors/source/Hbase.md | 4 +-- docs/zh/connectors/source/Kingbase.md | 9 ++++--- docs/zh/connectors/source/Opengauss-CDC.md | 2 ++ docs/zh/connectors/source/SmbFile.md | 3 +++ 20 files changed, 127 insertions(+), 77 deletions(-) diff --git a/docs/en/connectors/sink/DuckDB.md b/docs/en/connectors/sink/DuckDB.md index 9040c9f965..b525871ab0 100644 --- a/docs/en/connectors/sink/DuckDB.md +++ b/docs/en/connectors/sink/DuckDB.md @@ -50,18 +50,22 @@ works against a local database file path (`jdbc:duckdb:/path/to/database.db`) or | SeaTunnel Data Type | DuckDB Data Type | |---------------------------------------------------------------------|------------------| -| BOOLEAN | BOOLEAN | -| TINYINT<br/>SMALLINT<br/>INT | INTEGER | -| BIGINT | BIGINT | -| DECIMAL(x,y)(Get the designated column's specified column size.<38) | DECIMAL(x,y) | -| DECIMAL(x,y)(Get the designated column's specified column size.>38) | DECIMAL(38,18) | -| FLOAT | FLOAT | -| DOUBLE | DOUBLE | -| STRING | VARCHAR | -| DATE | DATE | -| TIME | TIME | -| TIMESTAMP | TIMESTAMP | -| BYTES<br/>ARRAY<br/>ROW<br/>MAP | BLOB | +| BOOLEAN | BOOLEAN | +| TINYINT | TINYINT | +| SMALLINT | SMALLINT | +| INT | INTEGER | +| BIGINT | BIGINT | +| DECIMAL(x,y)(Get the designated column's specified column size.<38) | DECIMAL(x,y) | +| DECIMAL(x,y)(Get the designated column's specified column size.>38) | DECIMAL(38,y) | +| FLOAT | FLOAT | +| DOUBLE | DOUBLE | +| STRING | VARCHAR | +| DATE | DATE | +| TIME | TIME | +| TIMESTAMP | TIMESTAMP | +| TIMESTAMP_TZ | TIMESTAMP WITH TIME ZONE | +| BYTES | BLOB | +| ARRAY<br/>ROW<br/>MAP | Not supported | DuckDB `TIME` values preserve microsecond precision when read or written through the JDBC connector. They represent a local time of day without a time zone. diff --git a/docs/en/connectors/sink/Hudi.md b/docs/en/connectors/sink/Hudi.md index 546793e498..f6ec61e93d 100644 --- a/docs/en/connectors/sink/Hudi.md +++ b/docs/en/connectors/sink/Hudi.md @@ -89,7 +89,7 @@ Note: When this configuration corresponds to a single table, you can flatten the `precombine_field` The precombine field of hudi table, its are used in preCombining before actual write. -### index_type [string] +### index_type [enum] `index_type` The index type of hudi table. Currently, `BLOOM`, `SIMPLE`, and `GLOBAL_BLOOM` are supported. diff --git a/docs/en/connectors/sink/Kingbase.md b/docs/en/connectors/sink/Kingbase.md index db0376c09c..1559534629 100644 --- a/docs/en/connectors/sink/Kingbase.md +++ b/docs/en/connectors/sink/Kingbase.md @@ -43,15 +43,16 @@ import ChangeLog from '../changelog/connector-jdbc.md'; |----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | BOOL | BOOLEAN | | INT2 | SHORT | -| SMALLSERIAL <br/>SERIAL <br/>INT4 | INT | +| SMALLSERIAL | SMALLINT | +| SERIAL <br/>INT4 | INT | | INT8 <br/>BIGSERIAL | BIGINT | | FLOAT4 | FLOAT | | FLOAT8 | DOUBLE | | NUMERIC | DECIMAL((Get the designated column's specified column size),<br/>(Gets the designated column's number of digits to right of the decimal point.))) | | BPCHAR <br/>CHARACTER <br/>VARCHAR <br/>TEXT | STRING | -| TIMESTAMP | LOCALDATETIME | -| TIME | LOCALTIME | -| DATE | LOCALDATE | +| TIMESTAMP | TIMESTAMP | +| TIME | TIME | +| DATE | DATE | | Other data type | Not supported yet | ## Sink Options diff --git a/docs/en/connectors/sink/Rabbitmq.md b/docs/en/connectors/sink/Rabbitmq.md index dcf8afce93..f2cfa74cab 100644 --- a/docs/en/connectors/sink/Rabbitmq.md +++ b/docs/en/connectors/sink/Rabbitmq.md @@ -130,6 +130,8 @@ connection TCP establishment timeout in milliseconds; zero for infinite In addition to the above parameters that must be specified by the RabbitMQ client, the user can also specify multiple non-mandatory parameters for the client, covering [all the parameters specified in the official RabbitMQ document](https://www.rabbitmq.com/configure.html). +Note: the sink currently parses this option but does not apply its entries to the RabbitMQ client. To tune connection behavior, use the dedicated options above such as `connection_timeout`, `network_recovery_interval`, `AUTOMATIC_RECOVERY_ENABLED`, and `topology_recovery_enabled`. + ### common options Sink plugin common parameters, please refer to [Sink Common Options](../common-options/sink-common-options.md) for details @@ -194,10 +196,6 @@ sink { username = "guest" password = "guest" queue_name = "test1" - rabbitmq.config = { - requested-heartbeat = 10 - connection-timeout = 10 - } } } ``` @@ -235,10 +233,6 @@ sink { durable = true exclusive = false auto_delete = false - rabbitmq.config = { - requested-heartbeat = 10 - connection-timeout = 10 - } } } ``` @@ -273,7 +267,7 @@ Yes. The sink publishes messages to RabbitMQ by binding to the target queue or r ### How does RabbitMQ sink handle network reconnects and timeouts? -You can tune client connection resilience using the `rabbitmq.config` block (such as `connection-timeout`, `requested-heartbeat`, and retry intervals) to prevent premature disconnection during transient network blips. +Use the dedicated options to tune client connection resilience: `AUTOMATIC_RECOVERY_ENABLED` enables automatic connection recovery, `network_recovery_interval` sets how long recovery waits before reconnecting (in ms), `topology_recovery_enabled` enables topology recovery, and `connection_timeout` bounds the TCP connection establishment time (in ms). The `rabbitmq.config` block is accepted for backward compatibility, but its entries are currently not applied by the sink. ## Changelog diff --git a/docs/en/connectors/source/DuckDB.md b/docs/en/connectors/source/DuckDB.md index 86bea19bf0..8d425f2890 100644 --- a/docs/en/connectors/source/DuckDB.md +++ b/docs/en/connectors/source/DuckDB.md @@ -57,22 +57,23 @@ MySQL automatic DDL cannot create a full-column primary key on `LONGTEXT`. If a | DuckDB Data Type | SeaTunnel Data Type | |---------------------------------------------------------------------|---------------------| | BOOLEAN | BOOLEAN | -| TINYINT | TINYINT | -| UTINYINT<br/>SMALLINT | SMALLINT | -| USMALLINT<br/>INTEGER | INT | -| UINTEGER<br/>BIGINT | BIGINT | -| UBIGINT | DECIMAL(20,0) | -| HUGEINT | DECIMAL(38,0) | +| TINYINT<br/>UTINYINT | TINYINT | +| SMALLINT<br/>USMALLINT | SMALLINT | +| INTEGER<br/>UINTEGER | INT | +| BIGINT<br/>UBIGINT | BIGINT | +| HUGEINT<br/>UHUGEINT<br/>BIGNUM | DECIMAL(38,0) | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL(x,y)(Get the designated column's specified column size.<38) | DECIMAL(x,y) | -| DECIMAL(x,y)(Get the designated column's specified column size.>38) | DECIMAL(38,18) | +| DECIMAL(x,y)(Get the designated column's specified column size.>38) | DECIMAL(38,y) | | VARCHAR<br/>CHAR<br/>TEXT<br/>JSON<br/>UUID<br/>INTERVAL | STRING | | BIT<br/>ENUM | STRING | | DATE | DATE | | TIME | TIME | -| TIMESTAMP<br/>TIMESTAMP WITH TIME ZONE | TIMESTAMP | -| BLOB<br/>ARRAY<br/>STRUCT<br/>MAP | BYTES | +| TIMESTAMP | TIMESTAMP | +| TIMESTAMP WITH TIME ZONE | TIMESTAMP_TZ | +| BLOB | BYTES | +| ARRAY<br/>STRUCT<br/>MAP | STRING | DuckDB `TIME` values preserve microsecond precision when read or written through the JDBC connector. They represent a local time of day without a time zone. @@ -86,7 +87,7 @@ DuckDB `TIME` values preserve microsecond precision when read or written through | driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use DuckDB the value is `org.duckdb.DuckDBDriver`. | | username | String | No | - | Connection instance user name | | password | String | No | - | Connection instance password | -| query | String | Yes | - | Query statement | +| query | String | No | - | Query statement. Either `query`, `table_path` or `table_list` must be configured. | | connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete | | partition_column | String | No | - | The column name for parallelism's partition, only support numeric type primary key, and only can config one column. | | partition_lower_bound | String | No | - | The partition_column min value for scan, if not set SeaTunnel will query database get min value. | diff --git a/docs/en/connectors/source/FakeSource.md b/docs/en/connectors/source/FakeSource.md index f7e04724c3..3474e62875 100644 --- a/docs/en/connectors/source/FakeSource.md +++ b/docs/en/connectors/source/FakeSource.md @@ -63,7 +63,7 @@ FakeSource is a virtual source that generates rows from a user-defined schema. I | float.min | float | No | 0 | The min value of float data that connector generated | | float.max | float | No | 0x1.fffffeP+127 | The max value of float data that connector generated | | float.template | list | No | - | The template list of float type that connector generated, if user configured it, connector will randomly select an item from the template list | -| double.fake.mode | string | No | range | The fake mode of generating float data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `double.template` option | +| double.fake.mode | string | No | range | The fake mode of generating double data, support `range` and `template`, default `range`，if use configured it to `template`, user should also configured `double.template` option | | double.min | double | No | 0 | The min value of double data that connector generated | | double.max | double | No | 0x1.fffffffffffffP+1023 | The max value of double data that connector generated | | double.template | list | No | - | The template list of double type that connector generated, if user configured it, connector will randomly select an item from the template list | diff --git a/docs/en/connectors/source/Hbase.md b/docs/en/connectors/source/Hbase.md index 409829e709..ac0456e383 100644 --- a/docs/en/connectors/source/Hbase.md +++ b/docs/en/connectors/source/Hbase.md @@ -23,7 +23,7 @@ source: changes that happen after the scan starts are not delivered. - [x] [batch](../../introduction/concepts/connector-v2-features.md) - [ ] [stream](../../introduction/concepts/connector-v2-features.md) - [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md) -- [x] [schema projection](../../introduction/concepts/connector-v2-features.md) +- [x] [column projection](../../introduction/concepts/connector-v2-features.md) - [x] [parallelism](../../introduction/concepts/connector-v2-features.md) - [ ] [support user-defined split](../../introduction/concepts/connector-v2-features.md) @@ -72,7 +72,7 @@ The caching parameter sets the number of rows fetched per server trip during sca ### batch -The batch parameter sets the maximum number of columns returned per scan. This is useful for rows with many columns to avoid fetching excessive data at once, thus saving memory and improving performance. +The batch parameter sets the maximum number of cells returned per scan. This is useful for rows with many columns to avoid fetching excessive data at once, thus saving memory and improving performance. ### cache_blocks diff --git a/docs/en/connectors/source/Kingbase.md b/docs/en/connectors/source/Kingbase.md index c818334b6b..2a6dac05d9 100644 --- a/docs/en/connectors/source/Kingbase.md +++ b/docs/en/connectors/source/Kingbase.md @@ -44,15 +44,16 @@ Read external data source data through JDBC. |-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | BOOL | BOOLEAN | | INT2 | SHORT | -| SMALLSERIAL <br/>SERIAL <br/>INT4 | INT | +| SMALLSERIAL | SMALLINT | +| SERIAL <br/>INT4 | INT | | INT8 <br/>BIGSERIAL | BIGINT | | FLOAT4 | FLOAT | | FLOAT8 | DOUBLE | | NUMERIC | DECIMAL((Get the designated column's specified column size),<br/>(Gets the designated column's number of digits to right of the decimal point.))) | | BPCHAR<br/>CHARACTER<br/>VARCHAR<br/>TEXT | STRING | -| TIMESTAMP | LOCALDATETIME | -| TIME | LOCALTIME | -| DATE | LOCALDATE | +| TIMESTAMP | TIMESTAMP | +| TIME | TIME | +| DATE | DATE | | Other data type | Not supported yet | ## Source Options diff --git a/docs/en/connectors/source/SmbFile.md b/docs/en/connectors/source/SmbFile.md index f24e4cddea..f4c7a6f3e3 100644 --- a/docs/en/connectors/source/SmbFile.md +++ b/docs/en/connectors/source/SmbFile.md @@ -97,6 +97,7 @@ The File does not have a specific type list, and we can indicate which SeaTunnel | read_columns | List | No | - | The read column list of the data source, user can use it to implement field projection. | | read_partitions | list | no | - | The partitions that the user wants to read, e.g. `["year=2024"]`. When set, only these partitions are read. | | sheet_name | String | No | - | Reader the sheet of the workbook, Only used when file_format is excel. | +| excel_use_header_line | Boolean | No | false | Use the first Excel row as column headers for name-based mapping. Only used when file_format_type is excel. | | xml_row_tag | String | No | - | Specifies the tag name of the data rows within the XML file, only used when file_format is xml. | | xml_use_attr_format | Boolean | No | - | Specifies whether to process data using the tag attribute format, only used when file_format is xml. | | csv_use_header_line | boolean | no | false | Whether to use the header line to parse the file, only used when the file_format is `csv` and the file contains the header line that match RFC 4180. | @@ -109,6 +110,8 @@ The File does not have a specific type list, and we can indicate which SeaTunnel | filename_extension | String | No | - | Filter filename extension, which used for filtering files with specific extension. Example: `csv` `.txt` `json` `.xml`. | | excel_engine | String | No | POI | Only used when file_format is excel. Supported engines are `POI` and `EasyExcel`. | | poi_excel_max_file_size | Long | No | 52428800 | Only used when file_format is excel and excel_engine is POI. The maximum Excel file size in bytes (default 50 MB). | +| markdown_rag_metadata_enabled | Boolean | No | false | Whether to append RAG-oriented metadata columns when reading markdown files. Only valid when file_format_type is markdown. | +| pdf_rag_metadata_enabled | Boolean | No | false | Whether to append RAG-oriented metadata columns when reading PDF files. Only valid when file_format_type is pdf. | | quote_char | String | No | " | A single character that encloses CSV fields, allowing fields with commas, line breaks, or quotes to be read correctly. | | escape_char | String | No | - | A single character that allows the quote or other special characters to appear inside a CSV field without ending the field. | | metalake_type | String | No | gravitino | The type of metalake service, currently supports `gravitino`. | diff --git a/docs/zh/connectors/sink/Aerospike.md b/docs/zh/connectors/sink/Aerospike.md index 65c4ac23ce..e77932360b 100644 --- a/docs/zh/connectors/sink/Aerospike.md +++ b/docs/zh/connectors/sink/Aerospike.md @@ -154,6 +154,46 @@ sink { } ``` +### 使用 `kv` 格式将 CDC 记录写入 Aerospike + +对于流式作业（例如 MySQL CDC），使用 `data_format = "kv"` 将每个配置的字段作为独立的 Aerospike bin 写入。这样每个字段单独存储，而不是打包进单个 JSON 或 map bin，可以让下游查询更高效。 + +```hocon +env { + parallelism = 2 + job.mode = "STREAMING" + checkpoint.interval = 10000 +} + +source { + Mysql-CDC { + url = "jdbc:mysql://127.0.0.1:3306/seatunnel" + username = "root" + password = "******" + table-names = ["seatunnel.user"] + } +} + +sink { + Aerospike { + host = "aerospike-host" + port = 3000 + namespace = "test" + set = "user" + key = "id" + data_format = "kv" + write_timeout = 500 + schema { + field { + id = "INTEGER" + name = "STRING" + email = "STRING" + } + } + } +} +``` + ## 更新日志 <ChangeLog /> diff --git a/docs/zh/connectors/sink/ClickhouseFile.md b/docs/zh/connectors/sink/ClickhouseFile.md index 6c612c2df8..6f16719c79 100644 --- a/docs/zh/connectors/sink/ClickhouseFile.md +++ b/docs/zh/connectors/sink/ClickhouseFile.md @@ -11,13 +11,13 @@ import ChangeLog from '../changelog/connector-clickhouse.md'; ## 主要特性 - [ ] [精准一次](../../introduction/concepts/connector-v2-features.md) +- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md) :::tip 提示 你也可以采用JDBC的方式将数据写入Clickhouse。 ::: -- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md) ## 接收器选项 diff --git a/docs/zh/connectors/sink/DuckDB.md b/docs/zh/connectors/sink/DuckDB.md index c180788a2b..cbebd7fd92 100644 --- a/docs/zh/connectors/sink/DuckDB.md +++ b/docs/zh/connectors/sink/DuckDB.md @@ -47,17 +47,21 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | SeaTunnel 数据类型 | DuckDB 数据类型 | |---------------------------------|----------------| | BOOLEAN | BOOLEAN | -| TINYINT<br/>SMALLINT<br/>INT | INTEGER | +| TINYINT | TINYINT | +| SMALLINT | SMALLINT | +| INT | INTEGER | | BIGINT | BIGINT | | DECIMAL(x,y)(获取指定列的指定列大小.<38) | DECIMAL(x,y) | -| DECIMAL(x,y)(获取指定列的指定列大小.>38) | DECIMAL(38,18) | +| DECIMAL(x,y)(获取指定列的指定列大小.>38) | DECIMAL(38,y) | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | STRING | VARCHAR | | DATE | DATE | | TIME | TIME | | TIMESTAMP | TIMESTAMP | -| BYTES<br/>ARRAY<br/>ROW<br/>MAP | BLOB | +| TIMESTAMP_TZ | TIMESTAMP WITH TIME ZONE | +| BYTES | BLOB | +| ARRAY<br/>ROW<br/>MAP | 不支持 | JDBC 连接器读取和写入 DuckDB `TIME` 时保留微秒精度。该类型表示不带时区的本地时刻。 diff --git a/docs/zh/connectors/sink/Hudi.md b/docs/zh/connectors/sink/Hudi.md index adf7b4ee2f..1c4498095d 100644 --- a/docs/zh/connectors/sink/Hudi.md +++ b/docs/zh/connectors/sink/Hudi.md @@ -89,7 +89,7 @@ SeaTunnel Hudi sink 会写入 Hudi 数据文件和 `.hoodie` 元数据，但不 `precombine_field` Hudi 表的预合并字段,它用于在写入前进行预合并. -### index_type [string] +### index_type [enum] `index_type` Hudi 表的索引类型。当前支持 `BLOOM`、`SIMPLE`、`GLOBAL_BLOOM`。 diff --git a/docs/zh/connectors/sink/Kingbase.md b/docs/zh/connectors/sink/Kingbase.md index 6998fccad5..494cbc0a73 100644 --- a/docs/zh/connectors/sink/Kingbase.md +++ b/docs/zh/connectors/sink/Kingbase.md @@ -42,15 +42,16 @@ import ChangeLog from '../changelog/connector-jdbc.md'; |----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| | BOOL | BOOLEAN | | INT2 | SHORT | -| SMALLSERIAL <br/>SERIAL <br/>INT4 | INT | +| SMALLSERIAL | SMALLINT | +| SERIAL <br/>INT4 | INT | | INT8 <br/>BIGSERIAL | BIGINT | | FLOAT4 | FLOAT | | FLOAT8 | DOUBLE | | NUMERIC | DECIMAL((获取指定列的指定列大小),<br/>(获取指定列小数点右边的位数。))) | | BPCHAR <br/>CHARACTER <br/>VARCHAR <br/>TEXT | STRING | -| TIMESTAMP | LOCALDATETIME | -| TIME | LOCALTIME | -| DATE | LOCALDATE | +| TIMESTAMP | TIMESTAMP | +| TIME | TIME | +| DATE | DATE | | 其他数据类型 | 暂不支持 | ## Sink 选项 diff --git a/docs/zh/connectors/sink/Rabbitmq.md b/docs/zh/connectors/sink/Rabbitmq.md index 4d0674c862..9221c00d5f 100644 --- a/docs/zh/connectors/sink/Rabbitmq.md +++ b/docs/zh/connectors/sink/Rabbitmq.md @@ -150,6 +150,8 @@ TCP连接建立的超时时间，单位为毫秒；0代表不限制。 除了上面提及必须设置的RabbitMQ客户端参数，你也还可以为客户端指定多个非强制参数，参见 [RabbitMQ官方文档参数设置](https://www.rabbitmq.com/configure.html)。 +注意：Sink 目前仅解析该配置项，但不会将其中的参数应用到 RabbitMQ 客户端。如需调优连接行为，请使用上文的专用选项，例如 `connection_timeout`、`network_recovery_interval`、`AUTOMATIC_RECOVERY_ENABLED` 和 `topology_recovery_enabled`。 + ### common options Sink插件常用参数，请参考[Sink常用选项](../common-options/sink-common-options.md)获取更多细节信息。 @@ -193,10 +195,6 @@ sink { username = "guest" password = "guest" queue_name = "test1" - rabbitmq.config = { - requested-heartbeat = 10 - connection-timeout = 10 - } } } ``` @@ -234,10 +232,6 @@ sink { durable = true exclusive = false auto_delete = false - rabbitmq.config = { - requested-heartbeat = 10 - connection-timeout = 10 - } } } ``` @@ -272,7 +266,7 @@ sink { ### RabbitMQ Sink 如何处理网络重连和超时？ -可以通过 `rabbitmq.config` 配置块调优客户端连接参数（如 `connection-timeout`、`requested-heartbeat` 等），以应对网络短暂抖动并提高连接稳定性。 +使用专用选项调优客户端连接能力：`AUTOMATIC_RECOVERY_ENABLED` 用于启用自动连接恢复，`network_recovery_interval` 设置重连前的等待时长（毫秒），`topology_recovery_enabled` 用于启用拓扑恢复，`connection_timeout` 限制 TCP 连接建立的超时时间（毫秒）。`rabbitmq.config` 配置块仅为兼容旧配置保留，其中的参数目前不会被 Sink 应用。 ## 变更日志 diff --git a/docs/zh/connectors/source/DuckDB.md b/docs/zh/connectors/source/DuckDB.md index d980655836..9a50be1583 100644 --- a/docs/zh/connectors/source/DuckDB.md +++ b/docs/zh/connectors/source/DuckDB.md @@ -54,22 +54,23 @@ MySQL 自动建表无法在 `LONGTEXT` 上创建使用完整列值的主键。 | DuckDB 数据类型 | SeaTunnel 数据类型 | |----------------------------------------------------------|----------------| | BOOLEAN | BOOLEAN | -| TINYINT | TINYINT | -| UTINYINT<br/>SMALLINT | SMALLINT | -| USMALLINT<br/>INTEGER | INT | -| UINTEGER<br/>BIGINT | BIGINT | -| UBIGINT | DECIMAL(20,0) | -| HUGEINT | DECIMAL(38,0) | +| TINYINT<br/>UTINYINT | TINYINT | +| SMALLINT<br/>USMALLINT | SMALLINT | +| INTEGER<br/>UINTEGER | INT | +| BIGINT<br/>UBIGINT | BIGINT | +| HUGEINT<br/>UHUGEINT<br/>BIGNUM | DECIMAL(38,0) | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL(x,y)(获取指定列的指定列大小.<38) | DECIMAL(x,y) | -| DECIMAL(x,y)(获取指定列的指定列大小.>38) | DECIMAL(38,18) | +| DECIMAL(x,y)(获取指定列的指定列大小.>38) | DECIMAL(38,y) | | VARCHAR<br/>CHAR<br/>TEXT<br/>JSON<br/>UUID<br/>INTERVAL | STRING | | BIT<br/>ENUM | STRING | | DATE | DATE | | TIME | TIME | -| TIMESTAMP<br/>TIMESTAMP WITH TIME ZONE | TIMESTAMP | -| BLOB<br/>ARRAY<br/>STRUCT<br/>MAP | BYTES | +| TIMESTAMP | TIMESTAMP | +| TIMESTAMP WITH TIME ZONE | TIMESTAMP_TZ | +| BLOB | BYTES | +| ARRAY<br/>STRUCT<br/>MAP | STRING | JDBC 连接器读取和写入 DuckDB `TIME` 时保留微秒精度。该类型表示不带时区的本地时刻。 @@ -83,7 +84,7 @@ JDBC 连接器读取和写入 DuckDB `TIME` 时保留微秒精度。该类型表 | driver | String | 是 | - | 用于连接到远程数据源的 jdbc 类名，<br/> 如果您使用 DuckDB，值为 `org.duckdb.DuckDBDriver`。 | | username | String | 否 | - | 连接实例用户名 | | password | String | 否 | - | 连接实例密码 | -| query | String | 是 | - | 查询语句 | +| query | String | 否 | - | 查询语句。`query`、`table_path`、`table_list` 三者必须配置其一 | | connection_check_timeout_sec | Int | 否 | 30 | 等待用于验证连接的数据库操作完成的时间（以秒为单位） | | partition_column | String | 否 | - | 并行度分区的列名，仅支持数字类型主键，并且只能配置一列。 | | partition_lower_bound | String | 否 | - | 扫描的 partition_column 最小值，如果未设置，SeaTunnel 将查询数据库获取最小值。 | diff --git a/docs/zh/connectors/source/Hbase.md b/docs/zh/connectors/source/Hbase.md index 6a6b897133..e9d33c067b 100644 --- a/docs/zh/connectors/source/Hbase.md +++ b/docs/zh/connectors/source/Hbase.md @@ -21,7 +21,7 @@ import ChangeLog from '../changelog/connector-hbase.md'; - [x] [批处理](../../introduction/concepts/connector-v2-features.md) - [ ] [流处理](../../introduction/concepts/connector-v2-features.md) - [ ] [精确一次](../../introduction/concepts/connector-v2-features.md) -- [x] [模式投影](../../introduction/concepts/connector-v2-features.md) +- [x] [列投影](../../introduction/concepts/connector-v2-features.md) - [x] [并行度](../../introduction/concepts/connector-v2-features.md) - [ ] [支持用户定义的拆分](../../introduction/concepts/connector-v2-features.md) @@ -70,7 +70,7 @@ caching 参数用于设置在扫描过程中一次从服务器端获取的行数 ### batch -batch 参数用于设置在扫描过程中每次返回的最大列数。这对于处理有很多列的行特别有用，可以避免一次性返回过多数据，从而节省内存并提高性能。 +batch 参数用于设置在扫描过程中每次返回的最大单元格数。这对于处理有很多列的行特别有用，可以避免一次性返回过多数据，从而节省内存并提高性能。 ### cache_blocks diff --git a/docs/zh/connectors/source/Kingbase.md b/docs/zh/connectors/source/Kingbase.md index 30f64c1428..f0f085e645 100644 --- a/docs/zh/connectors/source/Kingbase.md +++ b/docs/zh/connectors/source/Kingbase.md @@ -44,15 +44,16 @@ import ChangeLog from '../changelog/connector-jdbc.md'; |------------------|------------------| | BOOL | BOOLEAN | | INT2 | SHORT | -| SMALLSERIAL <br/>SERIAL <br/>INT4 | INT | +| SMALLSERIAL | SMALLINT | +| SERIAL <br/>INT4 | INT | | INT8 <br/>BIGSERIAL | BIGINT | | FLOAT4 | FLOAT | | FLOAT8 | DOUBLE | | NUMERIC | DECIMAL((获取指定列的指定列大小),<br/>(获取指定列小数点右边的位数。))) | | BPCHAR<br/>CHARACTER<br/>VARCHAR<br/>TEXT | STRING | -| TIMESTAMP | LOCALDATETIME | -| TIME | LOCALTIME | -| DATE | LOCALDATE | +| TIMESTAMP | TIMESTAMP | +| TIME | TIME | +| DATE | DATE | | 其他数据类型 | 暂不支持 | ## 源选项 diff --git a/docs/zh/connectors/source/Opengauss-CDC.md b/docs/zh/connectors/source/Opengauss-CDC.md index 9f334c27e7..768e03827b 100644 --- a/docs/zh/connectors/source/Opengauss-CDC.md +++ b/docs/zh/connectors/source/Opengauss-CDC.md @@ -45,6 +45,8 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL; select 'ALTER TABLE ' || schemaname || '.' || tablename || ' REPLICA IDENTITY FULL;' from pg_tables where schemaname = 'YourTableSchema' ``` +3. 当 `startup.mode = initial` 时，连接器会在开始增量读取之前先读取表快照。快照阶段使用与增量阶段相同的拆分规则并行执行，快照完成后从增量 LSN 继续读取。 + ## 数据类型映射 | Opengauss Data type | SeaTunnel Data type | diff --git a/docs/zh/connectors/source/SmbFile.md b/docs/zh/connectors/source/SmbFile.md index 6c42be4837..578efa01b7 100644 --- a/docs/zh/connectors/source/SmbFile.md +++ b/docs/zh/connectors/source/SmbFile.md @@ -97,6 +97,7 @@ import ChangeLog from '../changelog/connector-file-smb.md'; | read_columns | List | 否 | - | 数据源的读取列列表，用户可以用它实现字段投影 | | read_partitions | list | 否 | - | 用户希望读取的分区列表，例如 `["year=2024"]`。设置后仅读取这些分区。 | | sheet_name | String | 否 | - | 读取工作簿中的 sheet，仅在 file_format 为 excel 时使用 | +| excel_use_header_line | Boolean | 否 | false | 使用 Excel 的第一行作为列标题进行基于名称的字段映射。仅在 file_format_type 为 excel 时使用 | | xml_row_tag | String | 否 | - | 指定 XML 文件中数据行的标签名，仅在 file_format 为 xml 时使用 | | xml_use_attr_format | Boolean | 否 | - | 指定是否使用标签属性格式处理数据，仅在 file_format 为 xml 时使用 | | csv_use_header_line | boolean | 否 | false | 是否使用标题行来解析文件，仅在 file_format 为 `csv` 且文件包含符合 RFC 4180 的标题行时使用 | @@ -109,6 +110,8 @@ import ChangeLog from '../changelog/connector-file-smb.md'; | filename_extension | String | 否 | - | 文件扩展名过滤，用于过滤特定扩展名的文件。例如：`csv` `.txt` `json` `.xml` | | excel_engine | String | 否 | POI | 仅在 file_format 为 excel 时使用。支持的引擎为 `POI` 和 `EasyExcel` | | poi_excel_max_file_size | Long | 否 | 52428800 | 仅在 file_format 为 excel 且 excel_engine 为 POI 时使用。POI 引擎可读取的最大 Excel 文件大小（默认 50 MB） | +| markdown_rag_metadata_enabled | Boolean | 否 | false | 读取 markdown 文件时是否附加面向 RAG 的元数据列。仅在 file_format_type 为 markdown 时有效 | +| pdf_rag_metadata_enabled | Boolean | 否 | false | 读取 PDF 文件时是否附加面向 RAG 的元数据列。仅在 file_format_type 为 pdf 时有效 | | quote_char | String | 否 | " | 用于包围 CSV 字段的单个字符，使包含逗号、换行或引号的字段能被正确读取 | | escape_char | String | 否 | - | 用于在 CSV 字段内转义引号或其他特殊字符的单个字符 | | metalake_type | String | 否 | gravitino | metalake 服务类型，目前支持 `gravitino` |
