This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 573e4ea620 [Docs][Connector-V2] Improve Iceberg Kafka and Socket docs
(#11335)
573e4ea620 is described below
commit 573e4ea62060205c21d2302dd774741ca2e3bdb6
Author: Daniel Carter <[email protected]>
AuthorDate: Sat Jul 4 21:31:02 2026 +0800
[Docs][Connector-V2] Improve Iceberg Kafka and Socket docs (#11335)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Iceberg.md | 33 ++++++++++++++++++++++++++++++++-
docs/en/connectors/sink/Socket.md | 8 +++++++-
docs/en/connectors/source/Iceberg.md | 13 +++++++------
docs/en/connectors/source/Kafka.md | 4 ++--
docs/en/connectors/source/Socket.md | 25 +++++++++----------------
docs/zh/connectors/sink/Iceberg.md | 34 ++++++++++++++++++++++++++++++++--
docs/zh/connectors/sink/Socket.md | 26 ++++++++++++++++----------
docs/zh/connectors/source/Iceberg.md | 10 +++++-----
docs/zh/connectors/source/Kafka.md | 4 ++--
docs/zh/connectors/source/Socket.md | 24 ++++++++----------------
10 files changed, 120 insertions(+), 61 deletions(-)
diff --git a/docs/en/connectors/sink/Iceberg.md
b/docs/en/connectors/sink/Iceberg.md
index 527dd1202d..759d3f247c 100644
--- a/docs/en/connectors/sink/Iceberg.md
+++ b/docs/en/connectors/sink/Iceberg.md
@@ -16,10 +16,11 @@ import ChangeLog from '../changelog/connector-iceberg.md';
## Description
-Sink connector for Apache Iceberg. It can support cdc mode 、auto create table
and table schema evolution.
+Sink connector for Apache Iceberg. It supports CDC writes, automatic table
creation, table schema evolution, and multi-table write jobs.
## Key features
+- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
## Supported DataSource Info
@@ -80,12 +81,25 @@ libfb303-xxx.jar
| data_save_mode | Enum | no | APPEND_DATA
| the data save mode, please refer to `data_save_mode` below
|
| custom_sql | string | no | -
| Custom `delete` data sql for data save mode. e.g: `delete from
... where ...`
|
| iceberg.table.commit-branch | string | no | -
| Default branch for commits
|
+| multi_table_sink_replica | int | no | -
| Replica number for every table writer in multi-table sink mode.
Use this when one upstream job writes to multiple Iceberg tables and each table
needs more than one sink writer.
|
| krb5_path | string | no | /etc/krb5.conf
| The path of `krb5.conf`, used for Kerberos authentication.
|
| kerberos_principal | string | no | -
| The principal for Kerberos authentication.
|
| kerberos_keytab_path | string | no | -
| The keytab file path for Kerberos authentication.
|
## Sink Option descriptions
+### iceberg.table.upsert-mode-enabled [boolean]
+
+When this option is `true`, `iceberg.table.primary-keys` must be configured
explicitly. The sink does not inherit primary keys from the source table
automatically.
+
+### iceberg.table.partition-keys [string]
+
+Use a comma-separated list, for example `dt,region` or Iceberg transforms such
as `days(ts)`. In multi-table jobs, `${partition_keys}` can be used as a
placeholder from the upstream table metadata.
+
+### iceberg.table.commit-branch [string]
+
+Write commits to the specified Iceberg branch. Leave it empty to commit to the
table's default branch.
+
### krb5_path [string]
The path of `krb5.conf`, used for Kerberos authentication.
@@ -196,6 +210,23 @@ sink {
}
```
+### Commit To Iceberg Branch
+
+```hocon
+sink {
+ Iceberg {
+ catalog_name = "seatunnel_test"
+ iceberg.catalog.config = {
+ type = "hadoop"
+ warehouse = "file:///tmp/seatunnel/iceberg/hadoop-sink/"
+ }
+ namespace = "seatunnel_namespace"
+ table = "iceberg_sink_table"
+ iceberg.table.commit-branch = "audit_branch"
+ }
+}
+```
+
### Glue Catalog
```hocon
diff --git a/docs/en/connectors/sink/Socket.md
b/docs/en/connectors/sink/Socket.md
index 53ac809291..e2c5b105dc 100644
--- a/docs/en/connectors/sink/Socket.md
+++ b/docs/en/connectors/sink/Socket.md
@@ -16,7 +16,7 @@ import ChangeLog from '../changelog/connector-socket.md';
## Description
-Used to send data to Socket Server. Both support streaming and batch mode.
+Used to send data to a socket server in streaming or batch mode. Each
SeaTunnel row is serialized as one JSON line.
> For example, if the data from upstream is [`age: 12, name: jared`], the
> content send to socket server is the following: `{"name":"jared","age":17}`
@@ -29,6 +29,12 @@ Used to send data to Socket Server. Both support streaming
and batch mode.
| max_retries | Integer | No | 3 | The number of retries to
send record failed
|
| common-options | | No | - | Sink plugin common
parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details |
+:::tip
+
+Socket sink is mainly used for local debugging and simple integrations. It
reconnects and retries failed writes according to `max_retries`, but it does
not provide exactly-once delivery.
+
+:::
+
## Task Example
> This is randomly generated data written to the Socket side
diff --git a/docs/en/connectors/source/Iceberg.md
b/docs/en/connectors/source/Iceberg.md
index 5511ff6979..8765e4cf2c 100644
--- a/docs/en/connectors/source/Iceberg.md
+++ b/docs/en/connectors/source/Iceberg.md
@@ -77,8 +77,8 @@ libfb303-xxx.jar
|--------------------------|---------|----------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| catalog_name | string | yes | - |
User-specified catalog name.
[...]
| namespace | string | yes | - | The
iceberg database name in the backend catalog.
[...]
-| table | string | no | - | The
iceberg table name in the backend catalog.
[...]
-| table_list | string | no | - | The
iceberg table list in the backend catalog.
[...]
+| table | string | no | - | The
Iceberg table name in the backend catalog. Configure exactly one of `table` and
`table_list`.
[...]
+| table_list | list | no | - | The
Iceberg table list in the backend catalog. Configure exactly one of `table` and
`table_list`. Each item can set `table`, `query`, and snapshot/stream scan
options for that table.
[...]
| iceberg.catalog.config | map | yes | - |
Specify the properties for initializing the Iceberg catalog, which can be
referenced in this file:
[CatalogProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java)
[...]
| hadoop.config | map | no | - |
Properties passed through to the Hadoop configuration
[...]
| iceberg.hadoop-conf-path | string | no | - | The
specified loading paths for the 'core-site.xml', 'hdfs-site.xml',
'hive-site.xml' files.
[...]
@@ -92,7 +92,7 @@ libfb303-xxx.jar
| stream_scan_strategy | enum | no | FROM_LATEST_SNAPSHOT |
Starting strategy for stream mode execution, Default to use
`FROM_LATEST_SNAPSHOT` if don't specify any value,The optional values
are:<br/>TABLE_SCAN_THEN_INCREMENTAL: Do a regular table scan then switch to
the incremental mode.<br/>FROM_LATEST_SNAPSHOT: Start incremental mode from the
latest snapshot inclusive.<br/>FROM_EARLIEST_SNAPSHOT: Start incremental mode
from the earliest snapshot inclusive.<br/>FROM_SNAPSHO [...]
| increment.scan-interval | long | no | 2000 | The
interval of increment scan(mills)
[...]
| common-options | | no | - |
Source plugin common parameters, please refer to [Source Common
Options](../common-options/source-common-options.md) for details.
[...]
-| query | String | no | - | The
select DML to select the iceberg data. It mustn't contain the table name, and
doesn’t support alias. For example: `select * from table where f1 > 100`,
`select fn from table where f1 > 100`. The current support for the LIKE syntax
is limited: the LIKE clause shouldn't start with `%`. The supported one is:
`select f1 from t where f2 like 'tom%' `
[...]
+| query | String | no | - | The
select DML used to filter or project Iceberg data. Use the literal placeholder
table name `table`, and do not use table aliases. For example: `select * from
table where f1 > 100`, `select fn from table where f1 > 100`. The current LIKE
support is limited: the LIKE pattern should not start with `%`; for example,
`select f1 from table where f2 like 'tom%'` is supported.
[...]
| krb5_path | string | no | /etc/krb5.conf
| The path to the `krb5.conf` file for Kerberos authentication.
|
| kerberos_principal | string | no | -
| The principal for Kerberos authentication.
|
| kerberos_keytab_path | string | no | -
| The path to the keytab file for Kerberos authentication.
|
@@ -134,6 +134,8 @@ sink {
### Multi-Table Read
+Use `table_list` when one Iceberg source needs to read more than one table. Do
not set `table` at the same time.
+
```hocon
source {
Iceberg {
@@ -145,10 +147,10 @@ source {
namespace = "database1"
table_list = [
{
- table = "table_1
+ table = "table_1"
},
{
- table = "table_2
+ table = "table_2"
query = "select fn from table where f1 > 100"
}
]
@@ -193,7 +195,6 @@ source {
uri = "thrift://localhost:9083"
warehouse = "hdfs://your_cluster//tmp/seatunnel/iceberg/"
}
- catalog_type = "hive"
namespace = "your_iceberg_database"
table = "your_iceberg_table"
diff --git a/docs/en/connectors/source/Kafka.md
b/docs/en/connectors/source/Kafka.md
index 95fb278fdb..28e23bfef9 100644
--- a/docs/en/connectors/source/Kafka.md
+++ b/docs/en/connectors/source/Kafka.md
@@ -38,7 +38,7 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
|-------------------------------------|----------------------------------------------------------------------------|----------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| topic | String
| No | - |
Topic name(s) to read data from when the table is used as source. Required
unless `tables_configs` or `table_list` is used. It also supports a
comma-separated topic list like `topic-1,topic-2`.
[...]
| tables_configs | List
| No | - |
Preferred multi-topic table configuration. Only one of `topic`,
`tables_configs`, and `table_list` can be configured.
[...]
-| table_list | Map
| No | - |
Deprecated compatibility option for multi-topic table configuration. Only one
of `topic`, `tables_configs`, and `table_list` can be configured.
[...]
+| table_list | List
| No | - |
Deprecated compatibility option for multi-topic table configuration. Only one
of `topic`, `tables_configs`, and `table_list` can be configured.
[...]
| bootstrap.servers | String
| Yes | - |
Comma separated list of Kafka brokers.
[...]
| pattern | Boolean
| No | false | If
`pattern` is set to `true`,the regular expression for a pattern of topic names
to read from. All topics in clients with names that match the specified regular
expression will be subscribed by the consumer.
[...]
| consumer.group | String
| No | SeaTunnel-Consumer-Group |
`Kafka consumer group id`, used to distinguish different consumer groups.
[...]
@@ -49,6 +49,7 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| format | String
| No | json |
Data format. The default format is json. Optional text format, canal_json,
debezium_json, maxwell_json, ogg_json, avro , protobuf and native. If you use
json or text format. The default field separator is ", ". If you customize the
delimiter, add the "field_delimiter" option.If you use canal format, please
refer to [canal-json](../form [...]
| avro_schema | String
| No | - |
Effective when `format` is `avro`. Provides the writer Avro schema used to
deserialize binary Avro messages whose record name, namespace, or union layout
differs from the SeaTunnel schema.
[...]
| format_error_handle_way | String
| No | fail | The
processing method of data format error. The default value is fail, and the
optional value is (fail, skip). When fail is selected, data format error will
block and an exception will be thrown. When skip is selected, data format error
will skip this line data.
[...]
+| debezium_record_include_schema | Boolean
| No | true |
Effective when `format` is `debezium_json`. Set whether Debezium records
include schema information.
[...]
| debezium_record_table_filter | Config
| No | - |
Used for filtering data in debezium format, only when the format is set to
`debezium_json`. Please refer `debezium_record_table_filter` below
[...]
| field_delimiter | String
| No | , |
Customize the field delimiter for data format.
[...]
| start_mode |
StartMode[earliest],[group_offsets],[latest],[specific_offsets],[timestamp] |
No | group_offsets | The initial consumption pattern of
consumers.
[...]
@@ -62,7 +63,6 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| protobuf_schema | String
| No | - |
Effective when the format is set to protobuf, specifies the Schema definition
[...]
| strip_schema_registry_header | Boolean
| No | false |
Effective when the format is set to protobuf. Whether to strip the Confluent
Schema Registry wire format header (magic byte, schema id and message indexes)
before protobuf deserialization. This option is useful when consuming Protobuf
messages that were encoded using Confluent Schema Registry. When enabled, the
connector will try to de [...]
| reader_cache_queue_size | Integer
| No | 2 |
The capacity of the fetcher-to-reader element queue. Each element is one
`consumer.poll()` batch, not a single message. See
[reader_cache_queue_size](#reader_cache_queue_size) for details. |
-| is_native | Boolean
| No | false |
Supports retaining the source information of the record.
> On restore from checkpoint or savepoint, Kafka Source resumes from the
> checkpointed split offsets.
> `start_mode` and consumer-group offsets are only used for the first startup
> or for newly
diff --git a/docs/en/connectors/source/Socket.md
b/docs/en/connectors/source/Socket.md
index b370958b03..2493801027 100644
--- a/docs/en/connectors/source/Socket.md
+++ b/docs/en/connectors/source/Socket.md
@@ -21,28 +21,15 @@ import ChangeLog from '../changelog/connector-socket.md';
## Description
-Used to read data from Socket.
+Used to read newline-delimited text data from a socket server. Each line
received from the socket becomes one SeaTunnel row.
## Data Type Mapping
-The File does not have a specific type list, and we can indicate which
SeaTunnel data type the corresponding data needs to be converted to by
specifying the Schema in the config.
+Socket source reads each incoming line as a string record.
| SeaTunnel Data type |
|---------------------|
| STRING |
-| SHORT |
-| INT |
-| BIGINT |
-| BOOLEAN |
-| DOUBLE |
-| DECIMAL |
-| FLOAT |
-| DATE |
-| TIME |
-| TIMESTAMP |
-| BYTES |
-| ARRAY |
-| MAP |
## Options
@@ -52,6 +39,12 @@ The File does not have a specific type list, and we can
indicate which SeaTunnel
| port | Integer | Yes | _ | socket server port
|
| common-options | | no | - | Source plugin common
parameters, please refer to [Source Common
Options](../common-options/source-common-options.md) for details. |
+:::tip
+
+Socket source is mainly used for local debugging and simple text streams. It
does not checkpoint socket-server offsets, so it should not be used when
replayable, exactly-once reads are required.
+
+:::
+
## How to Create a Socket Data Synchronization Jobs
* Configuring the SeaTunnel config file
@@ -110,4 +103,4 @@ spark
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/Iceberg.md
b/docs/zh/connectors/sink/Iceberg.md
index 0f75ea64df..585bba1f7b 100644
--- a/docs/zh/connectors/sink/Iceberg.md
+++ b/docs/zh/connectors/sink/Iceberg.md
@@ -16,12 +16,12 @@ import ChangeLog from '../changelog/connector-iceberg.md';
## 描述
-Apache Iceberg 目标连接器支持cdc模式、自动建表及表结构变更.
+Apache Iceberg 目标连接器支持 CDC 写入、自动建表、表结构变更以及多表写入。
## 主要特性
+- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
## 支持的数据源信息
@@ -81,12 +81,25 @@ libfb303-xxx.jar
| data_save_mode | Enum | no | APPEND_DATA
| 数据写入方式, 请参考下面的 `data_save_mode`
|
| custom_sql | string | no | -
| 自定义 `delete` 数据的 SQL 语句，用于数据写入方式。例如： `delete from ... where ...`
|
| iceberg.table.commit-branch | string | no | -
| 提交的默认分支
|
+| multi_table_sink_replica | int | no | -
| 多表写入模式下每张表对应的 Sink 写入并发数。一个作业写入多张 Iceberg 表，并且每张表都需要多个写入器时使用。
|
| krb5_path | string | no | /etc/krb5.conf
| `krb5.conf` 文件的路径，用于 Kerberos 认证。
|
| kerberos_principal | string | no | -
| Kerberos 认证的 principal。
|
| kerberos_keytab_path | string | no | -
| Kerberos 认证的 keytab 文件路径。
|
## Sink 选项说明
+### iceberg.table.upsert-mode-enabled [boolean]
+
+当该选项为 `true` 时，必须显式配置 `iceberg.table.primary-keys`。Sink 不会再自动继承 Source 表的主键。
+
+### iceberg.table.partition-keys [string]
+
+使用英文逗号分隔多个分区字段，例如 `dt,region`，也可以使用 Iceberg transform，例如 `days(ts)`。多表写入场景可以使用
`${partition_keys}` 占位符，从上游表元信息中取分区字段。
+
+### iceberg.table.commit-branch [string]
+
+将提交写入指定的 Iceberg 分支。不配置时写入表的默认分支。
+
### krb5_path [string]
`krb5.conf` 文件的路径，用于 Kerberos 认证。
@@ -198,6 +211,23 @@ sink {
```
+### 提交到 Iceberg 分支
+
+```hocon
+sink {
+ Iceberg {
+ catalog_name = "seatunnel_test"
+ iceberg.catalog.config = {
+ type = "hadoop"
+ warehouse = "file:///tmp/seatunnel/iceberg/hadoop-sink/"
+ }
+ namespace = "seatunnel_namespace"
+ table = "iceberg_sink_table"
+ iceberg.table.commit-branch = "audit_branch"
+ }
+}
+```
+
### AWS S3 Tables REST Catalog
Amazon S3 表类数据存储服务提供针对分析工作负载进行优化的 S3 存储，其功能旨在持续提高查询性能并降低表的存储成本。S3
表类数据存储服务专为存储表数据而设计，例如每日购买交易、流传感器数据或广告展示次数。表数据以列和行表示数据，就像在数据库表中一样。
diff --git a/docs/zh/connectors/sink/Socket.md
b/docs/zh/connectors/sink/Socket.md
index 097ce669f2..133ad93060 100644
--- a/docs/zh/connectors/sink/Socket.md
+++ b/docs/zh/connectors/sink/Socket.md
@@ -12,27 +12,33 @@ import ChangeLog from '../changelog/connector-socket.md';
## 主要特性
-- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
-用于向Socket Server发送数据。两者都支持流媒体和批处理模式。
+用于向 Socket Server 发送数据，支持流模式和批模式。每条 SeaTunnel 数据会被序列化为一行 JSON。
->
例如，如果来自上游的数据是[`age:12，name:jared`]，则发送到Socket服务器的内容如下：`{"name"："jared"，"age"：17}`
+> 例如，如果来自上游的数据是 [`age: 17, name: jared`]，则发送到 Socket Server
的内容如下：`{"name":"jared","age":17}`
## Sink 选项
| 名称 | 类型 | 是否必传 | 默认值 |
描述 |
|----------------|---------|----------|---------|-----------------------------------------------------------------------------------------------------------------|
-| host | String | 是 | | socket 服务器主机
|
-| port | Integer | 是 | | socket 服务器端口
|
-| max_retries | Integer | 否 | 3 | 发送记录的重试失败次数
|
-| common-options | | 否 | - | Sink插件常用参数，详见[Sink common
Options](../common-options/sink-common-options.md) |
+| host | String | 是 | - | socket 服务器主机
|
+| port | Integer | 是 | - | socket 服务器端口
|
+| max_retries | Integer | 否 | 3 | 发送失败后的最大重试次数
|
+| common-options | | 否 | - | Sink 插件通用参数，详见 [Sink
通用选项](../common-options/sink-common-options.md) |
+
+:::tip
+
+Socket Sink 更适合本地调试和简单集成。它会根据 `max_retries` 进行重连和重试，但不提供精确一次写入保证。
+
+:::
## 任务示例
-> 这是写入Socket端的随机生成数据
+> 以下示例把 FakeSource 随机生成的数据写入 Socket Server。
```hocon
env {
@@ -66,7 +72,7 @@ sink {
nc -l -v 9999
```
-* 启动SeaTunnel任务
+* 启动 SeaTunnel 任务
* Socket 服务器控制台打印数据
diff --git a/docs/zh/connectors/source/Iceberg.md
b/docs/zh/connectors/source/Iceberg.md
index 722023ef34..41a9617419 100644
--- a/docs/zh/connectors/source/Iceberg.md
+++ b/docs/zh/connectors/source/Iceberg.md
@@ -77,8 +77,8 @@ libfb303-xxx.jar
|--------------------------|---------|------|----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
| catalog_name | string | 是 | - | 用户指定的目录名称。
[...]
| namespace | string | 是 | - | 后端目录中的
iceberg 数据库名称。
[...]
-| table | string | 否 | - | 后端目录中的
iceberg 表名称。
[...]
-| table_list | string | 否 | - | 后端目录中的
iceberg 表列表。
[...]
+| table | string | 否 | - | 后端目录中的
Iceberg 表名称。`table` 和 `table_list` 必须二选一配置，不能同时配置。
[...]
+| table_list | list | 否 | - | 后端目录中的
Iceberg 表列表。`table` 和 `table_list` 必须二选一配置，不能同时配置。每一项都可以配置 `table`、`query`
以及该表自己的快照/流式扫描参数。
[...]
| iceberg.catalog.config | map | 是 | - | 指定初始化
Iceberg
目录的属性，可以在此文件中引用：[CatalogProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java)
[...]
| hadoop.config | map | 否 | - | 传递给 Hadoop
配置的属性
[...]
| iceberg.hadoop-conf-path | string | 否 | - | 为
'core-site.xml'、'hdfs-site.xml'、'hive-site.xml' 文件指定的加载路径。
[...]
@@ -92,7 +92,7 @@ libfb303-xxx.jar
| stream_scan_strategy | enum | 否 | FROM_LATEST_SNAPSHOT |
流模式执行的启动策略，如果不指定任何值，默认使用
`FROM_LATEST_SNAPSHOT`，可选值为：<br/>TABLE_SCAN_THEN_INCREMENTAL：执行常规表扫描，然后切换到增量模式。<br/>FROM_LATEST_SNAPSHOT：从最新快照（包含）开始增量模式。<br/>FROM_EARLIEST_SNAPSHOT：从最早快照（包含）开始增量模式。<br/>FROM_SNAPSHOT_ID：从具有特定
id（包含）的快照开始增量模式。<br/>FROM_SNAPSHOT_TIMESTAMP：从具有特定时间戳（包含）的快照开始增量模式。 |
| increment.scan-interval | long | 否 | 2000 |
增量扫描的间隔（毫秒）
[...]
| common-options | | 否 | - |
源插件通用参数，请参考 [源通用选项](../common-options/source-common-options.md) 详见。
[...]
-| query | String | 否 | - | 用于选择
iceberg 数据的 select DML。它不能包含表名，也不支持别名。例如：`select * from table where f1 >
100`、`select fn from table where f1 > 100`。当前对 LIKE 语法的支持是有限的：LIKE 子句不应以 `%`
开头。支持的是：`select f1 from t where f2 like 'tom%' `
[...]
+| query | String | 否 | - | 用于过滤或投影
Iceberg 数据的 select DML。这里使用固定占位表名 `table`，不要写真实表名，也不支持表别名。例如：`select * from
table where f1 > 100`、`select fn from table where f1 > 100`。当前对 LIKE
语法的支持有限：LIKE 模式不应以 `%` 开头，例如 `select f1 from table where f2 like 'tom%'`。
[...]
| krb5_path | string | no | /etc/krb5.conf
| `krb5.conf` 文件的路径，用于 Kerberos 认证。
|
| kerberos_principal | string | no | -
| Kerberos 认证的 principal。
|
| kerberos_keytab_path | string | no | -
| Kerberos 认证的 keytab 文件路径。
|
@@ -147,6 +147,8 @@ sink {
### 多表读取
+一个 Iceberg Source 需要读取多张表时使用 `table_list`，不要同时配置 `table`。
+
```hocon
source {
Iceberg {
@@ -206,7 +208,6 @@ source {
uri = "thrift://localhost:9083"
warehouse = "hdfs://your_cluster//tmp/seatunnel/iceberg/"
}
- catalog_type = "hive"
namespace = "your_iceberg_database"
table = "your_iceberg_table"
@@ -271,4 +272,3 @@ source {
<ChangeLog />
-
diff --git a/docs/zh/connectors/source/Kafka.md
b/docs/zh/connectors/source/Kafka.md
index bc35e3ab9f..89921afaa2 100644
--- a/docs/zh/connectors/source/Kafka.md
+++ b/docs/zh/connectors/source/Kafka.md
@@ -38,7 +38,7 @@ import ChangeLog from '../changelog/connector-kafka.md';
|-------------------------------------|-------------------------------------|------|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String |
否 | - | 使用表作为数据源时要读取数据的主题名称。未使用 `tables_configs`
或 `table_list` 时需要配置。也支持通过逗号分隔多个主题，例如 `topic-1,topic-2`。
|
| tables_configs | List |
否 | - | 推荐使用的多主题表配置。当不同 topic 需要不同 schema 或
format 时使用。`topic`、`tables_configs`、`table_list` 三者只能配置一个。
|
-| table_list | Map |
否 | - |
旧版兼容的多主题表配置。`topic`、`tables_configs`、`table_list` 三者只能配置一个。
|
+| table_list | List |
否 | - |
旧版兼容的多主题表配置。`topic`、`tables_configs`、`table_list` 三者只能配置一个。
|
| bootstrap.servers | String |
是 | - | 逗号分隔的 Kafka brokers 列表。
|
| pattern | Boolean |
否 | false | 如果 `pattern` 设置为
`true`，则会使用指定的正则表达式匹配并订阅主题。
|
| consumer.group | String |
否 | SeaTunnel-Consumer-Group | `Kafka 消费者组 ID`，用于区分不同的消费者组。
|
@@ -49,6 +49,7 @@ import ChangeLog from '../changelog/connector-kafka.md';
| format | String |
否 | json | 数据格式。默认格式为 json。可选格式包括 text, canal_json,
debezium_json, ogg_json, maxwell_json, avro , protobuf和native。默认字段分隔符为 ",
"。如果自定义分隔符，添加 "field_delimiter" 选项。如果使用 canal 格式，请参考
[canal-json](../formats/canal-json.md) 了解详细信息。如果使用 debezium 格式，请参考
[debezium-json](../formats/debezium-json.md)。一些Format的详细信息请参考
[formats](../formats) |
| avro_schema | String |
否 | - | 当 `format` 为 `avro` 时生效。用于提供二进制 Avro 消息的
writer schema，适用于消息的 record 名称、namespace 或 union 结构与 SeaTunnel schema 不完全一致的场景。
|
| format_error_handle_way | String |
否 | fail | 数据格式错误的处理方式。默认值为 fail，可选值为 fail 和
skip。当选择 fail 时，数据格式错误将阻塞并抛出异常。当选择 skip 时，数据格式错误将跳过此行数据。
|
+| debezium_record_include_schema | Boolean |
否 | true | 当 `format` 为 `debezium_json` 时生效，用于说明
Debezium 记录中是否携带 schema 信息。
|
| debezium_record_table_filter | Config |
否 | - | 用于过滤 debezium 格式的数据，仅当格式设置为
`debezium_json` 时使用。请参阅下面的 `debezium_record_table_filter`
|
| field_delimiter | String |
否 | , | 自定义数据格式的字段分隔符。
|
| start_mode |
StartMode[earliest],[group_offsets],[latest],[specific_offsets],[timestamp] | 否
| group_offsets | 消费者的初始消费模式。
|
@@ -62,7 +63,6 @@ import ChangeLog from '../changelog/connector-kafka.md';
| protobuf_schema | String |
否 | - | 当格式设置为 protobuf 时有效，指定 Schema 定义。
|
| strip_schema_registry_header | Boolean |
否 | false | 当格式设置为 protobuf 时有效。是否在 Protobuf 反序列化之前去除
Confluent Schema Registry 线格式头部（magic byte、schema id 和 message indexes）。当消费使用
Confluent Schema Registry 编码的 Protobuf 消息时，此选项非常有用。启用后，连接器将尝试在解析 Protobuf
消息之前检测并删除 Schema Registry 头部。如果未检测到头部，它将回退到标准的 Protobuf 反序列化。
[...]
| reader_cache_queue_size | Integer |
否 | 2 | Fetcher 与 Reader 线程之间缓冲队列的容量。每个元素是一次
`consumer.poll()` 的整批结果，而非单条消息。详见
[reader_cache_queue_size](#reader_cache_queue_size)。 |
-| is_native | Boolean |
否 | false | 支持保留record的源信息。
|
> 从 checkpoint 或 savepoint 恢复时，Kafka Source 会优先使用 checkpoint 中保存的 split offset。
> `start_mode` 和 consumer group offset 只在首次启动，或为尚未存在 checkpoint
> 状态的新发现分区初始化位点时生效。
diff --git a/docs/zh/connectors/source/Socket.md
b/docs/zh/connectors/source/Socket.md
index 2fe6d612c5..7d1d3d719f 100644
--- a/docs/zh/connectors/source/Socket.md
+++ b/docs/zh/connectors/source/Socket.md
@@ -21,28 +21,15 @@ import ChangeLog from '../changelog/connector-socket.md';
## 描述
-用于从 Socket 读取数据。
+用于从 Socket 服务端读取按行分隔的文本数据。Socket 中收到的每一行都会成为一条 SeaTunnel 数据。
## 数据类型映射
-文件没有特定的类型列表，我们可以通过在配置中指定 Schema 来指示相应的数据需要转换为哪种 SeaTunnel 数据类型。
+Socket Source 会把每一行输入读取为字符串。
| SeaTunnel 数据类型 |
|------------------|
| STRING |
-| SHORT |
-| INT |
-| BIGINT |
-| BOOLEAN |
-| DOUBLE |
-| DECIMAL |
-| FLOAT |
-| DATE |
-| TIME |
-| TIMESTAMP |
-| BYTES |
-| ARRAY |
-| MAP |
## 选项
@@ -52,6 +39,12 @@ import ChangeLog from '../changelog/connector-socket.md';
| port | Integer | 是 | - | socket 服务器端口 |
| common-options | | 否 | - | 源插件通用参数，请参考
[源通用选项](../common-options/source-common-options.md) 详见。 |
+:::tip
+
+Socket Source 更适合本地调试和简单文本流读取。它不会保存 Socket 服务端的读取位点，如果需要可重放或精确一次读取，请使用 Kafka
等具备位点管理能力的 Source。
+
+:::
+
## 如何创建 Socket 数据同步作业
* 配置 SeaTunnel 配置文件
@@ -111,4 +104,3 @@ spark
## 变更日志
<ChangeLog />
-
