This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 8d08d23276 [Docs][Connector-V2] Improve Elasticsearch HdfsFile and
Hive docs (#11300)
8d08d23276 is described below
commit 8d08d23276d4e2ca7b09f5ed6e0c14881fe2c630
Author: Daniel Carter <[email protected]>
AuthorDate: Fri Jul 3 22:58:23 2026 +0800
[Docs][Connector-V2] Improve Elasticsearch HdfsFile and Hive docs (#11300)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Elasticsearch.md | 8 ++++
docs/en/connectors/sink/HdfsFile.md | 7 +++-
docs/en/connectors/sink/Hive.md | 5 +++
docs/en/connectors/source/Elasticsearch.md | 6 ++-
docs/en/connectors/source/Hive.md | 21 ++++++++++-
docs/zh/connectors/sink/Elasticsearch.md | 14 +++++--
docs/zh/connectors/sink/HdfsFile.md | 9 ++++-
docs/zh/connectors/sink/Hive.md | 5 +++
docs/zh/connectors/source/Elasticsearch.md | 60 +++++++++++++++---------------
docs/zh/connectors/source/Hive.md | 21 ++++++++++-
10 files changed, 115 insertions(+), 41 deletions(-)
diff --git a/docs/en/connectors/sink/Elasticsearch.md
b/docs/en/connectors/sink/Elasticsearch.md
index 816573585f..5d72862b3d 100644
--- a/docs/en/connectors/sink/Elasticsearch.md
+++ b/docs/en/connectors/sink/Elasticsearch.md
@@ -10,6 +10,7 @@ Output data to `Elasticsearch`.
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
:::tip
@@ -47,6 +48,8 @@ Engine Supported
| common-options | | no | - |
| vectorization_fields | array | no | - |
| vector_dimensions | int | no | - |
+| multi_table_sink_replica | int | no | 1
|
+
### hosts [array]
`Elasticsearch` cluster http address, the format is `host:port` , allowing
multiple hosts to be specified. Such as `["host1:9200", "host2:9200"]`.
@@ -68,6 +71,10 @@ Primary key fields used to generate the document `_id`, this
is cdc required opt
Delimiter for composite keys ("_" by default), e.g., "$" would result in
document `_id` "KEY1$KEY2$KEY3".
+### multi_table_sink_replica [int]
+
+The replica number of sink writers used for each table in a multi-table sink
job. Keep the default value unless one table needs more sink writer parallelism.
+
## Authentication
The Elasticsearch connector supports multiple authentication methods to
connect to secured Elasticsearch clusters. You can choose the appropriate
authentication method based on your Elasticsearch security configuration.
@@ -232,6 +239,7 @@ sink {
hosts = ["localhost:9200"]
index = "${table_name}"
schema_save_mode="IGNORE"
+ multi_table_sink_replica = 1
}
}
```
diff --git a/docs/en/connectors/sink/HdfsFile.md
b/docs/en/connectors/sink/HdfsFile.md
index 208fb99f86..e40e3b7c33 100644
--- a/docs/en/connectors/sink/HdfsFile.md
+++ b/docs/en/connectors/sink/HdfsFile.md
@@ -91,6 +91,7 @@ Output data to hdfs file
| remote_user | string | no | -
| The remote user name of hdfs.
[...]
| schema_save_mode | string | no |
CREATE_SCHEMA_WHEN_NOT_EXIST | Existing dir processing method
[...]
| data_save_mode | string | no | APPEND_DATA
| Existing data processing method
[...]
+| multi_table_sink_replica | int | no | 1
| The replica number of sink writers used for each
table in a multi-table sink job.
[...]
| merge_update_event | boolean | no | false
| Only used when file_format_type is
canal_json,debezium_json or maxwell_json. When value is true, the UPDATE_AFTER
and UPDATE_BEFORE event will be merged into UPDATE event data
[...]
| schema_evolution_enabled | boolean | no | false
| Enable schema evolution support for CDC pipelines.
When true, ADD/DROP/RENAME/MODIFY column events from the source are applied to
the sink without a job restart. Not supported for binary format. |
@@ -113,6 +114,10 @@ Existing data processing method.
- APPEND_DATA: preserve dir, preserve data files
- ERROR_WHEN_DATA_EXISTS: when there is data files, an error is reported
+### multi_table_sink_replica [int]
+
+The replica number of sink writers used for each table in a multi-table sink
job. The default value is `1`; increase it only when each table needs more sink
writer parallelism.
+
### merge_update_event [boolean]
Only used when file_format_type is canal_json,debezium_json or maxwell_json.
@@ -324,4 +329,4 @@ LocalFile {
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/en/connectors/sink/Hive.md b/docs/en/connectors/sink/Hive.md
index d87b4100ff..5fd55a3a10 100644
--- a/docs/en/connectors/sink/Hive.md
+++ b/docs/en/connectors/sink/Hive.md
@@ -42,6 +42,7 @@ By default, we use 2PC commit to ensure `exactly-once`
| hive_site_path | string | no | - |
| hive.hadoop.conf | Map | no | - |
| hive.hadoop.conf-path | string | no | - |
+| remote_user | string | no | - |
| krb5_path | string | no | /etc/krb5.conf |
| kerberos_principal | string | no | - |
| kerberos_keytab_path | string | no | - |
@@ -77,6 +78,10 @@ Properties in hadoop conf('core-site.xml', 'hdfs-site.xml',
'hive-site.xml')
The specified loading path for the 'core-site.xml', 'hdfs-site.xml',
'hive-site.xml' files
+### remote_user [string]
+
+Hadoop remote user name used when connecting to HDFS/Hive storage without
Kerberos credentials.
+
### krb5_path [string]
The path of `krb5.conf`, used to authentication kerberos
diff --git a/docs/en/connectors/source/Elasticsearch.md
b/docs/en/connectors/source/Elasticsearch.md
index 2af91121de..003ccdbb94 100644
--- a/docs/en/connectors/source/Elasticsearch.md
+++ b/docs/en/connectors/source/Elasticsearch.md
@@ -30,8 +30,8 @@ support version >= 2.x and <= 8.x.
| auth.api_key_id | string | no | -
|
| auth.api_key | string | no | -
|
| auth.api_key_encoded | string | no | -
|
-| index | string | no | If the index list does not
exist, the index must be configured |
-| index_list | array | no | used to define a multiple
table task |
+| index | string | no | Required when `index_list` is
not configured |
+| index_list | array | no | Used to define a multi-index
task |
| source | array | no | -
|
| query | json | no | {"match_all": {}}
|
| search_type | enum | no | Query type, SQL or DSL,
default DSL |
@@ -145,6 +145,8 @@ source {
Elasticsearch index name, support * fuzzy matching.
+Either `index` or `index_list` must be configured. Use `index` for a single
index or an index pattern, and use `index_list` when different indexes need
their own `query`, `source`, `schema`, or paging options.
+
### source [array]
The fields of index.
diff --git a/docs/en/connectors/source/Hive.md
b/docs/en/connectors/source/Hive.md
index c92a6d4450..1e0eee81e8 100644
--- a/docs/en/connectors/source/Hive.md
+++ b/docs/en/connectors/source/Hive.md
@@ -50,9 +50,11 @@ Read all the data in a split in a pollNext call. What splits
are read will be sa
| name | type | required | default value |
|-----------------------|--------|----------|----------------|
-| table_name | string | yes | - |
+| table_name | string | no | Required for single-table mode |
+| table_list | array | no | - |
+| tables_configs | array | no | Deprecated, use `table_list`
instead |
| use_regex | boolean| no | false |
-| metastore_uri | string | yes | - |
+| metastore_uri | string | no | Required for single-table mode |
| krb5_path | string | no | /etc/krb5.conf |
| kerberos_principal | string | no | - |
| kerberos_keytab_path | string | no | - |
@@ -60,6 +62,7 @@ Read all the data in a split in a pollNext call. What splits
are read will be sa
| hive_site_path | string | no | - |
| hive.hadoop.conf | Map | no | - |
| hive.hadoop.conf-path | string | no | - |
+| remote_user | string | no | - |
| read_partitions | list | no | - |
| read_columns | list | no | - |
| compress_codec | string | no | none |
@@ -69,6 +72,16 @@ Read all the data in a split in a pollNext call. What splits
are read will be sa
Target Hive table name eg: `db1.table1`. When `use_regex = true`, this field
uses `databasePattern.tablePattern` (Hive has no schema) to match multiple
tables from Hive metastore.
+For a single-table source, configure `table_name` and `metastore_uri` at the
root level. For multi-table reading, configure `table_list`. `tables_configs`
is still accepted for compatibility, but `table_list` is preferred.
+
+### table_list [array]
+
+List of Hive table configurations for multi-table reading. Each item can
contain `table_name`, `metastore_uri`, `use_regex`, `read_partitions`,
`read_columns`, and the same authentication/Hadoop options as the root
connector block.
+
+### tables_configs [array]
+
+Deprecated multi-table configuration list. New jobs should use `table_list`.
+
### use_regex [boolean]
Whether to treat `table_name` as a regular expression pattern for matching
multiple tables (whole database / subset). This also works inside each entry of
`table_list` / `tables_configs`.
@@ -85,6 +98,10 @@ Regex syntax notes:
Hive metastore uri. Supports comma-separated multiple URIs for HA/failover
(whitespace is ignored). SeaTunnel passes this value to Hive
`hive.metastore.uris` and uses Hive `RetryingMetaStoreClient` (if available) to
retry/failover between URIs. This is client-side endpoint failover; make sure
your metastores share/replicate the same backend to keep metadata consistent.
+### remote_user [string]
+
+Hadoop remote user name used when connecting to HDFS/Hive storage without
Kerberos credentials.
+
### hdfs_site_path [string]
The path of `hdfs-site.xml`, used to load ha configuration of namenodes
diff --git a/docs/zh/connectors/sink/Elasticsearch.md
b/docs/zh/connectors/sink/Elasticsearch.md
index 7f21981559..ef2bf5001c 100644
--- a/docs/zh/connectors/sink/Elasticsearch.md
+++ b/docs/zh/connectors/sink/Elasticsearch.md
@@ -9,7 +9,8 @@ import ChangeLog from '../changelog/connector-elasticsearch.md';
## 主要特性
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
-- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
+- [x] [变更数据捕获](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
:::tip
@@ -18,7 +19,7 @@ import ChangeLog from
'../changelog/connector-elasticsearch.md';
* 支持 `ElasticSearch 版本 >= 2.x 并且 <= 8.x`
:::
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 选项
@@ -48,6 +49,8 @@ import ChangeLog from
'../changelog/connector-elasticsearch.md';
| common-options | | 否 | - |
| vectorization_fields | array | 否 | - |
| vector_dimensions | int | 否 | - |
+| multi_table_sink_replica | int | 否 | 1 |
+
### hosts [array]
@@ -69,6 +72,10 @@ import ChangeLog from
'../changelog/connector-elasticsearch.md';
设定复合键的分隔符（默认为 `_`），例如，如果使用 `$` 作为分隔符，那么文档的 `_id` 将呈现为 `KEY1$KEY2$KEY3` 的格式
+### multi_table_sink_replica [int]
+
+多表写入时，每张表对应的 Sink Writer 副本数。通常保持默认值即可；只有单表写入压力较大、需要更多写入并行度时再调大。
+
## 认证
Elasticsearch 连接器支持多种认证方式连接到安全的 Elasticsearch 集群。您可以根据 Elasticsearch
的安全配置选择合适的认证方式。
@@ -231,6 +238,7 @@ sink {
hosts = ["localhost:9200"]
index = "${table_name}"
schema_save_mode="IGNORE"
+ multi_table_sink_replica = 1
}
}
```
@@ -383,4 +391,4 @@ sink {
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/HdfsFile.md
b/docs/zh/connectors/sink/HdfsFile.md
index b187e0c3a8..477a51df81 100644
--- a/docs/zh/connectors/sink/HdfsFile.md
+++ b/docs/zh/connectors/sink/HdfsFile.md
@@ -33,7 +33,7 @@ import ChangeLog from '../changelog/connector-file-hadoop.md';
- [x] maxwell_json
- [x] 压缩编解码器
- [x] lzo
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -82,6 +82,7 @@ import ChangeLog from '../changelog/connector-file-hadoop.md';
| remote_user | string | 否 | -
| Hdfs的远端用户名。
|
| schema_save_mode | string | 否 |
CREATE_SCHEMA_WHEN_NOT_EXIST | 现有目录处理方式
|
| data_save_mode | string | 否 | APPEND_DATA
| 现有数据处理方式
|
+| multi_table_sink_replica | int | 否 | 1
| 多表写入时，每张表对应的 Sink Writer 副本数。
|
| merge_update_event | boolean | 否 | false
| 仅当file_format_type为canal_json、debezium_json、maxwell_json.
|
| schema_evolution_enabled | boolean | 否 | false
| 开启 Schema 演变支持，适用于 CDC 管道。为 true 时，来自上游的
ADD/DROP/RENAME/MODIFY 列事件无需重启作业即可应用到 Sink。不支持 binary 格式。 |
@@ -106,6 +107,10 @@ import ChangeLog from
'../changelog/connector-file-hadoop.md';
- APPEND_DATA：保留目录，保留数据文件
- ERROR_WHEN_DATA_EXISTS：当有数据文件时，会报告错误
+### multi_table_sink_replica [int]
+
+多表写入时，每张表对应的 Sink Writer 副本数。默认值为 `1`；只有单表写入压力较大、需要更多写入并行度时再调大。
+
### merge_update_event [boolean]
仅当file_format_type为canal_json、debezium_json、maxwell_json时使用.
@@ -320,4 +325,4 @@ LocalFile {
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/Hive.md b/docs/zh/connectors/sink/Hive.md
index 5869d55647..ecd5667edd 100644
--- a/docs/zh/connectors/sink/Hive.md
+++ b/docs/zh/connectors/sink/Hive.md
@@ -42,6 +42,7 @@ import ChangeLog from '../changelog/connector-hive.md';
| hive_site_path | string | 否 | - |
| hive.hadoop.conf | Map | 否 | - |
| hive.hadoop.conf-path | string | 否 | - |
+| remote_user | string | 否 | - |
| krb5_path | string | 否 | /etc/krb5.conf |
| kerberos_principal | string | 否 | - |
| kerberos_keytab_path | string | 否 | - |
@@ -77,6 +78,10 @@ Hadoop 配置中的属性（`core-site.xml`、`hdfs-site.xml`、`hive-site.xml`
指定加载 `core-site.xml`、`hdfs-site.xml`、`hive-site.xml` 文件的路径
+### remote_user [string]
+
+未使用 Kerberos 凭据连接 HDFS/Hive 存储时使用的 Hadoop 远端用户名。
+
### krb5_path [string]
`krb5.conf` 的路径，用于 Kerberos 认证
diff --git a/docs/zh/connectors/source/Elasticsearch.md
b/docs/zh/connectors/source/Elasticsearch.md
index 8c3b4a6044..843178229b 100644
--- a/docs/zh/connectors/source/Elasticsearch.md
+++ b/docs/zh/connectors/source/Elasticsearch.md
@@ -8,7 +8,7 @@ import ChangeLog from '../changelog/connector-elasticsearch.md';
支持读取 Elasticsearch2.x 版本和 8.x 版本之间的数据
-## Key features
+## 主要特性
- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
@@ -21,34 +21,34 @@ import ChangeLog from
'../changelog/connector-elasticsearch.md';
| 参数名称 | 类型 | 是否必须 | 默认值或者描述 |
| ----------------------- | ------- | --------
|-------------------------------------|
-| hosts | 数组 | yes | -
|
-| auth_type | string | no | basic
|
-| username | string | no | -
|
-| password | string | no | -
|
-| auth.api_key_id | string | no | -
|
-| auth.api_key | string | no | -
|
-| auth.api_key_encoded | string | no | -
|
-| index | string | No |
单索引同步配置，如果index_list没有配置，则必须配置index |
-| index_list | array | no | 用来定义多索引同步任务
|
-| source | array | no | -
|
-| query | json | no | {"match_all": {}}
|
-| search_type | enum | no | 查询类型，SQL 或 DSL，默认 DSL
|
-| search_api_type | enum | no | 分页 API 类型，SCROLL 或 PIT，默认
SCROLL |
-| sql_query | json | no | SQL 查询语句，当 search_type 为 SQL
时必须 |
-| scroll_time | string | no | 1m
|
-| scroll_size | int | no | 100
|
-| tls_verify_certificate | boolean | no | true
|
-| tls_verify_hostname | boolean | no | true
|
-| array_column | map | no |
|
-| tls_keystore_path | string | no | -
|
-| tls_keystore_password | string | no | -
|
-| tls_truststore_path | string | no | -
|
-| tls_truststore_password | string | no | -
|
-| pit_keep_alive | long | no | 60000 (1 minute)
|
-| pit_batch_size | int | no | 100
|
-| slice_max | int | no | 1（SCROLL 需 ES >= 5.0，PIT 需 ES
>= 7.10） |
-| runtime_fields | array | no | -
|
-| common-options | | no | -
|
+| hosts | 数组 | 是 | -
|
+| auth_type | string | 否 | basic
|
+| username | string | 否 | -
|
+| password | string | 否 | -
|
+| auth.api_key_id | string | 否 | -
|
+| auth.api_key | string | 否 | -
|
+| auth.api_key_encoded | string | 否 | -
|
+| index | string | 否 | 未配置 `index_list` 时必须配置
|
+| index_list | array | 否 | 用来定义多索引同步任务
|
+| source | array | 否 | -
|
+| query | json | 否 | {"match_all": {}}
|
+| search_type | enum | 否 | 查询类型，SQL 或 DSL，默认 DSL
|
+| search_api_type | enum | 否 | 分页 API 类型，SCROLL 或 PIT，默认
SCROLL |
+| sql_query | json | 否 | SQL 查询语句，当 search_type 为 SQL
时必须 |
+| scroll_time | string | 否 | 1m
|
+| scroll_size | int | 否 | 100
|
+| tls_verify_certificate | boolean | 否 | true
|
+| tls_verify_hostname | boolean | 否 | true
|
+| array_column | map | 否 |
|
+| tls_keystore_path | string | 否 | -
|
+| tls_keystore_password | string | 否 | -
|
+| tls_truststore_path | string | 否 | -
|
+| tls_truststore_password | string | 否 | -
|
+| pit_keep_alive | long | 否 | 60000 (1 minute)
|
+| pit_batch_size | int | 否 | 100
|
+| slice_max | int | 否 | 1（SCROLL 需 ES >= 5.0，PIT 需 ES
>= 7.10） |
+| runtime_fields | array | 否 | -
|
+| common-options | | 否 | -
|
### hosts [array]
@@ -135,6 +135,8 @@ source {
Elasticsearch 索引名称，支持 * 模糊匹配。比如存在索引index1,index2,可以指定index*同时读取两个索引的数据。
+`index` 和 `index_list` 至少需要配置一个。单索引或索引通配场景使用 `index`；不同索引需要单独配置
`query`、`source`、`schema` 或分页参数时，使用 `index_list`。
+
### source [array]
索引的字段
diff --git a/docs/zh/connectors/source/Hive.md
b/docs/zh/connectors/source/Hive.md
index 1b6eb3894a..04b01e427b 100644
--- a/docs/zh/connectors/source/Hive.md
+++ b/docs/zh/connectors/source/Hive.md
@@ -50,9 +50,11 @@ import ChangeLog from '../changelog/connector-hive.md';
| 名称 | 类型 | 必需 | 默认值 |
|-----------------------|--------|------|---------|
-| table_name | string | 是 | - |
+| table_name | string | 否 | 单表模式必填 |
+| table_list | array | 否 | - |
+| tables_configs | array | 否 | 已废弃，请使用 `table_list` |
| use_regex | boolean| 否 | false |
-| metastore_uri | string | 是 | - |
+| metastore_uri | string | 否 | 单表模式必填 |
| krb5_path | string | 否 | /etc/krb5.conf |
| kerberos_principal | string | 否 | - |
| kerberos_keytab_path | string | 否 | - |
@@ -60,6 +62,7 @@ import ChangeLog from '../changelog/connector-hive.md';
| hive_site_path | string | 否 | - |
| hive.hadoop.conf | Map | 否 | - |
| hive.hadoop.conf-path | string | 否 | - |
+| remote_user | string | 否 | - |
| read_partitions | list | 否 | - |
| read_columns | list | 否 | - |
| compress_codec | string | 否 | none |
@@ -69,6 +72,16 @@ import ChangeLog from '../changelog/connector-hive.md';
目标 Hive 表名，例如：`db1.table1`。当 `use_regex = true` 时，该字段支持 `数据库正则.表正则`（Hive 没有
schema）来匹配 Hive 元存储中的多张表。
+单表读取时，在根配置中填写 `table_name` 和 `metastore_uri`。多表读取时，建议使用
`table_list`。`tables_configs` 仍兼容旧配置，但新作业建议使用 `table_list`。
+
+### table_list [array]
+
+Hive 多表读取配置列表。每个元素可以包含
`table_name`、`metastore_uri`、`use_regex`、`read_partitions`、`read_columns`，以及与根配置相同的认证和
Hadoop 配置。
+
+### tables_configs [array]
+
+已废弃的多表配置列表。新作业请使用 `table_list`。
+
### use_regex [boolean]
是否将 `table_name` 视为正则表达式进行匹配。开启后，`table_name` 可用于整库/多表同步；同样也支持在 `table_list` /
`tables_configs` 的每个表配置里单独开启。
@@ -85,6 +98,10 @@ import ChangeLog from '../changelog/connector-hive.md';
Hive 元存储 URI。支持通过逗号分隔配置多个 URI 用于高可用/故障切换（会自动去除空格）。SeaTunnel 会将该值写入 Hive 的
`hive.metastore.uris`，并在运行时优先使用 Hive 的 `RetryingMetaStoreClient`
实现重试/切换。注意：该能力仅做客户端连接端点切换，元数据一致性需要由 metastore 部署保证。
+### remote_user [string]
+
+未使用 Kerberos 凭据连接 HDFS/Hive 存储时使用的 Hadoop 远端用户名。
+
### hdfs_site_path [string]
`hdfs-site.xml` 的路径，用于加载 Namenode 的高可用配置
