This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 5db6a76358 [Docs][Connector-V2] Improve HBase Web3j and MaxCompute
docs (#11293)
5db6a76358 is described below
commit 5db6a763587965f3e0f84aa7e02950621d4fc854
Author: Daniel Carter <[email protected]>
AuthorDate: Fri Jul 3 18:11:29 2026 +0800
[Docs][Connector-V2] Improve HBase Web3j and MaxCompute docs (#11293)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Hbase.md | 38 +++++++++++++-
docs/en/connectors/sink/Maxcompute.md | 88 ++++++++++++++++++++++++++-------
docs/en/connectors/source/Hbase.md | 59 +++++++++++++++-------
docs/en/connectors/source/Maxcompute.md | 7 ++-
docs/en/connectors/source/Web3j.md | 23 ++++++---
docs/zh/connectors/sink/Hbase.md | 81 ++++++++++++++++++++----------
docs/zh/connectors/sink/Maxcompute.md | 86 +++++++++++++++++++++++++-------
docs/zh/connectors/source/Hbase.md | 79 ++++++++++++++++++-----------
docs/zh/connectors/source/Maxcompute.md | 24 ++++++---
docs/zh/connectors/source/Web3j.md | 29 ++++++-----
10 files changed, 377 insertions(+), 137 deletions(-)
diff --git a/docs/en/connectors/sink/Hbase.md b/docs/en/connectors/sink/Hbase.md
index 104fde4955..115c3f7c9e 100644
--- a/docs/en/connectors/sink/Hbase.md
+++ b/docs/en/connectors/sink/Hbase.md
@@ -6,11 +6,14 @@ import ChangeLog from '../changelog/connector-hbase.md';
## Description
-Output data to Hbase
+Writes SeaTunnel rows to Apache HBase. The sink can create or reuse target
tables, write one or
+multiple upstream tables, map fields to one column family or multiple column
families, and control
+how null values, row keys, WAL, timestamps, and existing data are handled.
## Key features
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
## Options
@@ -23,9 +26,11 @@ Output data to Hbase
| rowkey_delimiter | string | no | "" |
| version_column | string | no | - |
| null_mode | string | no | skip |
-| wal_write | boolean | yes | false |
+| wal_write | boolean | no | false |
| write_buffer_size | string | no | 8 * 1024 * 1024 |
| encoding | string | no | utf8 |
+| schema_save_mode | enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST |
+| data_save_mode | enum | no | APPEND_DATA |
| hbase_extra_config | config | no | - |
| common-options | | no | - |
| ttl | long | no | - |
@@ -106,6 +111,16 @@ Hbase stores bytes. The connector supports:
The extra configuration of hbase
+### schema_save_mode [enum]
+
+Controls how the sink handles the target HBase table before writing. Supported
values include
+`RECREATE_SCHEMA`, `CREATE_SCHEMA_WHEN_NOT_EXIST`, and
`ERROR_WHEN_SCHEMA_NOT_EXIST`.
+
+### data_save_mode [enum]
+
+Controls how the sink handles existing target data before writing. Supported
values are
+`DROP_DATA`, `APPEND_DATA`, and `ERROR_WHEN_DATA_EXISTS`.
+
### ttl [long]
Hbase writes data TTL time, the default is based on the TTL set in the table,
unit: milliseconds
@@ -116,6 +131,8 @@ Sink plugin common parameters, please refer to [Sink Common
Options](../common-o
## Example
+### Write One Table
+
```hocon
Hbase {
@@ -129,6 +146,23 @@ Hbase {
```
+### Create Table When It Does Not Exist
+
+```hocon
+sink {
+ Hbase {
+ zookeeper_quorum = "hbase_e2e:2181"
+ table = "seatunnel_test_with_create_when_not_exists"
+ rowkey_column = ["name"]
+ family_name {
+ all_columns = info
+ }
+ schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
+ data_save_mode = "APPEND_DATA"
+ }
+}
+```
+
## Kerberos Example
Note:
diff --git a/docs/en/connectors/sink/Maxcompute.md
b/docs/en/connectors/sink/Maxcompute.md
index 4d6ee7fb1b..a7849157df 100644
--- a/docs/en/connectors/sink/Maxcompute.md
+++ b/docs/en/connectors/sink/Maxcompute.md
@@ -6,7 +6,7 @@ import ChangeLog from '../changelog/connector-maxcompute.md';
## Description
-Used to read data from Maxcompute.
+Used to write data to Maxcompute.
## Key features
@@ -14,27 +14,33 @@ Used to read data from Maxcompute.
## Options
-| name | type | required | default value |
-|----------------|---------|----------|---------------|
-| accessId | string | no | - |
-| accesskey | string | no | - |
-| sts_token | string | no | - |
-| endpoint | string | yes | - |
-| project | string | yes | - |
-| table_name | string | yes | - |
-| schema_name | string | no | - |
-| partition_spec | string | no | - |
-| overwrite | boolean | no | false |
-| insert_strategy| string | no | upload |
-| common-options | string | no | |
+| name | type | required | default value |
+|---------------------------|---------|----------|---------------|
+| accessId | string | no | - |
+| accesskey | string | no | - |
+| sts_token | string | no | - |
+| endpoint | string | yes | - |
+| project | string | yes | - |
+| table_name | string | yes | - |
+| schema_name | string | no | - |
+| partition_spec | string | no | - |
+| overwrite | boolean | no | false |
+| schema_save_mode | enum | no |
CREATE_SCHEMA_WHEN_NOT_EXIST |
+| data_save_mode | enum | no | APPEND_DATA |
+| custom_sql | string | no | - |
+| save_mode_create_template | string | no | see below |
+| datetime_format | string | no | yyyy-MM-dd HH:mm:ss |
+| tunnel_endpoint | string | no | - |
+| insert_strategy | string | no | upload |
+| common-options | string | no | |
### accessId [string]
-`accessId` Your Maxcompute accessId which cloud be access from Alibaba Cloud.
+`accessId` Your Maxcompute accessId that can access Alibaba Cloud.
### accesskey [string]
-`accesskey` Your Maxcompute accessKey which cloud be access from Alibaba Cloud.
+`accesskey` Your Maxcompute accessKey that can access Alibaba Cloud.
### sts_token [string]
@@ -184,6 +190,8 @@ Sink plugin common parameters, please refer to [Sink Common
Options](../common-o
## Examples
+### Append Data
+
```hocon
sink {
Maxcompute {
@@ -198,6 +206,52 @@ sink {
}
```
+### Upsert or Delete Rows
+
+Use `insert_strategy = "upsert"` when the upstream schema has a primary key
and the job contains
+update or delete rows. The example below uses an update row; delete rows use
the same sink settings.
+
+```hocon
+source {
+ FakeSource {
+ tables_configs = [
+ {
+ schema = {
+ table = "test_table_sink"
+ fields {
+ ID = int
+ NAME = string
+ AGE = int
+ }
+ primaryKey {
+ name = "ID"
+ columnNames = [ID]
+ }
+ }
+ rows = [
+ {
+ kind = UPDATE_AFTER
+ fields = [1, "UPSERT_TEST", 100]
+ }
+ ]
+ }
+ ]
+ }
+}
+
+sink {
+ Maxcompute {
+ accessId = "ak"
+ accesskey = "sk"
+ endpoint = "http://maxcompute:8080";
+ tunnel_endpoint = "http://maxcompute:8080";
+ project = "mocked_mc"
+ table_name = "test_table_sink"
+ insert_strategy = "upsert"
+ }
+}
+```
+
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/en/connectors/source/Hbase.md
b/docs/en/connectors/source/Hbase.md
index 8c837b670e..fe66c36bd8 100644
--- a/docs/en/connectors/source/Hbase.md
+++ b/docs/en/connectors/source/Hbase.md
@@ -6,7 +6,8 @@ import ChangeLog from '../changelog/connector-hbase.md';
## Description
-Reads data from Apache Hbase.
+Reads data from Apache HBase tables. The source supports normal scans, row key
range scans, timestamp
+range scans, binary row keys, custom namespaces, and parallel batch reading.
## Key Features
@@ -19,23 +20,23 @@ Reads data from Apache Hbase.
## Options
-| Name | Type | Required | Default |
-|----------------------|-----------|-----------|---------|
-| zookeeper_quorum | string | Yes | - |
-| table | string | Yes | - |
-| schema | config | Yes | - |
-| hbase_extra_config | config | No | - |
-| caching | int | No | -1 |
-| batch | int | No | -1 |
-| cache_blocks | boolean | No | false |
-| is_binary_rowkey | boolean | No | false |
-| start_rowkey | string | No | - |
-| end_rowkey | string | No | - |
+| Name | Type | Required | Default |
+|---------------------|---------|----------|---------|
+| zookeeper_quorum | string | Yes | - |
+| table | string | Yes | - |
+| schema | config | Yes | - |
+| hbase_extra_config | config | No | - |
+| caching | int | No | -1 |
+| batch | int | No | -1 |
+| cache_blocks | boolean | No | false |
+| is_binary_rowkey | boolean | No | false |
+| start_rowkey | string | No | - |
+| end_rowkey | string | No | - |
| start_row_inclusive | boolean | No | true |
| end_row_inclusive | boolean | No | false |
-| start_timestamp | long | No | - |
-| end_timestamp | long | No | - |
-| common-options | | No | - |
+| start_timestamp | long | No | - |
+| end_timestamp | long | No | - |
+| common-options | | No | - |
### zookeeper_quorum [string]
@@ -43,12 +44,14 @@ The zookeeper quorum for Hbase cluster hosts, e.g.,
"hadoop001:2181,hadoop002:21
### table [string]
-The name of the table to write to, e.g., "seatunnel".
+The name of the table to read from, e.g., "seatunnel".
If your table lives in a custom namespace, use the `namespace:table` form (for
example, `ns1:seatunnel_test`); when the namespace is omitted SeaTunnel will
read from HBase's default namespace (`default`).
### schema [config]
-Hbase stores data in byte arrays. Therefore, you need to configure the data
types for each column in the table. For more information, see:
[guide](../../introduction/concepts/schema-feature.md#how-to-declare-type-supported).
+Hbase stores data in byte arrays. Therefore, you need to configure the data
types for each column in the table.
+Use `rowkey` for the row key column and `family:qualifier` for HBase cells,
such as `info:name`.
+For more information, see:
[guide](../../introduction/concepts/schema-feature.md#how-to-declare-type-supported).
### hbase_extra_config [config]
@@ -114,6 +117,8 @@ Common parameters for Source plugins, refer to [Common
Source Options](../common
## Example
+### Read by Row Key and Time Range
+
```bash
source {
Hbase {
@@ -151,6 +156,23 @@ source {
}
```
+### Read a Namespace Table
+
+```hocon
+source {
+ Hbase {
+ zookeeper_quorum = "hbase_e2e:2181"
+ table = "ns1:seatunnel_test"
+ schema = {
+ columns = [
+ { name = rowkey, type = string },
+ { name = "info:name", type = string }
+ ]
+ }
+ }
+}
+```
+
## Kerberos Example
Note:
@@ -192,4 +214,3 @@ source {
## Changelog
<ChangeLog />
-
diff --git a/docs/en/connectors/source/Maxcompute.md
b/docs/en/connectors/source/Maxcompute.md
index 9395a857a7..754faebec4 100644
--- a/docs/en/connectors/source/Maxcompute.md
+++ b/docs/en/connectors/source/Maxcompute.md
@@ -31,16 +31,17 @@ Used to read data from Maxcompute.
| split_row | int | no | 10000 |
| read_columns | Array | no | - |
| table_list | Array | No | - |
+| tunnel_endpoint | string | no | - |
| common-options | string | no | |
| schema | config | no | |
### accessId [string]
-`accessId` Your Maxcompute accessId which cloud be access from Alibaba Cloud.
+`accessId` Your Maxcompute accessId that can access Alibaba Cloud.
### accesskey [string]
-`accesskey` Your Maxcompute accessKey which cloud be access from Alibaba Cloud.
+`accesskey` Your Maxcompute accessKey that can access Alibaba Cloud.
### sts_token [string]
@@ -121,6 +122,7 @@ source {
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>"
table_name="<your table name>"
+ #tunnel_endpoint="<your tunnel endpoint>"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
@@ -137,6 +139,7 @@ source {
accesskey="<your access Key>"
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>" # default project
+ #tunnel_endpoint="<your tunnel endpoint>"
table_list = [
{
table_name = "test_table"
diff --git a/docs/en/connectors/source/Web3j.md
b/docs/en/connectors/source/Web3j.md
index 6badcf8b96..cafe557f1d 100644
--- a/docs/en/connectors/source/Web3j.md
+++ b/docs/en/connectors/source/Web3j.md
@@ -21,15 +21,23 @@ import ChangeLog from '../changelog/connector-web3j.md';
## Description
-Source connector for web3j. It is used to read data from the blockchain, such
as block information, transactions, smart contract events, etc. Currently, it
supports reading block height data.
+Source connector for Web3j. It reads blockchain data through a Web3 provider
endpoint. Currently,
+the connector reads the latest block number and emits one `value` field. The
`value` field is a JSON
+string that contains `blockNumber` and the read timestamp.
## Source Options
-| Name | Type | Required | Default |
Description |
-|------|--------|----------|---------|---------------------------------------------------------------------------------------------------------|
-| url | String | Yes | - | When using Infura as the service
provider, the URL is used for communication with the Ethereum network. |
+| Name | Type | Required | Default | Description |
+|------|--------|----------|---------|-------------|
+| url | String | Yes | - | Web3 provider endpoint used to
communicate with the Ethereum network, for example an Infura URL. |
-## How to Create a Http Data Synchronization Jobs
+## Output Schema
+
+| Field | Type | Description |
+|-------|--------|-------------|
+| value | String | JSON string containing the latest block number and a
timestamp generated by the connector. |
+
+## Example
```hocon
env {
@@ -40,12 +48,13 @@ env {
source {
Web3j {
url = "https://mainnet.infura.io/v3/xxxxx";
+ plugin_output = "web3j"
}
}
-# Console printing of the read Http data
sink {
Console {
+ plugin_input = "web3j"
parallelism = 1
}
}
@@ -54,7 +63,7 @@ sink {
Then you will get the following data:
```json
-{"blockNumber":19525949,"timestamp":"2024-03-27T13:28:45.605Z"}
+{"value":"{\"blockNumber\":19525949,\"timestamp\":\"2024-03-27T13:28:45.605Z\"}"}
```
## Changelog
diff --git a/docs/zh/connectors/sink/Hbase.md b/docs/zh/connectors/sink/Hbase.md
index 15a9e9b25e..cfb753cc0b 100644
--- a/docs/zh/connectors/sink/Hbase.md
+++ b/docs/zh/connectors/sink/Hbase.md
@@ -2,38 +2,41 @@ import ChangeLog from '../changelog/connector-hbase.md';
# Hbase
-> Hbase 数据连接器
+> Hbase 接收器连接器
## 描述
-将数据输出到hbase
+将 SeaTunnel 数据写入 Apache
HBase。支持创建或复用目标表、写入单表或多表、把字段映射到一个或多个列簇，并可控制空值、行键、WAL、时间戳和已有数据的处理方式。
## 主要特性
-- [ ] [精准一次](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 选项
-| 名称 | 类型 | 是否必须 | 默认值 |
-|--------------------|---------|------|-----------------|
-| zookeeper_quorum | string | yes | - |
-| table | string | yes | - |
-| rowkey_column | list | yes | - |
-| family_name | config | yes | - |
-| rowkey_delimiter | string | no | "" |
-| version_column | string | no | - |
-| null_mode | string | no | skip |
-| wal_write | boolean | yes | false |
-| write_buffer_size | string | no | 8 * 1024 * 1024 |
-| encoding | string | no | utf8 |
-| hbase_extra_config | config | no | - |
-| common-options | | no | - |
-| ttl | long | no | - |
+| 名称 | 类型 | 是否必须 | 默认值 |
+|--------------------|---------|----------|--------|
+| zookeeper_quorum | string | 是 | - |
+| table | string | 是 | - |
+| rowkey_column | list | 是 | - |
+| family_name | config | 是 | - |
+| rowkey_delimiter | string | 否 | "" |
+| version_column | string | 否 | - |
+| null_mode | string | 否 | skip |
+| wal_write | boolean | 否 | false |
+| write_buffer_size | string | 否 | 8 * 1024 * 1024 |
+| encoding | string | 否 | utf8 |
+| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST |
+| data_save_mode | enum | 否 | APPEND_DATA |
+| hbase_extra_config | config | 否 | - |
+| common-options | | 否 | - |
+| ttl | long | 否 | - |
### zookeeper_quorum [string]
-hbase的zookeeper集群主机, 示例: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
+HBase 的 zookeeper 集群主机，示例: "hadoop001:2181,hadoop002:2181,hadoop003:2181"
### table [string]
@@ -52,14 +55,14 @@ hbase的zookeeper集群主机, 示例:
"hadoop001:2181,hadoop002:2181,hadoop003:
|----|---------------|-----|
| 1 | tyrantlucifer | 27 |
-id作为行键和其他写入不同列簇的字段，可以分配
+`id` 作为行键，其他字段可以写入不同列簇。例如:
family_name {
name = "info1"
age = "info2"
}
-这主要是name写入列簇info1,age写入将写给列簇 info2
+这表示 `name` 写入列簇 `info1`，`age` 写入列簇 `info2`。
如果要将其他字段写入同一列簇，可以分配
@@ -67,7 +70,7 @@ family_name {
all_columns = "info"
}
-这意味着所有字段都将写入该列簇 info
+这表示所有字段都会写入列簇 `info`。
### rowkey_delimiter [string]
@@ -82,7 +85,7 @@ all_columns = "info"
写入 null 值的模式，支持 [ skip , empty], 默认 skip
- skip: 当字段为 null ,连接器不会将此字段写入 hbase
-- empty: 当字段为null时,连接器将写入并为此字段生成空值
+- empty: 当字段为 null 时，连接器会为此字段写入空值
### wal_write [boolean]
@@ -105,7 +108,16 @@ Hbase 存储字节，连接器支持：
### hbase_extra_config [config]
-hbase扩展配置
+hbase 扩展配置
+
+### schema_save_mode [enum]
+
+写入前如何处理目标 HBase 表结构。常用值包括 `RECREATE_SCHEMA`、`CREATE_SCHEMA_WHEN_NOT_EXIST` 和
+`ERROR_WHEN_SCHEMA_NOT_EXIST`。
+
+### data_save_mode [enum]
+
+写入前如何处理目标端已有数据。支持 `DROP_DATA`、`APPEND_DATA` 和 `ERROR_WHEN_DATA_EXISTS`。
### ttl [long]
@@ -117,6 +129,8 @@ Sink 插件常用参数，详见 Sink 常用选项 [Sink Common Options](../comm
## 案例
+### 写入单表
+
```hocon
Hbase {
@@ -130,6 +144,23 @@ Hbase {
```
+### 目标表不存在时自动创建
+
+```hocon
+sink {
+ Hbase {
+ zookeeper_quorum = "hbase_e2e:2181"
+ table = "seatunnel_test_with_create_when_not_exists"
+ rowkey_column = ["name"]
+ family_name {
+ all_columns = info
+ }
+ schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
+ data_save_mode = "APPEND_DATA"
+ }
+}
+```
+
## Kerberos 示例
备注：
diff --git a/docs/zh/connectors/sink/Maxcompute.md
b/docs/zh/connectors/sink/Maxcompute.md
index 87f3984d9c..2d6a888a97 100644
--- a/docs/zh/connectors/sink/Maxcompute.md
+++ b/docs/zh/connectors/sink/Maxcompute.md
@@ -2,31 +2,37 @@ import ChangeLog from '../changelog/connector-maxcompute.md';
# Maxcompute
-> Maxcompute Sink 连接器
+> Maxcompute 接收器连接器
## 描述
-用于从 Maxcompute 读取数据。
+用于向 Maxcompute 写入数据。
-## 关键特性
+## 主要特性
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
## 选项
-| 参数名 | 类型 | 必须 | 默认值 |
-|----------------|---------|------|--------|
-| accessId | string | 否 | - |
-| accesskey | string | 否 | - |
-| sts_token | string | 否 | - |
-| endpoint | string | 是 | - |
-| project | string | 是 | - |
-| table_name | string | 是 | - |
-| schema_name | string | 否 | - |
-| partition_spec | string | 否 | - |
-| overwrite | boolean | 否 | false |
-| insert_strategy| string | no | upload |
-| common-options | string | 否 | |
+| 参数名 | 类型 | 必须 | 默认值 |
+|---------------------------|---------|------|--------|
+| accessId | string | 否 | - |
+| accesskey | string | 否 | - |
+| sts_token | string | 否 | - |
+| endpoint | string | 是 | - |
+| project | string | 是 | - |
+| table_name | string | 是 | - |
+| schema_name | string | 否 | - |
+| partition_spec | string | 否 | - |
+| overwrite | boolean | 否 | false |
+| schema_save_mode | enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST |
+| data_save_mode | enum | 否 | APPEND_DATA |
+| custom_sql | string | 否 | - |
+| save_mode_create_template | string | 否 | 见下文 |
+| datetime_format | string | 否 | yyyy-MM-dd HH:mm:ss |
+| tunnel_endpoint | string | 否 | - |
+| insert_strategy | string | 否 | upload |
+| common-options | string | 否 | |
### accessId [string]
@@ -182,6 +188,8 @@ Sink 插件通用参数，请参考 [Sink 通用选项](../common-options/sink-c
## 示例
+### 追加写入
+
```hocon
sink {
Maxcompute {
@@ -196,8 +204,52 @@ sink {
}
```
+### 更新插入或删除数据
+
+当上游表结构有主键，并且任务里包含更新或删除数据时，建议配置 `insert_strategy = "upsert"`。
+
+```hocon
+source {
+ FakeSource {
+ tables_configs = [
+ {
+ schema = {
+ table = "test_table_sink"
+ fields {
+ ID = int
+ NAME = string
+ AGE = int
+ }
+ primaryKey {
+ name = "ID"
+ columnNames = [ID]
+ }
+ }
+ rows = [
+ {
+ kind = UPDATE_AFTER
+ fields = [1, "UPSERT_TEST", 100]
+ }
+ ]
+ }
+ ]
+ }
+}
+
+sink {
+ Maxcompute {
+ accessId = "ak"
+ accesskey = "sk"
+ endpoint = "http://maxcompute:8080";
+ tunnel_endpoint = "http://maxcompute:8080";
+ project = "mocked_mc"
+ table_name = "test_table_sink"
+ insert_strategy = "upsert"
+ }
+}
+```
+
## 变更日志
<ChangeLog />
-
diff --git a/docs/zh/connectors/source/Hbase.md
b/docs/zh/connectors/source/Hbase.md
index b115e42dc8..ccb0bb7458 100644
--- a/docs/zh/connectors/source/Hbase.md
+++ b/docs/zh/connectors/source/Hbase.md
@@ -6,53 +6,55 @@ import ChangeLog from '../changelog/connector-hbase.md';
## 描述
-从 Apache Hbase 读取数据。
+从 Apache HBase 表读取数据。支持普通扫描、行键范围扫描、时间戳范围扫描、二进制行键、自定义命名空间和并行批读取。
-## 主要功能
+## 主要特性
- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
-- [x] [Schema](../../introduction/concepts/connector-v2-features.md)
+- [x] [模式投影](../../introduction/concepts/connector-v2-features.md)
- [x] [并行度](../../introduction/concepts/connector-v2-features.md)
- [ ] [支持用户定义的拆分](../../introduction/concepts/connector-v2-features.md)
## 选项
-| 名称 | 类型 | 必填 | 默认值 |
-|----------------------|----------|----|-------|
-| zookeeper_quorum | string | 是 | - |
-| table | string | 是 | - |
-| schema | config | 是 | - |
-| hbase_extra_config | config | 否 | - |
-| caching | int | 否 | -1 |
-| batch | int | 否 | -1 |
-| cache_blocks | boolean | 否 | false |
-| is_binary_rowkey | boolean | 否 | false |
-| start_rowkey | string | 否 | - |
-| end_rowkey | string | 否 | - |
-| start_row_inclusive | boolean | 否 | true |
-| end_row_inclusive | boolean | 否 | false |
-| start_timestamp | long | 否 | - |
-| end_timestamp | long | 否 | - |
-| common-options | | 否 | - |
+| 名称 | 类型 | 必填 | 默认值 |
+|----------------------|---------|------|--------|
+| zookeeper_quorum | string | 是 | - |
+| table | string | 是 | - |
+| schema | config | 是 | - |
+| hbase_extra_config | config | 否 | - |
+| caching | int | 否 | -1 |
+| batch | int | 否 | -1 |
+| cache_blocks | boolean | 否 | false |
+| is_binary_rowkey | boolean | 否 | false |
+| start_rowkey | string | 否 | - |
+| end_rowkey | string | 否 | - |
+| start_row_inclusive | boolean | 否 | true |
+| end_row_inclusive | boolean | 否 | false |
+| start_timestamp | long | 否 | - |
+| end_timestamp | long | 否 | - |
+| common-options | | 否 | - |
### zookeeper_quorum [string]
-hbase的zookeeper集群主机，例如：“hadoop001:2181,hadoop002:2181,hadoop003:2181”
+HBase 的 zookeeper 集群主机，例如：“hadoop001:2181,hadoop002:2181,hadoop003:2181”
### table [string]
-要写入的表名，例如：“seatunnel”
+要读取的表名，例如：“seatunnel”
如果表在自定义 namespace 下，请使用 `namespace:table` 形式（如 `ns1:seatunnel_test`）；未填写
namespace 时，SeaTunnel 会使用 HBase 的默认命名空间 `default`。
### schema [config]
-Hbase
使用字节数组进行存储。因此，您需要为表中的每一列配置数据类型。有关更多信息，请参阅：[guide](../../introduction/concepts/schema-feature.md#how-to-declare-type-supported)。
+HBase 使用字节数组进行存储。因此，您需要为表中的每一列配置数据类型。
+行键列使用 `rowkey`，HBase 单元格使用 `列簇:列名` 形式，例如 `info:name`。
+更多信息请参阅：[模式声明指南](../../introduction/concepts/schema-feature.md#how-to-declare-type-supported)。
### hbase_extra_config [config]
-hbase 的额外配置
+HBase 的额外配置
### caching
@@ -64,7 +66,7 @@ batch 参数用于设置在扫描过程中每次返回的最大列数。这对
### cache_blocks
-cache_blocks 参数用于设置在扫描过程中是否缓存数据块。默认情况下，HBase 会在扫描时将数据块缓存到块缓存中。如果设置为
false，则在扫描过程中不会缓存数据块，从而减少内存的使用。在SeaTunnel中默认值为: false
+cache_blocks 参数用于设置在扫描过程中是否缓存数据块。默认情况下，HBase 会在扫描时将数据块缓存到块缓存中。如果设置为
false，则在扫描过程中不会缓存数据块，从而减少内存使用。在 SeaTunnel 中默认值为 false。
### is_binary_rowkey
@@ -90,10 +92,10 @@ HBase 的行键既可以是文本字符串，也可以是二进制数据。在 S
**注意:** 在大多数情况下,应保持默认值 (false),这遵循 HBase 标准的左闭右开区间约定。仅当您需要在扫描结果中包含结束行时才修改此参数。
-**重要提示:** 在使用多个 split 并行读取时,这两个参数的组合对数据完整性至关重要:
-- **默认配置 (start_row_inclusive=true, end_row_inclusive=false)**: 这是推荐的配置,可以确保跨
split 时不会丢失数据或产生重复数据。每个 split 遵循 [start, end) 左闭右开区间约定。
-- **都设置为 false (start_row_inclusive=false, end_row_inclusive=false)**:
这可能会导致**数据丢失**,因为边界行会被所有 split 排除在外。
-- **都设置为 true (start_row_inclusive=true, end_row_inclusive=true)**:
这可能会导致**数据重复**,因为边界行会被相邻的多个 split 重复包含。
+**重要提示:** 使用多个分片并行读取时，这两个参数的组合对数据完整性很重要:
+- **默认配置 (start_row_inclusive=true, end_row_inclusive=false)**:
推荐配置，可以确保跨分片时不丢失数据、不产生重复数据。每个分片遵循 [start, end) 左闭右开区间约定。
+- **都设置为 false (start_row_inclusive=false, end_row_inclusive=false)**:
可能导致**数据丢失**，因为边界行会被所有分片排除在外。
+- **都设置为 true (start_row_inclusive=true, end_row_inclusive=true)**:
可能导致**数据重复**，因为边界行会被相邻的多个分片重复包含。
### start_timestamp
@@ -114,6 +116,8 @@ Source 插件常用参数，具体请参考 [Source 常用选项](../common-opti
## 示例
+### 按行键和时间范围读取
+
```bash
source {
Hbase {
@@ -151,6 +155,23 @@ source {
}
```
+### 读取命名空间下的表
+
+```hocon
+source {
+ Hbase {
+ zookeeper_quorum = "hbase_e2e:2181"
+ table = "ns1:seatunnel_test"
+ schema = {
+ columns = [
+ { name = rowkey, type = string },
+ { name = "info:name", type = string }
+ ]
+ }
+ }
+}
+```
+
## Kerberos 示例
备注：
diff --git a/docs/zh/connectors/source/Maxcompute.md
b/docs/zh/connectors/source/Maxcompute.md
index 78b7f86591..68e98fcb9d 100644
--- a/docs/zh/connectors/source/Maxcompute.md
+++ b/docs/zh/connectors/source/Maxcompute.md
@@ -6,9 +6,9 @@ import ChangeLog from '../changelog/connector-maxcompute.md';
## 描述
-用于从 Maxcompute 读取数据.
+用于从 Maxcompute 读取数据。
-## 关键特性
+## 主要特性
- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
@@ -31,6 +31,7 @@ import ChangeLog from '../changelog/connector-maxcompute.md';
| split_row | int | 否 | 10000 |
| read_columns | Array | 否 | - |
| table_list | Array | 否 | - |
+| tunnel_endpoint | string | 否 | - |
| common-options | string | 否 | |
| schema | config | 否 | |
@@ -51,19 +52,19 @@ import ChangeLog from
'../changelog/connector-maxcompute.md';
### endpoint [string]
-`endpoint` 您的 Maxcompute 端点以 http 开头.
+`endpoint` 您的 Maxcompute 端点，以 http 开头。
### project [string]
-`project` 您在阿里云中创建的Maxcompute项目.
+`project` 您在阿里云中创建的 Maxcompute 项目。
### table_name [string]
-`table_name` 目标Maxcompute表名，例如：fake.
+`table_name` 目标 Maxcompute 表名，例如：fake。
### partition_spec [string]
-`partition_spec` Maxcompute分区表的此规范，例如:ds='20220101'.
+`partition_spec` Maxcompute 分区表的规范，例如: ds='20220101'。
### schema_name [string]
@@ -85,7 +86,12 @@ import ChangeLog from '../changelog/connector-maxcompute.md';
### table_list [Array]
-要读取的表列表，您可以使用此配置代替 `table_name`.
+要读取的表列表，您可以使用此配置代替 `table_name`。
+
+### tunnel_endpoint [String]
+
+MaxCompute Tunnel 服务的自定义端点。未配置时，连接器会根据区域自动推断默认 Tunnel 端点。
+一般只有自定义网络、调试或本地开发时才需要配置，例如 `http://maxcompute:8080`。
### common options
@@ -103,6 +109,7 @@ source {
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>"
table_name="<your table name>"
+ #tunnel_endpoint="<your tunnel endpoint>"
#partition_spec="<your partition spec>"
#split_row = 10000
#read_columns = ["col1", "col2"]
@@ -119,6 +126,7 @@ source {
accesskey="<your access Key>"
endpoint="<http://service.odps.aliyun.com/api>"
project="<your project>" # default project
+ #tunnel_endpoint="<your tunnel endpoint>"
table_list = [
{
table_name = "test_table"
@@ -140,4 +148,4 @@ source {
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/source/Web3j.md
b/docs/zh/connectors/source/Web3j.md
index 99a509028b..b8aa7d77ad 100644
--- a/docs/zh/connectors/source/Web3j.md
+++ b/docs/zh/connectors/source/Web3j.md
@@ -10,26 +10,33 @@ import ChangeLog from '../changelog/connector-web3j.md';
> Flink<br/>
> Seatunnel Zeta<br/>
-## 关键特性
+## 主要特性
-- [x] [批](../../introduction/concepts/connector-v2-features.md)
-- [x] [流](../../introduction/concepts/connector-v2-features.md)
+- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
+- [x] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [ ] [列投影](../../introduction/concepts/connector-v2-features.md)
-- [ ] [并行性](../../introduction/concepts/connector-v2-features.md)
-- [ ] [支持用户自定义split](../../introduction/concepts/connector-v2-features.md)
+- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
+- [ ] [支持用户自定义分片](../../introduction/concepts/connector-v2-features.md)
## 描述
-Web3j 的源连接器。用于从区块链读取数据，例如区块信息、交易、智能合约事件等。目前支持读取区块高度数据。
+Web3j 源连接器用于通过 Web3 服务端点读取区块链数据。目前连接器读取最新区块号，并输出一个
+`value` 字段。`value` 字段是 JSON 字符串，里面包含 `blockNumber` 和连接器生成的读取时间戳。
## 源选项
| 参数名 | 类型 | 必须 | 默认值 | 描述 |
|--------|------|------|--------|------|
-| url | String | 是 | - | 使用 Infura 作为服务提供商时，URL 用于与以太坊网络通信。 |
+| url | String | 是 | - | 用于和以太坊网络通信的 Web3 服务端点，例如 Infura URL。 |
-## 如何创建 Http 数据同步作业
+## 输出字段
+
+| 字段 | 类型 | 描述 |
+|------|------|------|
+| value | String | JSON 字符串，包含最新区块号和连接器生成的时间戳。 |
+
+## 示例
```hocon
env {
@@ -40,12 +47,13 @@ env {
source {
Web3j {
url = "https://mainnet.infura.io/v3/xxxxx";
+ plugin_output = "web3j"
}
}
-# 控制台打印读取的 Http 数据
sink {
Console {
+ plugin_input = "web3j"
parallelism = 1
}
}
@@ -54,10 +62,9 @@ sink {
然后您将获得以下数据：
```json
-{"blockNumber":19525949,"timestamp":"2024-03-27T13:28:45.605Z"}
+{"value":"{\"blockNumber\":19525949,\"timestamp\":\"2024-03-27T13:28:45.605Z\"}"}
```
## 变更日志
<ChangeLog />
-
