This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13216-f5374addb396b3515fc2aff358b110ad35fe7874 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 7fd23f83f7836321bdde1397b785d3d3cb5d0ea9 Author: Jast <[email protected]> AuthorDate: Wed Nov 11 14:02:24 2026 +0000 [Docs] Sync diverged zh connector docs with English and fix inaccurate multi-table source section (#13216) Co-authored-by: zhangshenghang <[email protected]> --- docs/en/architecture/features/multi-table.md | 108 +-------- docs/zh/architecture/features/multi-table.md | 11 + .../common-options/sink-common-options.md | 2 + docs/zh/connectors/sink/AmazonSqs.md | 12 + docs/zh/connectors/sink/Easysearch.md | 2 +- docs/zh/connectors/sink/GraphQL.md | 49 ++++ docs/zh/connectors/sink/Http.md | 69 ++++++ docs/zh/connectors/sink/Paimon.md | 58 +++++ docs/zh/connectors/source/BosFile.md | 51 ++++ docs/zh/connectors/source/ObsFile.md | 258 ++++++++++++++++++++- docs/zh/connectors/source/Paimon.md | 12 + docs/zh/connectors/source/WooCommerce.md | 8 + 12 files changed, 539 insertions(+), 101 deletions(-) diff --git a/docs/en/architecture/features/multi-table.md b/docs/en/architecture/features/multi-table.md index a87d2e9851..77290da754 100644 --- a/docs/en/architecture/features/multi-table.md +++ b/docs/en/architecture/features/multi-table.md @@ -119,107 +119,17 @@ public class SinkIdentifier implements Serializable { ## 3. MultiTableSource Architecture -### 3.1 Structure +How a multi-table source is implemented depends on the connector (for example, CDC +connectors usually emit changes at the database/table level). -```java -public class MultiTableSource<T, SplitT, StateT> - implements SeaTunnelSource<T, SplitT, StateT> { - - // Underlying sources (one per table) - private final Map<TablePath, SeaTunnelSource<T, SplitT, StateT>> sources; - - // Produced catalog tables - private final List<CatalogTable> catalogTables; -} -``` - -### 3.2 Creation - -```java -// From configuration -MultiTableSource<SeaTunnelRow, ?, ?> multiSource = - MultiTableSource.builder() - .addSource(orderTablePath, orderSource) - .addSource(userTablePath, userSource) - .addSource(productTablePath, productSource) - .build(); -``` - -### 3.3 Enumerator: Unified Split Assignment - -```java -public class MultiTableSourceSplitEnumerator { - private final Map<TablePath, SourceSplitEnumerator> enumerators; - - @Override - public void handleSplitRequest(int subtaskId) { - // Round-robin across table enumerators - for (Map.Entry<TablePath, SourceSplitEnumerator> entry : enumerators.entrySet()) { - TablePath tablePath = entry.getKey(); - SourceSplitEnumerator enumerator = entry.getValue(); - - // Request split from table enumerator - enumerator.handleSplitRequest(subtaskId); - } - } - - @Override - public void addReader(int subtaskId) { - // Register reader with all table enumerators - for (SourceSplitEnumerator enumerator : enumerators.values()) { - enumerator.addReader(subtaskId); - } - } -} -``` - -### 3.4 Reader: Multi-Table Data Reading - -```java -public class MultiTableSourceReader { - private final Map<TablePath, SourceReader> readers; - private final Queue<TablePath> readOrder; // Round-robin queue - - @Override - public void pollNext(Collector<SeaTunnelRow> output) { - if (readOrder.isEmpty()) { - return; - } +To let downstream route rows by table, the core requirements are: +- Every emitted `SeaTunnelRow` must carry a `tableId` (usually the serialized string of `TablePath`) +- In change-stream scenarios it must also carry `rowKind` (INSERT/UPDATE/DELETE, etc.) so downstream can apply the correct semantics - // Round-robin read from tables - TablePath currentTable = readOrder.poll(); - SourceReader reader = readers.get(currentTable); - - // Read from current table - reader.pollNext(new Collector<SeaTunnelRow>() { - @Override - public void collect(SeaTunnelRow row) { - // Tag row with table path - row.setTableId(currentTable.serialize()); - output.collect(row); - } - }); - - // Re-add to queue for next round - readOrder.offer(currentTable); - } - - @Override - public void addSplits(List<SplitT> splits) { - // Route splits to correct table readers - for (SplitT split : splits) { - TablePath tablePath = extractTablePath(split); - SourceReader reader = readers.get(tablePath); - reader.addSplits(Collections.singletonList(split)); - - // Add table to read order if not present - if (!readOrder.contains(tablePath)) { - readOrder.offer(tablePath); - } - } - } -} -``` +Whether a connector internally maintains a TablePath-to-reader/enumerator mapping, +how it fairly schedules multiple tables, and whether it shares underlying +connections are implementation choices of the connector itself; the framework does +not mandate a fixed source-side structure. ## 4. MultiTableSink Architecture diff --git a/docs/zh/architecture/features/multi-table.md b/docs/zh/architecture/features/multi-table.md index c0bb7ca407..d03476b155 100644 --- a/docs/zh/architecture/features/multi-table.md +++ b/docs/zh/architecture/features/multi-table.md @@ -516,4 +516,15 @@ if (table.getWriteRate() > threshold) { ## 14. 参考资料 +### 关键源码文件 + +- [MultiTableSink.java](../../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java) +- [SinkIdentifier.java](../../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/SinkIdentifier.java) +- [TablePath.java](../../../../seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TablePath.java) + +### 示例实现 + +- MySQL CDC Source:`seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/` +- JDBC Sink:`seatunnel-connectors-v2/connector-jdbc/` + 如需进一步了解 Schema、Sink 语义与 DAG 执行,请从“相关资源”章节继续阅读。 diff --git a/docs/zh/connectors/common-options/sink-common-options.md b/docs/zh/connectors/common-options/sink-common-options.md index dfe6cd3ef0..a938fb7d32 100644 --- a/docs/zh/connectors/common-options/sink-common-options.md +++ b/docs/zh/connectors/common-options/sink-common-options.md @@ -69,5 +69,7 @@ sink { } ``` +> 注意:sink 只消费数据集,因此没有 `plugin_output` 选项。`plugin_input` 的值必须与上游 source 或 transform 声明的 `plugin_output` 一致。 +> > 如果作业只有一个 source 和一个(或零个)transform 和一个 sink ,则不需要为连接器指定 `plugin_input` 和 > `plugin_output`。 > 如果 source 、transform 和 sink 中任意运算符的数量大于 1,则必须为作业中的每个连接器指定 `plugin_input` 和 > `plugin_output` diff --git a/docs/zh/connectors/sink/AmazonSqs.md b/docs/zh/connectors/sink/AmazonSqs.md index 9a3be28722..d8a443ba62 100644 --- a/docs/zh/connectors/sink/AmazonSqs.md +++ b/docs/zh/connectors/sink/AmazonSqs.md @@ -48,6 +48,18 @@ Amazon SQS 写入连接器用于把每条输入的 SeaTunnel 行数据写入一 - `access_key_id` 和 `secret_access_key` 是可选项;如果使用静态 AWS 凭证,需要两个一起配置。 - 该写入连接器会把每条 SeaTunnel 行数据发送成一条 SQS 消息,不会把多行数据合并到一次 SQS 请求里。 +## 认证 + +连接器按以下顺序解析 AWS 凭证: + +1. 如果 `access_key_id` 和 `secret_access_key` 都已配置,则使用这对静态凭证。 +2. 否则,使用 AWS 默认凭证提供链(环境变量、实例 profile 等)。 + +如果要在 LocalStack 或 ElasticMQ 等 SQS 兼容服务上进行本地测试,把 `url` 指向本地 +endpoint(例如 `http://sqs-host:4566/...`),并提供任意非空的 +`access_key_id` / `secret_access_key` 即可。SQS 兼容测试服务通常不会校验请求的 +SigV4 签名,因此任意静态凭证对都可以被接受。 + ## 任务示例 ### 在本地兼容队列之间复制消息 diff --git a/docs/zh/connectors/sink/Easysearch.md b/docs/zh/connectors/sink/Easysearch.md index 633acf338e..2597538621 100644 --- a/docs/zh/connectors/sink/Easysearch.md +++ b/docs/zh/connectors/sink/Easysearch.md @@ -270,7 +270,7 @@ sink { } ``` -配置表生成策略 +### Save Mode(保存模式) ```hocon sink { diff --git a/docs/zh/connectors/sink/GraphQL.md b/docs/zh/connectors/sink/GraphQL.md index d14d448d43..b449f00984 100644 --- a/docs/zh/connectors/sink/GraphQL.md +++ b/docs/zh/connectors/sink/GraphQL.md @@ -157,6 +157,55 @@ sink { } ``` +### 向 GraphQL 服务流式发送 Mutation + +在流式模式下持续执行同一个 mutation。sink 会为每一行发送一次 mutation,并通过 +`retry` 加指数退避来吸收瞬时的 HTTP 失败。 + +```hocon +env { + parallelism = 2 + job.mode = "STREAMING" + checkpoint.interval = 30000 +} + +source { + FakeSource { + plugin_output = "events" + schema = { + fields { + id = int + val_string = string + } + } + rows = [ + { kind = INSERT, fields = [1, "first"] } + { kind = INSERT, fields = [2, "second"] } + ] + } +} + +sink { + GraphQL { + plugin_input = "events" + url = "http://graphql:8080/v1/graphql" + headers = { + Authorization = "Bearer ${secret}" + } + query = """ + mutation MyMutation($id: Int!, $val_string: String!) { + insert_event(objects: {id: $id, val_string: $val_string}) { + affected_rows + } + } + """ + retry = 5 + retry_backoff_multiplier_ms = 200 + retry_backoff_max_ms = 5000 + } +} +``` + ### 写入多张上游表 ```hocon diff --git a/docs/zh/connectors/sink/Http.md b/docs/zh/connectors/sink/Http.md index ff541ca986..8ac2dd3983 100644 --- a/docs/zh/connectors/sink/Http.md +++ b/docs/zh/connectors/sink/Http.md @@ -81,6 +81,75 @@ Http { } ``` +### 多表模式 + +#### 示例 1 + +```hocon +env { + parallelism = 1 + job.mode = "STREAMING" + checkpoint.interval = 5000 +} + +source { + Mysql-CDC { + url = "jdbc:mysql://127.0.0.1:3306/seatunnel" + username = "root" + password = "******" + + table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"] + } +} + +transform { +} + +sink { + Http { + ... + url = "http://localhost/test/${database_name}_test/${table_name}_test" + } +} +``` + +#### 示例 2 + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + Jdbc { + driver = oracle.jdbc.driver.OracleDriver + url = "jdbc:oracle:thin:@localhost:1521/XE" + user = testUser + password = testPassword + + table_list = [ + { + table_path = "TESTSCHEMA.TABLE_1" + }, + { + table_path = "TESTSCHEMA.TABLE_2" + } + ] + } +} + +transform { +} + +sink { + Http { + ... + url = "http://localhost/test/${schema_name}_test/${table_name}_test" + } +} +``` + ## 变更日志 <ChangeLog /> diff --git a/docs/zh/connectors/sink/Paimon.md b/docs/zh/connectors/sink/Paimon.md index 61e25aecd3..57535c0a8f 100644 --- a/docs/zh/connectors/sink/Paimon.md +++ b/docs/zh/connectors/sink/Paimon.md @@ -663,6 +663,64 @@ sink { } ``` +### 使用 Hive catalog 在启动时清空目标表 + +该示例在每次作业启动时依据上游 schema 重建 Hive catalog 中的 Paimon 表 +(`schema_save_mode = "RECREATE_SCHEMA"`),并清空已有数据,使 sink 表现为全新覆写。 +适用于需要全量刷新的批处理作业。 + +```hocon +env { + parallelism = 1 + job.mode = "BATCH" +} + +source { + FakeSource { + schema = { + fields { + pk_id = bigint + name = string + score = int + } + primaryKey { + name = "pk_id" + columnNames = [pk_id] + } + } + rows = [ + { kind = INSERT, fields = [1, "A", 100] } + { kind = INSERT, fields = [2, "B", 100] } + { kind = INSERT, fields = [3, "C", 100] } + { kind = UPDATE_BEFORE, fields = [1, "A", 100] } + { kind = UPDATE_AFTER, fields = [1, "A_1", 100] } + { kind = DELETE, fields = [2, "B", 100] } + ] + } +} + +sink { + Paimon { + warehouse = "hdfs:///tmp/paimon" + catalog_type = "hive" + catalog_uri = "thrift://hadoop04:9083" + database = "seatunnel_namespace12" + table = "st_test" + schema_save_mode = "RECREATE_SCHEMA" + data_save_mode = "DROP_DATA" + paimon.hadoop.conf = { + fs.defaultFS = "hdfs://nameservice1" + dfs.nameservices = "nameservice1" + dfs.ha.namenodes.nameservice1 = "nn1,nn2" + dfs.namenode.rpc-address.nameservice1.nn1 = "hadoop03:8020" + dfs.namenode.rpc-address.nameservice1.nn2 = "hadoop04:8020" + dfs.client.failover.proxy.provider.nameservice1 = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" + dfs.client.use.datanode.hostname = "true" + } + } +} +``` + ## FAQ ### Paimon Sink 连接器是否支持自动建表? diff --git a/docs/zh/connectors/source/BosFile.md b/docs/zh/connectors/source/BosFile.md index bb4852b1d2..f789831ac6 100644 --- a/docs/zh/connectors/source/BosFile.md +++ b/docs/zh/connectors/source/BosFile.md @@ -92,6 +92,34 @@ import ChangeLog from '../changelog/connector-file-bos.md'; | sort_files_by_modification_time | boolean | 否 | false | | common-options | | 否 | - | +### path [string] + +桶下的源文件路径。 + +### bucket [string] + +BOS 桶地址,例如 `bos://my-bucket`。 + +### access_key [string] + +百度云 BOS 的 access key。 + +### secret_key [string] + +百度云 BOS 的 secret key。 + +### endpoint [string] + +BOS 的 endpoint,例如 `http://bj.bcebos.com`。 + +### file_format_type [string] + +支持的文件类型:`text`、`csv`、`parquet`、`orc`、`json`、`excel`、`xml`、`binary`、`markdown`、`pdf`。 + +### common-options + +Source 插件通用参数,详情请参见 [Source Common Options](../common-options/source-common-options.md)。 + ## 示例 ```hocon @@ -118,6 +146,7 @@ source { access_key = "your-access-key" secret_key = "your-secret-key" endpoint = "http://bj.bcebos.com" + binary_chunk_size = 2048 } } @@ -133,6 +162,28 @@ sink { } ``` +### 文件过滤 + +```hocon +source { + BosFile { + bucket = "bos://source-bucket" + path = "/read/data/" + file_format_type = "text" + access_key = "your-access-key" + secret_key = "your-secret-key" + endpoint = "http://bj.bcebos.com" + file_filter_pattern = "abc[DX]*.*" + schema { + fields { + id = int + name = string + } + } + } +} +``` + ## 变更日志 <ChangeLog /> diff --git a/docs/zh/connectors/source/ObsFile.md b/docs/zh/connectors/source/ObsFile.md index cc56deffae..9860420ff4 100644 --- a/docs/zh/connectors/source/ObsFile.md +++ b/docs/zh/connectors/source/ObsFile.md @@ -103,11 +103,137 @@ import ChangeLog from '../changelog/connector-file-obs.md'; | recursive_file_scan | boolean | 否 | true | 是否递归扫描子目录。 如果设置为 `false`,将忽略子目录,仅扫描指定路径下的文件。 | | sort_files_by_modification_time | boolean | 否 | false | 是否按修改时间降序排序文件。启用此选项后,在读取不断演化的 schema 时可确保 schema 推断使用最新的文件。 | +### parse_partition_from_path [boolean] + +> 控制是否从文件路径解析分区键和值 +> +> 例如,从路径 `obs://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26` 读取文件时, +> +> 文件中的每条记录数据都会被追加以下两个字段: + +| name | age | +|---------------|-----| +| tyrantlucifer | 26 | + +> 不要在 schema 选项中定义分区字段 + +### date_format [string] + +> 日期类型格式,用于告诉连接器如何将字符串转换为日期,支持以下格式: +> +> `yyyy-MM-dd` `yyyy.MM.dd` `yyyy/MM/dd` +> +> 默认值为 `yyyy-MM-dd` + +### datetime_format [string] + +> 日期时间类型格式,用于告诉连接器如何将字符串转换为日期时间,支持以下格式: +> +> `yyyy-MM-dd HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd HH:mm:ss` `yyyyMMddHHmmss` +> +> 默认值为 `yyyy-MM-dd HH:mm:ss` + +### time_format [string] + +> 时间类型格式,用于告诉连接器如何将字符串转换为时间,支持以下格式: +> +> `HH:mm:ss` `HH:mm:ss.SSS` +> +> 默认值为 `HH:mm:ss` + +### skip_header_row_number [long] + +> 跳过前几行,但仅适用于 txt 和 csv 文件。 +> +> 例如,配置如下: +> +> `skip_header_row_number = 2` +> +> 则 Seatunnel 会跳过源文件的前 2 行 + ### file_format_type [string] 文件类型,支持以下文件类型: -`text` `csv` `parquet` `orc` `json` `excel` `markdown` +`text` `csv` `parquet` `orc` `json` `excel` `xml` `binary` `markdown` `pdf` + +如果您将文件类型指定为 `json`,则需要同时配置 schema 选项,告诉连接器如何将数据解析成想要的行。 + +例如,上游数据如下: + +```json +{"code": 200, "data": "get success", "success": true} +``` + +> 一个文件中也可以保存多条数据,并用一个换行符分隔: + +```json lines + +{"code": 200, "data": "get success", "success": true} +{"code": 300, "data": "get failed", "success": false} + +``` + +> 此时 schema 应配置如下: + +```hocon + +schema { + fields { + code = int + data = string + success = boolean + } +} + +``` + +> 连接器会生成如下数据: + +| code | data | success | +|------|-------------|---------| +| 200 | get success | true | + +> 如果文件类型指定为 `parquet`、`orc`,则不需要配置 schema 选项,连接器可以自动获取上游数据的 schema。 +> +> 如果文件类型指定为 `text`、`csv`,可以选择是否指定 schema 信息。 +> +> 例如,上游数据如下: + +```text + +tyrantlucifer#26#male + +``` + +> 如果不配置数据 schema,连接器会将上游数据按如下处理: + +| content | +|-----------------------| +| tyrantlucifer#26#male | + +> 如果配置了数据 schema,除 CSV 文件类型外,还需要同时配置 `field_delimiter` 选项 +> +> schema 和 field_delimiter 应配置如下: + +```hocon + +field_delimiter = "#" +schema { + fields { + name = string + age = int + gender = string + } +} + +``` + +> 连接器会生成如下数据: + +| name | age | gender | +|---------------|-----|--------| +| tyrantlucifer | 26 | male | 如果您将文件类型指定为 `markdown`,SeaTunnel 可以解析 markdown 文件并提取结构化数据。 markdown 解析器提取各种元素,包括标题、段落、列表、代码块、表格等。 @@ -159,6 +285,31 @@ PDF 特有的解析行为如下: 注意:仅支持单栏(从上到下)PDF 布局。不支持多栏布局(例如并排的双栏文档),可能会产生不正确的文本顺序。 +### schema [config] + +#### fields + +> 上游数据的 schema。更多细节请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。 + +### read_columns [list] + +> 数据源的读取列列表,用户可以用它来实现字段投影。 +> +> 支持列投影的文件类型如下所示: + +- text +- json +- csv +- orc +- parquet +- excel + +> 如果用户在读取 `text` `json` `csv` 文件时想使用该特性,则必须配置 schema 选项 + +### common-options + +> Source 插件通用参数,详情请参见 [Source Common Options](../common-options/source-common-options.md)。 + ### sort_files_by_modification_time [boolean] 是否按修改时间降序排序文件。默认值为 `false`。 @@ -167,6 +318,111 @@ PDF 特有的解析行为如下: - 读取具有不断演化的 schema 的文件,且希望 schema 推断使用最新的文件 - 需要按时间顺序处理文件 +## 任务示例 + +### text 文件 + +> text 文件格式的简单配置 + +```hocon + + ObsFile { + path = "/seatunnel/text" + bucket = "obs://obs-bucket-name" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "obs.xxxxxx.myhuaweicloud.com" + file_format_type = "text" + } + +``` + +### parquet 文件 + +> parquet 文件格式的简单配置 + +```hocon + + ObsFile { + path = "/seatunnel/parquet" + bucket = "obs://obs-bucket-name" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "obs.xxxxxx.myhuaweicloud.com" + file_format_type = "parquet" + } + +``` + +### orc 文件 + +> orc 文件格式的简单配置 + +```hocon + + ObsFile { + path = "/seatunnel/orc" + bucket = "obs://obs-bucket-name" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "obs.xxxxxx.myhuaweicloud.com" + file_format_type = "orc" + } + +``` + +### json 文件 + +> json 文件格式的简单配置 + +```hocon + + ObsFile { + path = "/seatunnel/json" + bucket = "obs://obs-bucket-name" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "obs.xxxxxx.myhuaweicloud.com" + file_format_type = "json" + } + +``` + +### excel 文件 + +> excel 文件格式的简单配置 + +```hocon + + ObsFile { + path = "/seatunnel/excel" + bucket = "obs://obs-bucket-name" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "obs.xxxxxx.myhuaweicloud.com" + file_format_type = "excel" + } + +``` + +### csv 文件 + +> csv 文件格式的简单配置 + +```hocon + + ObsFile { + path = "/seatunnel/csv" + bucket = "obs://obs-bucket-name" + access_key = "xxxxxxxxxxxxxxxxx" + access_secret = "xxxxxxxxxxxxxxxxxxxxxx" + endpoint = "obs.xxxxxx.myhuaweicloud.com" + file_format_type = "csv" + field_delimiter = "," + } + +``` + ### 使用 OBS STS 临时安全凭证读取 生产环境建议通过 [OBS STS](https://support.huaweicloud.com/intl/zh-cn/api-obs/obs_04_0081.html) 颁发临时 AK/SK,并配合细粒度自定义策略限制只能访问指定 bucket 前缀,再通过 `hadoop_obs_properties` 传给连接器。 diff --git a/docs/zh/connectors/source/Paimon.md b/docs/zh/connectors/source/Paimon.md index a9e8add00b..662d8da2b5 100644 --- a/docs/zh/connectors/source/Paimon.md +++ b/docs/zh/connectors/source/Paimon.md @@ -72,6 +72,10 @@ import ChangeLog from '../changelog/connector-paimon.md'; Paimon warehouse 路径,不能为空白字符串。 +### catalog_name [string] + +Paimon catalog 的名称。默认值为 `paimon`。 + ### catalog_type [string] Paimon Catalog 类型,支持 filesystem 和 hive @@ -92,6 +96,14 @@ Paimon 的 catalog URI。当 `catalog_type` 为 `hive` 时必须配置。 `Paimon` 表名列表。当一个 Source 需要读取多张 Paimon 表时,使用此配置代替 `table`。`table` 和 `table_list` 必须二选一配置。每个配置项必须包含 `database` 和 `table`,也可以配置该表自己的 `query`。 +### user [string] + +用于访问 Paimon 表的用户名。需与 `password` 一起配置。当底层 catalog 开启认证时通常必须配置。 + +### password [string] + +Paimon 用户密码。配置了 `user` 时必须配置。 + ### hdfs_site_path [string] `hdfs-site.xml` 文件地址。该选项已废弃,新作业建议使用 `paimon.hadoop.conf` 或 `paimon.hadoop.conf-path`。 diff --git a/docs/zh/connectors/source/WooCommerce.md b/docs/zh/connectors/source/WooCommerce.md index fe14c6ff8f..b2b5065111 100644 --- a/docs/zh/connectors/source/WooCommerce.md +++ b/docs/zh/connectors/source/WooCommerce.md @@ -23,6 +23,14 @@ import ChangeLog from '../changelog/connector-http-woocommerce.md'; 通过 HTTPS 调用 `GET /wp-json/wc/v3/orders`,批量读取指定 UTC 创建时间范围内的订单。仅支持 BATCH 和 parallelism=1,不支持 CDC、流式读取、Exactly-once 或事务一致性快照。连接器只发送 GET,不修改订单。 +## 主要特性 + +- [x] 批处理 +- [ ] 流处理 +- [ ] 精确一次 + +并行度必须为 1。恢复时会重新读取整个配置的时间窗口,不会对远程分页位置做 checkpoint。作业失败时,可能已经有部分记录写入了非事务型 sink。如果重放语义重要,请使用按商店和订单 ID 幂等的 sink。 + ## 使用前提 为有订单读取权限的用户创建 Read 权限 REST API 密钥。配置最终 HTTPS 商店地址(可包含 WordPress 子目录);不跟随重定向。服务器证书必须通过 JVM 校验,私有 CA 应导入每个工作节点的 JVM truststore。WordPress 需要向 WooCommerce 转发 Authorization 头。连接器使用 consumer key/secret 的 Basic 认证,不使用 WordPress 登录密码或公开 Store API。
