This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13089-b3db27900bd4da855f0247e791e672f7fe301a4e in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 70ef5cd546cf2184cac0bd6d8c7f4399bffc1e9a Author: Jast <[email protected]> AuthorDate: Mon Nov 2 10:42:32 2026 +0000 [Docs] Fix file, Hive, Iceberg, MongoDB and DataHub doc issues verified against source code (#13089) Co-authored-by: zhangshenghang <[email protected]> Co-authored-by: jast <[email protected]> --- .../common-options/source-common-options.md | 2 +- docs/en/connectors/sink/Datahub.md | 17 +++--- docs/en/connectors/sink/HdfsFile.md | 7 ++- docs/en/connectors/sink/Hive.md | 4 +- docs/en/connectors/sink/LocalFile.md | 4 ++ docs/en/connectors/sink/MongoDB.md | 4 +- docs/en/connectors/source/HdfsFile.md | 7 ++- docs/en/connectors/source/Hive.md | 8 ++- docs/en/connectors/source/Iceberg.md | 19 +++++- docs/en/connectors/source/LocalFile.md | 4 +- .../common-options/source-common-options.md | 2 +- docs/zh/connectors/sink/Datahub.md | 17 +++--- docs/zh/connectors/sink/HdfsFile.md | 7 ++- docs/zh/connectors/sink/Hive.md | 2 +- docs/zh/connectors/sink/Iceberg.md | 68 +++++++++++++++------- docs/zh/connectors/sink/LocalFile.md | 4 ++ docs/zh/connectors/sink/MongoDB.md | 4 +- docs/zh/connectors/source/HdfsFile.md | 7 ++- docs/zh/connectors/source/Hive.md | 8 ++- docs/zh/connectors/source/Iceberg.md | 25 ++++++-- docs/zh/connectors/source/LocalFile.md | 4 +- 21 files changed, 162 insertions(+), 62 deletions(-) diff --git a/docs/en/connectors/common-options/source-common-options.md b/docs/en/connectors/common-options/source-common-options.md index f1d10d446cd..2b06db85369 100644 --- a/docs/en/connectors/common-options/source-common-options.md +++ b/docs/en/connectors/common-options/source-common-options.md @@ -17,7 +17,7 @@ The old configuration name `result_table_name` is deprecated, please migrate to | plugin_output | String | No | - | When `plugin_output` is not specified, the data processed by this plugin will not be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` <br/>When `plugin_output` is specified, the data processed by this plugin will be registered as a data set `(dataStream/dataset)` that can be directly accessed by other plugins, or called a temporary table `(table)` . The dat [...] | parallelism | Int | No | - | When `parallelism` is not specified, the `parallelism` in env is used by default. <br/>When parallelism is specified, it will override the parallelism in env. [...] | metadata_datasource_id | String | No | - | The data source ID for retrieving connection configuration from Metadata Center. When specified, the connector will fetch connection details (e.g., URL, username, password) from the external metadata service instead of using direct configuration. See [Metadata SPI](../../introduction/concepts/metadata-spi.md) for more information. [...] -| dag-parsing.mode | Enum | No | SINGLENESS | Whether to enable parsing support for multi-table source. `SINGLENESS` processes each table with a separate source and sink; `MULTIPLEXING` processes multiple tables (such as sharded tables) with one source and one sink. | +| dag-parsing.mode | Enum | No | SINGLENESS | Whether to enable parsing support for multi-table source. `SINGLENESS` processes each table with a separate source and sink; `SHARDING` processes multiple tables (such as sharded tables) with one source and one sink. `MULTIPLEX` is deprecated: it processes multiple tables with a single source, each table using a separate sink. | # Important note diff --git a/docs/en/connectors/sink/Datahub.md b/docs/en/connectors/sink/Datahub.md index e58bcca3da5..96543a8ef58 100644 --- a/docs/en/connectors/sink/Datahub.md +++ b/docs/en/connectors/sink/Datahub.md @@ -15,7 +15,7 @@ import ChangeLog from '../changelog/connector-datahub.md'; The DataHub sink writes SeaTunnel rows to Alibaba Cloud DataHub. The connector supports single-table writes and multi-table writes. In multi-table -jobs, use placeholders such as `${table}` in `topic` to route records from +jobs, use placeholders such as `${table_name}` in `topic` to route records from different input tables to different DataHub topics. ## Key features @@ -39,9 +39,10 @@ SeaTunnel schema fields, because the sink writes values by field name. | accessId | string | yes | - | Alibaba Cloud access ID used to access DataHub. | | accessKey | string | yes | - | Alibaba Cloud access key used to access DataHub. | | project | string | yes | - | DataHub project name. | -| topic | string | yes | - | DataHub topic name. Supports placeholders in multi-table jobs. | +| topic | string | yes | - | DataHub topic name. Supports placeholders such as `${table_name}` in multi-table jobs. | | timeout | int | no | 3000 | Maximum client connection timeout in milliseconds. | | retryTimes | int | no | 3 | Maximum retry count when writing a record fails. | +| multi_table_sink_replica | int | no | 1 | The replica number of multi table sink writer. Must be at least 1. | | common-options | config | no | - | Sink plugin common options. See [Sink Common Options](../common-options/sink-common-options.md). | ### endpoint [string] @@ -63,8 +64,10 @@ The DataHub project name. ### topic [string] The DataHub topic name. For multi-table writes, this value can contain -placeholders such as `${table}`. `${table_name}` is only kept as a -deprecated compatibility alias; use `${table}` for new jobs. +placeholders such as `${table_name}`, `${database_name}` and `${schema_name}` +(supported placeholders follow the SeaTunnel `TablePlaceholder` definitions, +for example `topic = "${table_name}"`). Note that `${table}` is **not** a valid +placeholder — the topic would be created with the literal string `${table}`. The SeaTunnel field names must match the DataHub topic fields, because the sink writes fields by name according to the topic schema. @@ -124,7 +127,7 @@ sink { ### Write Multiple Input Tables to Matching Topics When the upstream source provides multiple tables, configure `topic` with the -`${table}` placeholder so each input table is routed to a topic with the same name. +`${table_name}` placeholder so each input table is routed to a topic with the same name. ```hocon env { @@ -167,7 +170,7 @@ sink { accessId = "your-access-id" accessKey = "your-access-key" project = "demo_project" - topic = "${table}" + topic = "${table_name}" timeout = 3000 retryTimes = 3 } @@ -182,7 +185,7 @@ No. The connector performs best-effort writes with bounded retries (`retryTimes` ### How does multi-table routing work? -When the upstream source emits more than one table, set `topic` to a value containing the `${table}` placeholder (for example `topic = "${table}"`). Each input table is then routed to a DataHub topic that shares its name. `${table_name}` is still recognized as a deprecated alias — prefer `${table}` in new jobs. The connector does not automatically create the target topic; create it in the DataHub project beforehand and ensure its schema field names match the upstream SeaTunnel schema. +When the upstream source emits more than one table, set `topic` to a value containing the `${table_name}` placeholder (for example `topic = "${table_name}"`). Each input table is then routed to a DataHub topic that shares its name. The connector does not automatically create the target topic; create it in the DataHub project beforehand and ensure its schema field names match the upstream SeaTunnel schema. ### Why is `topic` a required field even for single-table jobs? diff --git a/docs/en/connectors/sink/HdfsFile.md b/docs/en/connectors/sink/HdfsFile.md index 1a04dac2d73..8da3a341adc 100644 --- a/docs/en/connectors/sink/HdfsFile.md +++ b/docs/en/connectors/sink/HdfsFile.md @@ -60,8 +60,11 @@ Output data to hdfs file | filename_time_format | string | no | "yyyy.MM.dd" | Only used when `custom_filename` is `true`.When the format in the `file_name_expression` parameter is `xxxx-${now}` , `filename_time_format` can specify the time format of the path, and the default value is `yyyy.MM.dd` . The commonly used time formats are listed as follows:[y:Year,M:Month,d:Day of month,H:Hour in day (0-23),m:Minute in hour,s:Second in minute] [...] | file_format_type | string | no | "csv" | We supported as the following file types:`text` `csv` `parquet` `orc` `json` `excel` `xml` `binary` `canal_json` `debezium_json` `maxwell_json`.Please note that, The final file name will end with the file_format's suffix, the suffix of the text file is `txt`. [...] | filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` [...] -| field_delimiter | string | no | '\001' for text and ',' for csv | Only used when file_format is text and csv,The separator between columns in a row of data. Only needed by `text` file format. [...] -| row_delimiter | string | no | "\n" | Only used when file_format is text,The separator between rows in a file. Only needed by `text`, `csv` and `json` file format. [...] +| date_format | string | no | yyyy-MM-dd | Only used when file_format is `text`, `csv`, `excel` or `xml`. Date type format, used to tell connector how to write date values. Supported formats (examples): `yyyy-MM-dd`, `yyyy.MM.dd`, `yyyy/MM/dd` [...] +| datetime_format | string | no | yyyy-MM-dd HH:mm:ss | Only used when file_format is `text`, `csv`, `excel` or `xml`. Datetime type format, used to tell connector how to write datetime values. Supported formats (examples): `yyyy-MM-dd HH:mm:ss`, `yyyy.MM.dd HH:mm:ss`, `yyyy/MM/dd HH:mm:ss`, `yyyyMMddHHmmss` [...] +| time_format | string | no | HH:mm:ss | Only used when file_format is `text`, `csv`, `excel` or `xml`. Time type format, used to tell connector how to write time values. Supported formats (examples): `HH:mm:ss`, `HH:mm:ss.SSS` [...] +| field_delimiter | string | no | '\001' for text and ',' for csv | Only used when file_format is `text` and `csv`. The separator between columns in a row of data. [...] +| row_delimiter | string | no | "\n" | Only used when file_format is `text`, `csv` and `json`. The separator between rows in a file. [...] | have_partition | boolean | no | false | Whether you need processing partitions. [...] | partition_by | array | no | - | Only used then have_partition is true,Partition data based on selected fields. [...] | partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true,If the `partition_by` is specified, we will generate the corresponding partition directory based on the partition information, and the final file will be placed in the partition directory. Default `partition_dir_expression` is `${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/`. `k0` is the first partition field and `v0` is the value of the first partit [...] diff --git a/docs/en/connectors/sink/Hive.md b/docs/en/connectors/sink/Hive.md index 37100d6229f..d7895c75ad1 100644 --- a/docs/en/connectors/sink/Hive.md +++ b/docs/en/connectors/sink/Hive.md @@ -55,7 +55,7 @@ By default, we use 2PC commit to ensure `exactly-once` | overwrite | boolean | no | false | Replace existing data before commit. Equivalent to `data_save_mode = "DROP_DATA"`. | | data_save_mode | enum | no | APPEND_DATA | How to handle existing data before writing. `APPEND_DATA` (default), `DROP_DATA`, `CUSTOM_PROCESSING`, `ERROR_WHEN_DATA_EXISTS`. | | schema_save_mode | enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | How to handle the target table schema before writing. `RECREATE_SCHEMA`, `CREATE_SCHEMA_WHEN_NOT_EXIST`, `ERROR_WHEN_SCHEMA_NOT_EXIST`, `IGNORE`. | -| save_mode_create_template | string | no | - | Custom DDL template used when auto-creating the target Hive table. Variables: `${database}`, `${table}`, `${rowtype_fields}`, `${rowtype_partition_fields}`, `${table_location}`. | +| save_mode_create_template | string | no | - | Custom DDL template used when auto-creating the target Hive table. Variables: `${database}`, `${table}`, `${rowtype_fields}`, `${rowtype_partition_fields}`, `${table_location}`, `${current_timestamp}`. | | common-options | | no | - | Sink plugin common parameters. See [Sink Common Options](../common-options/sink-common-options.md). | ### table_name [string] @@ -168,7 +168,7 @@ Option values: ### save_mode_create_template [string] -We use templates to automatically create Hive tables, which will create corresponding table creation statements based on the type of upstream data and schema type, and the default template can be modified according to the situation. Available template variables: ${database}, ${table}, ${rowtype_fields}, ${rowtype_partition_fields}, ${table_location}. +We use templates to automatically create Hive tables, which will create corresponding table creation statements based on the type of upstream data and schema type, and the default template can be modified according to the situation. Available template variables: ${database}, ${table}, ${rowtype_fields}, ${rowtype_partition_fields}, ${table_location}, ${current_timestamp}. **Default value**: When not specified, uses a default PARQUET non-partitioned table template: ```sql diff --git a/docs/en/connectors/sink/LocalFile.md b/docs/en/connectors/sink/LocalFile.md index f68f66cf189..2b873017006 100644 --- a/docs/en/connectors/sink/LocalFile.md +++ b/docs/en/connectors/sink/LocalFile.md @@ -59,6 +59,10 @@ By default, we use 2PC commit to ensure `exactly-once` | filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true. | | file_format_type | string | no | "csv" | File format type, supported: `text`, `csv`, `parquet`, `orc`, `json`, `excel`, `xml`, `binary`, `canal_json`, `debezium_json`, `maxwell_json`. | | filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` | +| date_format | string | no | yyyy-MM-dd | Only used when file_format_type is `text`, `csv`, `excel` or `xml`. Date type format, used to tell connector how to write date values. Supported formats (examples): `yyyy-MM-dd`, `yyyy.MM.dd`, `yyyy/MM/dd`. | +| datetime_format | string | no | yyyy-MM-dd HH:mm:ss | Only used when file_format_type is `text`, `csv`, `excel` or `xml`. Datetime type format, used to tell connector how to write datetime values. Supported formats (examples): `yyyy-MM-dd HH:mm:ss`, `yyyy.MM.dd HH:mm:ss`, `yyyy/MM/dd HH:mm:ss`, `yyyyMMddHHmmss`. | +| time_format | string | no | HH:mm:ss | Only used when file_format_type is `text`, `csv`, `excel` or `xml`. Time type format, used to tell connector how to write time values. Supported formats (examples): `HH:mm:ss`, `HH:mm:ss.SSS`. | +| multi_table_sink_replica | int | no | 1 | The replica number of multi table sink writer. Must be at least 1. | | field_delimiter | string | no | '\001' for text and ',' for csv | Only used when file_format_type is text and csv. | | row_delimiter | string | no | "\n" | Only used when file_format_type is `text`, `csv` and `json`. | | have_partition | boolean | no | false | Whether you need processing partitions. | diff --git a/docs/en/connectors/sink/MongoDB.md b/docs/en/connectors/sink/MongoDB.md index fe97c3a3607..340a6cfb631 100644 --- a/docs/en/connectors/sink/MongoDB.md +++ b/docs/en/connectors/sink/MongoDB.md @@ -74,7 +74,6 @@ The following table lists the field data type mapping from SeaTunnel data type t | SeaTunnel Data Type | MongoDB BSON Type | |---------------------|-------------------| -| STRING | ObjectId | | STRING | String | | BOOLEAN | Boolean | | BINARY | Binary | @@ -97,6 +96,9 @@ The following table lists the field data type mapping from SeaTunnel data type t > midnight of that day; a SeaTunnel `Timestamp` keeps its time-of-day. > 2. When using the `DECIMAL` type in SeaTunnel, the maximum range cannot > exceed 34 digits. Use > `decimal(34, 18)` to stay within the supported precision and scale. +> 3. SeaTunnel `STRING` values are always written as BSON `String`. They are not converted to +> `ObjectId` on write — even for the `_id` field. To work with Extended JSON values, encode the +> value as `{"_value": <extended-json>}` (see the source `Data Type Mapping` section). ## Sink Options diff --git a/docs/en/connectors/source/HdfsFile.md b/docs/en/connectors/source/HdfsFile.md index 5e776408275..2759fbe4df2 100644 --- a/docs/en/connectors/source/HdfsFile.md +++ b/docs/en/connectors/source/HdfsFile.md @@ -69,7 +69,7 @@ Read data from hdfs file system. | krb5_path | string | no | /etc/krb5.conf | The krb5 path of kerberos | | kerberos_principal | string | no | - | The principal of kerberos | | kerberos_keytab_path | string | no | - | The keytab path of kerberos | -| skip_header_row_number | long | no | 0 | Skip the first few lines, but only for the txt and csv.For example, set like following:`skip_header_row_number = 2`.then Seatunnel will skip the first 2 lines from source files | +| skip_header_row_number | long | no | 0 | Skip the first few lines, for text, CSV, and Excel files. For example, set like following:`skip_header_row_number = 2`.then Seatunnel will skip the first 2 lines from source files | | schema | config | no | - | the schema fields of upstream data. For more details, please refer to [Schema Feature](../../introduction/concepts/schema-feature.md). **metadata_table_id**: The table identifier in the metadata service to fetch table schema. For Gravitino, the format should be `{catalog}.{database}.{table}`, such as `mysql-catalog.test_db.users`. When using Gravitino as the metadata source, the column types from Gravitino [...] | sheet_name | string | no | - | Reader the sheet of the workbook,Only used when file_format is excel. | | excel_engine | string | no | POI | Only used when `file_format` is excel. Supported engines are `POI` and `EasyExcel`. | @@ -82,7 +82,10 @@ Read data from hdfs file system. | compress_codec | string | no | none | The compress codec of files | | archive_compress_codec | string | no | none | Which archive compress codec the files used. Supported: `none` `zip` `tar` `tar.gz` `gz`. | | encoding | string | no | UTF-8 | | -| null_format | string | no | - | Only used when file_format_type is text. null_format to define which strings can be represented as null. e.g: `\N` | +| null_format | string | no | - | Only used when file_format_type is `text` or `csv`. null_format to define which strings can be represented as null. e.g: `\N` | +| excel_use_header_line | boolean | no | false | Use the first Excel row as column headers for name-based mapping. Only used when file_format_type is `excel`. | +| markdown_rag_metadata_enabled | boolean | no | false | Whether to append RAG-oriented metadata columns when reading markdown files. Only valid when file_format_type is `markdown`. | +| pdf_rag_metadata_enabled | boolean | no | false | Whether to append RAG-oriented metadata columns when reading PDF files. Only valid when file_format_type is `pdf`. | | binary_chunk_size | int | no | 1024 | Only used when file_format_type is binary. The chunk size (in bytes) for reading binary files. Default is 1024 bytes. Larger values may improve performance for large files but use more memory. | | binary_complete_file_mode | boolean | no | false | Only used when file_format_type is binary. Whether to read the complete file as a single chunk instead of splitting into chunks. When enabled, the entire file content will be read into memory at once. Default is false. | | discovery_mode | string | no | once | File discovery mode. Supported values: `once` (default), `continuous`. When `continuous`, the source keeps scanning the path and processes new/changed files at runtime (unbounded). In the current implementation, `continuous` requires `sync_mode=update` (binary only). | diff --git a/docs/en/connectors/source/Hive.md b/docs/en/connectors/source/Hive.md index db262f9388e..cd84394542d 100644 --- a/docs/en/connectors/source/Hive.md +++ b/docs/en/connectors/source/Hive.md @@ -42,7 +42,7 @@ Read all the data in a split in a pollNext call. What splits are read will be sa | name | type | required | default value | description | |-----------------------|--------|----------|----------------|-------------| | table_name | string | no | Required for single-table mode | Target Hive table name in the form `db1.table1`. When `use_regex = true`, this field uses `databasePattern.tablePattern` to match multiple tables. | -| table_list | array | no | - | Multi-table configuration list. Each item can override any of the root-level options. Takes precedence if both `table_list` and `tables_configs` are set. | +| table_list | array | no | - | Multi-table configuration list. Each item is a self-contained configuration: options such as `table_name`, `metastore_uri` and the authentication/Hadoop settings are **not** inherited from the root block and must be repeated in every item (connector options outside `table_list` are ignored when `table_list` is present; framework-level options such as `plugin_output` and `parallelism` are still honoured). Takes precedence if b [...] | tables_configs | array | no | Deprecated, use `table_list` instead | Deprecated multi-table configuration list. Hive is a structured data source, new jobs should use `table_list`. Kept for backward compatibility. | | use_regex | boolean| no | false | Treat `table_name` as a regular expression that matches multiple tables. Works at the root level and inside each `table_list` / `tables_configs` entry. | | metastore_uri | string | no | Required for single-table mode | Hive metastore URI. Comma-separated values enable HA failover; whitespace is ignored. | @@ -56,6 +56,8 @@ Read all the data in a split in a pollNext call. What splits are read will be sa | remote_user | string | no | - | Hadoop remote user name used when connecting to HDFS / Hive storage without Kerberos. | | read_partitions | list | no | - | Restrict the read to a subset of partitions. All entries must have the same directory depth. | | read_columns | list | no | - | Column projection list. Only the listed columns are read from the source. | +| null_format | string | no | - | Custom string that represents null values in text tables. When set, it overrides the Hive table's `serialization.null.format` (which itself defaults to `\N`). | +| parse_partition_from_path | boolean | no | true | Whether to append the partition key columns to the projected schema (as STRING) when the schema is built from the Hive metastore. | | compress_codec | string | no | none | Compression codec for text tables. `lzo` and `none` are supported. Parquet / ORC auto-detect compression. | | common-options | | no | - | Source plugin common parameters. See [Source Common Options](../common-options/source-common-options.md). | @@ -67,7 +69,7 @@ For a single-table source, configure `table_name` and `metastore_uri` at the roo ### table_list [array] -Multi-table configuration list. Each item can contain `table_name`, `metastore_uri`, `use_regex`, `read_partitions`, `read_columns`, and the same authentication/Hadoop options as the root connector block. +Multi-table configuration list. Each item is a self-contained configuration and can contain `table_name`, `metastore_uri`, `use_regex`, `read_partitions`, `read_columns`, and authentication/Hadoop options — but nothing is inherited from the root block, so required options such as `metastore_uri` must be repeated in every item. ### tables_configs [array] @@ -75,7 +77,7 @@ Deprecated multi-table configuration list, kept for backward compatibility; new ### use_regex [boolean] -Whether to treat `table_name` as a regular expression pattern for matching multiple tables (whole database / subset). This also works inside each entry of `table_list` / `tables_configs`. +Whether to treat `table_name` as a regular expression pattern for matching multiple tables (whole database / subset). This also works inside each entry of `table_list` / `tables_configs`. Regex discovery requires a reachable Hive metastore: `metastore_uri` must be set (at root level or inside the corresponding `table_list` entry), otherwise the job fails. Regex syntax notes: - The dot (`.`) is treated as the separator between database and table patterns (Hive only supports `database.table`). diff --git a/docs/en/connectors/source/Iceberg.md b/docs/en/connectors/source/Iceberg.md index bd52a916636..67ec88a0580 100644 --- a/docs/en/connectors/source/Iceberg.md +++ b/docs/en/connectors/source/Iceberg.md @@ -84,7 +84,7 @@ libfb303-xxx.jar | catalog_name | string | no | default | User-specified catalog name. [...] | namespace | string | no | default | The iceberg database name in the backend catalog. [...] | table | string | no | - | The Iceberg table name in the backend catalog. Configure exactly one of `table` and `table_list`. [...] -| table_list | list | no | - | The Iceberg table list in the backend catalog. Configure exactly one of `table` and `table_list`. Each item can set `table`, `query`, and snapshot/stream scan options for that table. [...] +| table_list | list | no | - | The Iceberg table list in the backend catalog. Configure exactly one of `table` and `table_list`. Each item can set `table`, `query`, and snapshot/stream scan options for that table. Note: item keys use camelCase (e.g. `startSnapshotId`, `streamScanStrategy`), see [Multi-Table Read](#multi-table-read). [...] | iceberg.catalog.config | map | yes | - | Specify the properties for initializing the Iceberg catalog, which can be referenced in this file: [CatalogProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java) [...] | hadoop.config | map | no | - | Properties passed through to the Hadoop configuration [...] | iceberg.hadoop-conf-path | string | no | - | The specified loading paths for the 'core-site.xml', 'hdfs-site.xml', 'hive-site.xml' files. [...] @@ -142,6 +142,23 @@ sink { Use `table_list` when one Iceberg source needs to read more than one table. Do not set `table` at the same time. +Each `table_list` item is parsed as a standalone config and supports the following per-table keys (note that, unlike the root-level options, these keys are camelCase): + +| Item key | Type | Description | +|-------------------------|--------|------------------------------------------------------------------------------------| +| namespace | string | Ignored: each item is overwritten with the root-level `namespace`, so this key has no effect. To read from another database, use a qualified `db.table` value in `table`. | +| table | string | The Iceberg table name. | +| query | string | Filter query for this table, same as the root-level `query` option. | +| startSnapshotTimestamp | long | Millis since the Unix epoch; scan changes starting from the latest snapshot at this timestamp. | +| startSnapshotId | long | Scan changes starting from this snapshot (exclusive). | +| endSnapshotId | long | Scan changes up to this snapshot (inclusive). | +| useSnapshotId | long | Read this snapshot in batch mode. | +| useSnapshotTimestamp | long | Millis since the Unix epoch; read the latest snapshot as of this timestamp in batch mode. | +| streamScanStrategy | enum | Stream scan strategy for this table. Supported values: `FROM_LATEST_SNAPSHOT` (default), `TABLE_SCAN_THEN_INCREMENTAL`, `FROM_EARLIEST_SNAPSHOT`, `FROM_SNAPSHOT_ID`, `FROM_SNAPSHOT_TIMESTAMP`. | +| splitSize | long | Target split size in bytes for this table (Iceberg `read.split.target-size`). | +| splitLookback | int | How many manifests to look back when combining splits (Iceberg `read.split.planning-lookback`). | +| splitOpenFileCost | long | Estimated cost to open a file, used when combining splits (Iceberg `read.split.openfile-cost`). | + ```hocon source { Iceberg { diff --git a/docs/en/connectors/source/LocalFile.md b/docs/en/connectors/source/LocalFile.md index 290c7d68224..395b7c8fbf1 100644 --- a/docs/en/connectors/source/LocalFile.md +++ b/docs/en/connectors/source/LocalFile.md @@ -84,6 +84,8 @@ For bounded Zeta jobs, restore resumes pending file splits after readers registe | null_format | string | no | - | | binary_chunk_size | int | no | 1024 | | binary_complete_file_mode | boolean | no | false | +| markdown_rag_metadata_enabled | boolean | no | false | +| pdf_rag_metadata_enabled | boolean | no | false | | discovery_mode | string | no | once | | scan_interval | string | no | 10S | | start_mode | string | no | earliest | @@ -481,7 +483,7 @@ The encoding of the file to read. This param will be parsed by `Charset.forName( ### null_format [string] -Only used when file_format_type is text. +Only used when file_format_type is `text` or `csv`. null_format to define which strings can be represented as null. e.g: `\N` diff --git a/docs/zh/connectors/common-options/source-common-options.md b/docs/zh/connectors/common-options/source-common-options.md index 243b43ace36..86c675e1fce 100644 --- a/docs/zh/connectors/common-options/source-common-options.md +++ b/docs/zh/connectors/common-options/source-common-options.md @@ -17,7 +17,7 @@ sidebar_position: 3 | plugin_output | String | 否 | - | 当未指定 `plugin_output` 时，此插件处理的数据将不会被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。<br/>当指定了 `plugin_output` 时，此插件处理的数据将被注册为可由其他插件直接访问的数据集 `(dataStream/dataset)`，或称为临时表 `(table)`。此处注册的数据集 `(dataStream/dataset)` 可通过指定 `plugin_input` 直接被其他插件访问。 | | parallelism | Int | 否 | - | 当未指定 `parallelism` 时，默认使用环境中的 `parallelism`。<br/>当指定了 `parallelism` 时，将覆盖环境中的 `parallelism` 设置。 | | metadata_datasource_id | String | 否 | - | 用于从元数据中心获取连接配置的数据源 ID。当指定此参数时，连接器将从外部元数据服务获取连接详细信息（如 URL、用户名、密码），而不是使用直接配置。详见 [元数据 SPI](../../introduction/concepts/metadata-spi.md)。 | -| dag-parsing.mode | Enum | 否 | SINGLENESS | 是否启用多表 source 的解析支持。`SINGLENESS` 表示每张表由独立的 source 和 sink 处理；`MULTIPLEXING` 表示多张表（例如分片表）由一个 source 和一个 sink 处理。 | +| dag-parsing.mode | Enum | 否 | SINGLENESS | 是否启用多表 source 的解析支持。`SINGLENESS` 表示每张表由独立的 source 和 sink 处理；`SHARDING` 表示多张表（例如分片表）由一个 source 和一个 sink 处理。`MULTIPLEX` 已废弃：表示多张表由一个 source 处理，每张表使用独立的 sink。 | # 重要提示 diff --git a/docs/zh/connectors/sink/Datahub.md b/docs/zh/connectors/sink/Datahub.md index 338ded00ae7..883c8bfa6d3 100644 --- a/docs/zh/connectors/sink/Datahub.md +++ b/docs/zh/connectors/sink/Datahub.md @@ -15,7 +15,7 @@ import ChangeLog from '../changelog/connector-datahub.md'; DataHub Sink 用于将 SeaTunnel 数据写入阿里云 DataHub。 该连接器支持单表写入和多表写入。多表写入时，可以在 `topic` 中使用 -`${table}` 这类占位符，将不同输入表的数据写入不同的 DataHub Topic。 +`${table_name}` 这类占位符，将不同输入表的数据写入不同的 DataHub Topic。 ## 关键特性 @@ -36,9 +36,10 @@ DataHub Sink 用于将 SeaTunnel 数据写入阿里云 DataHub。 | accessId | string | 是 | - | 访问 DataHub 使用的阿里云 Access ID。 | | accessKey | string | 是 | - | 访问 DataHub 使用的阿里云 Access Key。 | | project | string | 是 | - | DataHub 项目名称。 | -| topic | string | 是 | - | DataHub Topic 名称，多表写入时支持占位符。 | +| topic | string | 是 | - | DataHub Topic 名称，多表写入时支持 `${table_name}` 等占位符。 | | timeout | int | 否 | 3000 | 客户端连接最大超时时间，单位为毫秒。 | | retryTimes | int | 否 | 3 | 写入记录失败时的最大重试次数。 | +| multi_table_sink_replica | int | 否 | 1 | 多表 sink writer 的副本数，必须至少为 1。 | | common-options | config | 否 | - | Sink 插件通用参数，详见 [Sink 通用选项](../common-options/sink-common-options.md)。 | ### endpoint [string] @@ -59,8 +60,10 @@ DataHub 项目名称。 ### topic [string] -DataHub Topic 名称。多表写入时可以使用占位符，例如 `${table}`。 -`${table_name}` 仅作为已废弃的兼容别名保留，新任务建议使用 `${table}`。 +DataHub Topic 名称。多表写入时可以使用占位符，例如 `${table_name}`、`${database_name}`、 +`${schema_name}`（支持的占位符遵循 SeaTunnel `TablePlaceholder` 定义， +例如 `topic = "${table_name}"`）。注意 `${table}` **不是**有效占位符—— +Topic 会被创建为字面量字符串 `${table}`。 SeaTunnel 字段名需要和 DataHub Topic 中的字段名一致，因为 sink 会按照 Topic 结构里的字段名写入数据。 @@ -116,7 +119,7 @@ sink { ### 多表写入匹配的 Topic -当上游 source 提供多个表时，可以使用 `${table}` 占位符配置 `topic`，让每个输入表路由到同名的 topic。 +当上游 source 提供多个表时，可以使用 `${table_name}` 占位符配置 `topic`，让每个输入表路由到同名的 topic。 ```hocon env { @@ -159,7 +162,7 @@ sink { accessId = "your-access-id" accessKey = "your-access-key" project = "demo_project" - topic = "${table}" + topic = "${table_name}" timeout = 3000 retryTimes = 3 } @@ -174,7 +177,7 @@ sink { ### 多表写入是如何路由的？ -当上游 source 输出多张表时，把 `topic` 设置成包含 `${table}` 占位符的值（例如 `topic = "${table}"`），每张输入表就会被写到同名 DataHub topic。`${table_name}` 仍然作为已废弃的别名可以识别——新任务建议使用 `${table}`。该连接器不会自动创建目标 topic，请提前在 DataHub 项目中创建好，并保证其字段名与上游 SeaTunnel schema 一致。 +当上游 source 输出多张表时，把 `topic` 设置成包含 `${table_name}` 占位符的值（例如 `topic = "${table_name}"`），每张输入表就会被写到同名 DataHub topic。该连接器不会自动创建目标 topic，请提前在 DataHub 项目中创建好，并保证其字段名与上游 SeaTunnel schema 一致。 ### 为什么单表任务 `topic` 也是必填？ diff --git a/docs/zh/connectors/sink/HdfsFile.md b/docs/zh/connectors/sink/HdfsFile.md index 22c793881ef..508912258dc 100644 --- a/docs/zh/connectors/sink/HdfsFile.md +++ b/docs/zh/connectors/sink/HdfsFile.md @@ -60,8 +60,11 @@ import ChangeLog from '../changelog/connector-file-hadoop.md'; | filename_time_format | string | 否 | "yyyy.MM.dd" | 仅在 `custom_filename` 为 `true` 时使用。当 `file_name_expression` 参数中的格式为 `xxxx-${now}` 时，`filename_time_format` 可以指定路径的时间格式，默认值为 `yyyy.MM.dd`。常用的时间格式如下所示：[y:年,M:月,d:月中的一天,H:一天中的小时（0-23），m:小时中的分钟，s:分钟中的秒] | | file_format_type | string | 否 | "csv" | 我们支持以下文件类型：`text` `json` `csv` `orc` `parquet` `excel` `xml` `binary` `canal_json` `debezium_json` `maxwell_json`。请注意，最终文件名将以文件格式的后缀结束，文本文件的后缀是 `txt`。 | | filename_extension | string | 否 | - | 使用自定义的文件扩展名覆盖默认的文件扩展名。 例如：`.xml`, `.json`, `dat`, `.customtype` | -| field_delimiter | string | 否 | '\001' for text and ',' for csv | 仅在 file_format 为 text 时使用，数据行中列之间的分隔符。仅需要 `text` 文件格式。 | -| row_delimiter | string | 否 | "\n" | 仅在 file_format 为 text 时使用，文件中行之间的分隔符。仅需要 `text`、`csv`、`json` 文件格式。 | +| date_format | string | 否 | yyyy-MM-dd | 仅在 file_format 为 `text`、`csv`、`excel` 或 `xml` 时使用。日期类型格式，用于告知 connector 如何写出 date 值。支持格式（示例）：`yyyy-MM-dd`、`yyyy.MM.dd`、`yyyy/MM/dd` | +| datetime_format | string | 否 | yyyy-MM-dd HH:mm:ss | 仅在 file_format 为 `text`、`csv`、`excel` 或 `xml` 时使用。时间戳类型格式，用于告知 connector 如何写出 datetime 值。支持格式：`yyyy-MM-dd HH:mm:ss`、`yyyy.MM.dd HH:mm:ss`、`yyyy/MM/dd HH:mm:ss`、`yyyyMMddHHmmss` | +| time_format | string | 否 | HH:mm:ss | 仅在 file_format 为 `text`、`csv`、`excel` 或 `xml` 时使用。时间类型格式，用于告知 connector 如何写出 time 值。支持格式（示例）：`HH:mm:ss`、`HH:mm:ss.SSS` | +| field_delimiter | string | 否 | '\001' for text and ',' for csv | 仅在 file_format 为 `text` 和 `csv` 时使用，数据行中列之间的分隔符。 | +| row_delimiter | string | 否 | "\n" | 仅在 file_format 为 `text`、`csv`、`json` 时使用，文件中行之间的分隔符。 | | have_partition | boolean | 否 | false | 是否需要处理分区。 | | partition_by | array | 否 | - | 仅在 have_partition 为 true 时使用，根据选定的字段对数据进行分区。 | | partition_dir_expression | string | 否 | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | 仅在 have_partition 为 true 时使用，如果指定了 `partition_by`，我们将根据分区信息生成相应的分区目录，并将最终文件放置在分区目录中。默认 `partition_dir_expression` 为 `${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/`。`k0` 是第一个分区字段，`v0` 是第一个分区字段的值。 | diff --git a/docs/zh/connectors/sink/Hive.md b/docs/zh/connectors/sink/Hive.md index 1935c0655f3..0f2bf195bb0 100644 --- a/docs/zh/connectors/sink/Hive.md +++ b/docs/zh/connectors/sink/Hive.md @@ -162,7 +162,7 @@ Kerberos 的 keytab 文件路径 ### save_mode_create_template [字符串] -我们使用模板来自动创建 Hive 表，它将根据上游数据类型和模式类型创建相应的建表语句，默认模板可以根据情况进行修改。可用的模板变量：${database}, ${table}, ${rowtype_fields}, ${rowtype_partition_fields}, ${table_location}。 +我们使用模板来自动创建 Hive 表，它将根据上游数据类型和模式类型创建相应的建表语句，默认模板可以根据情况进行修改。可用的模板变量：${database}, ${table}, ${rowtype_fields}, ${rowtype_partition_fields}, ${table_location}, ${current_timestamp}。 **默认值**: 当未指定时，使用默认的 PARQUET 非分区表模板： ```sql diff --git a/docs/zh/connectors/sink/Iceberg.md b/docs/zh/connectors/sink/Iceberg.md index ebcecb58085..a580319b6b9 100644 --- a/docs/zh/connectors/sink/Iceberg.md +++ b/docs/zh/connectors/sink/Iceberg.md @@ -69,27 +69,27 @@ libfb303-xxx.jar | 名称 | 类型 | 是否必须 | 默认 | 描述 | |----------------------------------------|---------|------|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| catalog_name | string | no | default | 用户指定的目录名称，默认为`default` | -| namespace | string | no | default | backend catalog（元数据存储的后端目录）中 Iceberg 数据库的名称，默认为 `default` | -| table | string | no | - | backend catalog（元数据存储的后端目录）中 Iceberg 表的名称。不配置时使用上游表的表名 | -| iceberg.catalog.config | map | yes | - | 用于指定初始化 Iceberg Catalog 的属性，这些属性可以参考此文件：[CatalogProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java) | -| hadoop.config | map | no | - | 传递给 Hadoop 配置的属性 | -| iceberg.hadoop-conf-path | string | no | - | 指定`core-site.xml`、`hdfs-site.xml`、`hive-site.xml` 文件的加载路径 | -| case_sensitive | boolean | no | false | 列名匹配时是否区分大小写 | -| iceberg.table.write-props | map | no | - | 传递给 Iceberg 写入器初始化的属性，这些属性具有最高优先级，例如 `write.format.default`、`write.target-file-size-bytes` 等设置。具体参数可以参考：[TableProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableProperties.java)。 | -| iceberg.table.auto-create-props | map | no | - | Iceberg 自动建表时指定的配置 | -| iceberg.table.schema-evolution-enabled | boolean | no | false | 设置为 true 时，Iceberg 表可以在同步过程中支持 schema 变更 | -| iceberg.table.primary-keys | string | no | - | 用于标识表中一行数据的主键列列表，默认情况下以逗号分隔 | -| iceberg.table.partition-keys | string | no | - | 创建表时使用的分区字段列表，默认情况下以逗号分隔。多表场景可使用占位符 `${partition_keys}` | -| iceberg.table.upsert-mode-enabled | boolean | no | false | 设置为 `true` 以启用 upsert 模式，默认值为 `false` | -| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | schema 变更方式, 请参考下面的 `schema_save_mode` | -| data_save_mode | Enum | no | APPEND_DATA | 数据写入方式, 请参考下面的 `data_save_mode` | +| catalog_name | string | 否 | default | 用户指定的目录名称，默认为`default` | +| namespace | string | 否 | default | backend catalog（元数据存储的后端目录）中 Iceberg 数据库的名称，默认为 `default` | +| table | string | 否 | - | backend catalog（元数据存储的后端目录）中 Iceberg 表的名称。不配置时使用上游表的表名 | +| iceberg.catalog.config | map | 是 | - | 用于指定初始化 Iceberg Catalog 的属性，这些属性可以参考此文件：[CatalogProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java) | +| hadoop.config | map | 否 | - | 传递给 Hadoop 配置的属性 | +| iceberg.hadoop-conf-path | string | 否 | - | 指定`core-site.xml`、`hdfs-site.xml`、`hive-site.xml` 文件的加载路径 | +| case_sensitive | boolean | 否 | false | 列名匹配时是否区分大小写 | +| iceberg.table.write-props | map | 否 | - | 传递给 Iceberg 写入器初始化的属性，这些属性具有最高优先级，例如 `write.format.default`、`write.target-file-size-bytes` 等设置。具体参数可以参考：[TableProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableProperties.java)。 | +| iceberg.table.auto-create-props | map | 否 | - | Iceberg 自动建表时指定的配置 | +| iceberg.table.schema-evolution-enabled | boolean | 否 | false | 设置为 true 时，Iceberg 表可以在同步过程中支持 schema 变更 | +| iceberg.table.primary-keys | string | 否 | - | 用于标识表中一行数据的主键列列表，默认情况下以逗号分隔 | +| iceberg.table.partition-keys | string | 否 | - | 创建表时使用的分区字段列表，默认情况下以逗号分隔。多表场景可使用占位符 `${partition_keys}` | +| iceberg.table.upsert-mode-enabled | boolean | 否 | false | 设置为 `true` 以启用 upsert 模式，默认值为 `false` | +| schema_save_mode | Enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | schema 变更方式, 请参考下面的 `schema_save_mode` | +| data_save_mode | Enum | 否 | APPEND_DATA | 数据写入方式, 请参考下面的 `data_save_mode` | | custom_sql | string | 当 `data_save_mode` 为 `CUSTOM_PROCESSING` 时是 | - | `CUSTOM_PROCESSING` 数据写入方式使用的自定义 `delete` SQL，例如 `delete from ... where ...`。 | -| iceberg.table.commit-branch | string | no | - | 提交的默认分支 | -| multi_table_sink_replica | int | no | 1 | 多表写入模式下每张表对应的 Sink 写入并发数。一个作业写入多张 Iceberg 表，并且每张表都需要多个写入器时使用。 | -| krb5_path | string | no | /etc/krb5.conf | `krb5.conf` 文件的路径，用于 Kerberos 认证。 | -| kerberos_principal | string | no | - | Kerberos 认证的 principal。 | -| kerberos_keytab_path | string | no | - | Kerberos 认证的 keytab 文件路径。 | +| iceberg.table.commit-branch | string | 否 | - | 提交的默认分支 | +| multi_table_sink_replica | int | 否 | 1 | 多表写入模式下每张表对应的 Sink 写入并发数。一个作业写入多张 Iceberg 表，并且每张表都需要多个写入器时使用。 | +| krb5_path | string | 否 | /etc/krb5.conf | `krb5.conf` 文件的路径，用于 Kerberos 认证。 | +| kerberos_principal | string | 否 | - | Kerberos 认证的 principal。 | +| kerberos_keytab_path | string | 否 | - | Kerberos 认证的 keytab 文件路径。 | ## Sink 选项说明 @@ -253,6 +253,34 @@ sink { } ``` +### Glue Catalog + +```hocon +sink { + Iceberg { + catalog_name = "seatunnel_test" + iceberg.catalog.config = { + warehouse = "s3://your-bucket/warehouse/" + catalog-impl = "org.apache.iceberg.aws.glue.GlueCatalog" + io-impl = "org.apache.iceberg.aws.s3.S3FileIO" + client.region = "your-region" + } + namespace = "seatunnel_namespace" + table = "iceberg_sink_table" + iceberg.table.write-props = { + write.format.default = "parquet" + write.target-file-size-bytes = 536870912 + } + iceberg.table.primary-keys = "id" + iceberg.table.partition-keys = "f_datetime" + iceberg.table.upsert-mode-enabled = true + iceberg.table.schema-evolution-enabled = true + case_sensitive = true + } +} + +``` + ### AWS S3 Tables REST Catalog Amazon S3 表类数据存储服务提供针对分析工作负载进行优化的 S3 存储，其功能旨在持续提高查询性能并降低表的存储成本。S3 表类数据存储服务专为存储表数据而设计，例如每日购买交易、流传感器数据或广告展示次数。表数据以列和行表示数据，就像在数据库表中一样。 diff --git a/docs/zh/connectors/sink/LocalFile.md b/docs/zh/connectors/sink/LocalFile.md index b1f50400cef..f50d85fb663 100644 --- a/docs/zh/connectors/sink/LocalFile.md +++ b/docs/zh/connectors/sink/LocalFile.md @@ -60,6 +60,10 @@ import ChangeLog from '../changelog/connector-file-local.md'; | filename_time_format | string | 否 | "yyyy.MM.dd" | 仅在 custom_filename 为 true 时使用 | | file_format_type | string | 否 | "csv" | 文件格式类型 | | filename_extension | string | 否 | - | 使用自定义的文件扩展名覆盖默认的文件扩展名。 例如：`.xml`, `.json`, `dat`, `.customtype` | +| date_format | string | 否 | yyyy-MM-dd | 仅在 file_format_type 为 `text`、`csv`、`excel` 或 `xml` 时使用。日期类型格式，用于告知 connector 如何写出 date 值。支持格式（示例）：`yyyy-MM-dd`、`yyyy.MM.dd`、`yyyy/MM/dd`。 | +| datetime_format | string | 否 | yyyy-MM-dd HH:mm:ss | 仅在 file_format_type 为 `text`、`csv`、`excel` 或 `xml` 时使用。时间戳类型格式，用于告知 connector 如何写出 datetime 值。支持格式：`yyyy-MM-dd HH:mm:ss`、`yyyy.MM.dd HH:mm:ss`、`yyyy/MM/dd HH:mm:ss`、`yyyyMMddHHmmss`。 | +| time_format | string | 否 | HH:mm:ss | 仅在 file_format_type 为 `text`、`csv`、`excel` 或 `xml` 时使用。时间类型格式，用于告知 connector 如何写出 time 值。支持格式（示例）：`HH:mm:ss`、`HH:mm:ss.SSS`。 | +| multi_table_sink_replica | int | 否 | 1 | 多表 sink writer 的副本数，必须至少为 1。 | | field_delimiter | string | 否 | '\001' for text and ',' for csv | 仅在 file_format_type 为 text 和 csv 时使用 | | row_delimiter | string | 否 | "\n" | 仅在 file_format_type 为 `text`、`csv`、`json` 时使用 | | have_partition | boolean | 否 | false | 是否需要处理分区 | diff --git a/docs/zh/connectors/sink/MongoDB.md b/docs/zh/connectors/sink/MongoDB.md index cdd2c4fbca5..d55c04536dc 100644 --- a/docs/zh/connectors/sink/MongoDB.md +++ b/docs/zh/connectors/sink/MongoDB.md @@ -68,7 +68,6 @@ MongoDB Sink 连接器将 SeaTunnel 行记录写入 MongoDB 集合。每行记 | SeaTunnel 数据类型 | MongoDB BSON 类型 | |--------------------|-------------------| -| STRING | ObjectId | | STRING | String | | BOOLEAN | Boolean | | BINARY | Binary | @@ -90,6 +89,9 @@ MongoDB Sink 连接器将 SeaTunnel 行记录写入 MongoDB 集合。每行记 > 类型（BsonDateTime，毫秒精度）。SeaTunnel 的 `Date` 会存储为当天零点，`Timestamp` 则保留时分秒信息。 > 2. 在 SeaTunnel 中使用 `DECIMAL` 类型时，最大精度不能超过 34 位。建议使用 > `decimal(34, 18)` 以满足支持的精度与标度。 +> 3. SeaTunnel 的 `STRING` 值始终写为 BSON `String`，写入时不会转换为 `ObjectId`——即使是 `_id` +> 字段也一样。如需写入 Extended JSON 值，请将值编码为 `{"_value": <extended-json>}` +> （参见 source 的「数据类型映射」说明）。 ## Sink 参数说明 diff --git a/docs/zh/connectors/source/HdfsFile.md b/docs/zh/connectors/source/HdfsFile.md index 49b5d974934..d82a40f6f39 100644 --- a/docs/zh/connectors/source/HdfsFile.md +++ b/docs/zh/connectors/source/HdfsFile.md @@ -69,7 +69,7 @@ import ChangeLog from '../changelog/connector-file-hadoop.md'; | krb5_path | string | 否 | /etc/krb5.conf | kerberos 的 krb5 路径 | | kerberos_principal | string | 否 | - | kerberos 的主体 | | kerberos_keytab_path | string | 否 | - | kerberos 的 keytab 路径 | -| skip_header_row_number | long | 否 | 0 | 跳过前几行，但仅适用于 txt 和 csv。例如，设置如下：`skip_header_row_number = 2`。然后 Seatunnel 将跳过源文件的前 2 行 | +| skip_header_row_number | long | 否 | 0 | 跳过 text、CSV 或 Excel 文件的前几行。例如，设置如下：`skip_header_row_number = 2`。然后 Seatunnel 将跳过源文件的前 2 行 | | schema | config | 否 | - | 上游数据的 schema 字段。更多详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。 | | sheet_name | string | 否 | - | 读取工作簿的工作表，仅在 file_format 为 excel 时使用。 | | excel_engine | string | 否 | POI | 仅在 `file_format` 为 excel 时使用。支持的引擎包括 `POI` 和 `EasyExcel`。 | @@ -82,7 +82,10 @@ import ChangeLog from '../changelog/connector-file-hadoop.md'; | compress_codec | string | 否 | none | 文件的压缩编解码器 | | archive_compress_codec | string | 否 | none | 文件使用的归档压缩编解码器。支持：`none` `zip` `tar` `tar.gz` `gz`。 | | encoding | string | 否 | UTF-8 | | -| null_format | string | 否 | - | 仅在 file_format_type 为 text 时使用。null_format 定义哪些字符串可以表示为 null。例如：`\N` | +| null_format | string | 否 | - | 仅在 file_format_type 为 `text` 或 `csv` 时使用。null_format 定义哪些字符串可以表示为 null。例如：`\N` | +| excel_use_header_line | boolean | 否 | false | 使用 Excel 第一行作为表头按名称映射字段。仅在 file_format_type 为 `excel` 时使用。 | +| markdown_rag_metadata_enabled | boolean | 否 | false | 读取 markdown 文件时是否追加面向 RAG 的元数据列。仅在 file_format_type 为 `markdown` 时有效。 | +| pdf_rag_metadata_enabled | boolean | 否 | false | 读取 PDF 文件时是否追加面向 RAG 的元数据列。仅在 file_format_type 为 `pdf` 时有效。 | | binary_chunk_size | int | 否 | 1024 | 仅在 file_format_type 为 binary 时使用。读取二进制文件的块大小（以字节为单位）。默认为 1024 字节。较大的值可能会提高大文件的性能，但会使用更多内存。 | | binary_complete_file_mode | boolean | 否 | false | 仅在 file_format_type 为 binary 时使用。是否将完整文件作为单个块读取，而不是分割成块。启用时，整个文件内容将一次性读入内存。默认为 false。 | | discovery_mode | string | 否 | once | 文件发现模式，支持：`once`（默认）、`continuous`。continuous 模式下将周期性扫描并处理新/变更文件（无界）。当前实现中 continuous 需要配合 `sync_mode=update`（仅 binary）使用，以避免重复传输。 | diff --git a/docs/zh/connectors/source/Hive.md b/docs/zh/connectors/source/Hive.md index 1cd61c0ab93..f76f24913d1 100644 --- a/docs/zh/connectors/source/Hive.md +++ b/docs/zh/connectors/source/Hive.md @@ -36,7 +36,7 @@ import ChangeLog from '../changelog/connector-hive.md'; | 名称 | 类型 | 必需 | 默认值 | 描述 | |-----------------------|--------|------|---------|------| | table_name | string | 否 | 单表模式必填 | 目标 Hive 表名，格式为 `db1.table1`。当 `use_regex = true` 时，该字段使用 `数据库正则.表正则` 匹配多张表。 | -| table_list | array | 否 | - | 多表读取时使用的 Hive 表配置列表，每项可覆盖根配置中的任意选项。若两者同时设置，`table_list` 优先。 | +| table_list | array | 否 | - | 多表读取时使用的 Hive 表配置列表。每一项都是独立的配置：`table_name`、`metastore_uri` 以及认证/Hadoop 等选项**不会**从根配置继承，必须在每一项中重复填写（设置 `table_list` 后，根配置中 `table_list` 之外的 connector 选项会被忽略；`plugin_output`、`parallelism` 等框架级选项仍然生效）。若两者同时设置，`table_list` 优先。 | | tables_configs | array | 否 | 已废弃，请使用 `table_list` | 已废弃的多表配置列表。Hive 是结构化数据源，新作业请使用 `table_list`。仅为向后兼容保留。 | | use_regex | boolean| 否 | false | 将 `table_name` 视为正则表达式以匹配多张表。在根级别以及 `table_list` / `tables_configs` 的每一项中均可使用。 | | metastore_uri | string | 否 | 单表模式必填 | Hive metastore URI。多个以逗号分隔的值可启用 HA 故障转移，空白字符会被忽略。 | @@ -50,6 +50,8 @@ import ChangeLog from '../changelog/connector-hive.md'; | remote_user | string | 否 | - | 未启用 Kerberos 时连接 HDFS / Hive 存储使用的 Hadoop 远程用户名。 | | read_partitions | list | 否 | - | 限制读取的分区子集。所有条目的目录层级深度必须一致。 | | read_columns | list | 否 | - | 列投影列表。仅读取列出的字段。 | +| null_format | string | 否 | - | text 表中表示 null 值的自定义字符串。设置后会覆盖 Hive 表参数 `serialization.null.format`（其默认值为 `\N`）。 | +| parse_partition_from_path | boolean | 否 | true | 当 schema 来自 Hive metastore 时，是否把分区键列（STRING 类型）追加到投影 schema 中。 | | compress_codec | string | 否 | none | text 表的压缩编解码器，支持 `lzo` 和 `none`。Parquet / ORC 会自动识别压缩。 | | common-options | | 否 | - | Source 插件通用参数，详情请参见 [Source Common Options](../common-options/source-common-options.md)。 | @@ -61,7 +63,7 @@ import ChangeLog from '../changelog/connector-hive.md'; ### table_list [array] -多表读取配置列表。每个元素可以包含 `table_name`、`metastore_uri`、`use_regex`、`read_partitions`、`read_columns`，以及与根配置相同的认证和 Hadoop 配置。 +多表读取配置列表。每个元素是独立的配置，可以包含 `table_name`、`metastore_uri`、`use_regex`、`read_partitions`、`read_columns` 以及认证和 Hadoop 配置——但不会从根配置继承任何选项，因此 `metastore_uri` 等必填项必须在每一项中重复填写。 ### tables_configs [array] @@ -69,7 +71,7 @@ import ChangeLog from '../changelog/connector-hive.md'; ### use_regex [boolean] -是否将 `table_name` 视为正则表达式进行匹配。开启后，`table_name` 可用于整库/多表同步；同样也支持在 `table_list` / `tables_configs` 的每个表配置里单独开启。 +是否将 `table_name` 视为正则表达式进行匹配。开启后，`table_name` 可用于整库/多表同步；同样也支持在 `table_list` / `tables_configs` 的每个表配置里单独开启。正则发现依赖可用的 Hive metastore：必须配置 `metastore_uri`（根级别或对应的 `table_list` 项中），否则作业会失败。 语法说明： - 点号（`.`）被视为数据库与表之间的分隔符（Hive 仅支持 `database.table`）。 diff --git a/docs/zh/connectors/source/Iceberg.md b/docs/zh/connectors/source/Iceberg.md index 56edd62c4be..67ff0e93761 100644 --- a/docs/zh/connectors/source/Iceberg.md +++ b/docs/zh/connectors/source/Iceberg.md @@ -84,7 +84,7 @@ libfb303-xxx.jar | catalog_name | string | 否 | default | 用户指定的目录名称。 [...] | namespace | string | 否 | default | 后端目录中的 iceberg 数据库名称。 [...] | table | string | 否 | - | 后端目录中的 Iceberg 表名称。`table` 和 `table_list` 必须二选一配置，不能同时配置。 [...] -| table_list | list | 否 | - | 后端目录中的 Iceberg 表列表。`table` 和 `table_list` 必须二选一配置，不能同时配置。每一项都可以配置 `table`、`query` 以及该表自己的快照/流式扫描参数。 [...] +| table_list | list | 否 | - | 后端目录中的 Iceberg 表列表。`table` 和 `table_list` 必须二选一配置，不能同时配置。每一项都可以配置 `table`、`query` 以及该表自己的快照/流式扫描参数。注意：每一项的键使用驼峰命名（例如 `startSnapshotId`、`streamScanStrategy`），详见[多表读取](#多表读取)。 [...] | iceberg.catalog.config | map | 是 | - | 指定初始化 Iceberg 目录的属性，可以在此文件中引用：[CatalogProperties.java](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java) [...] | hadoop.config | map | 否 | - | 传递给 Hadoop 配置的属性 [...] | iceberg.hadoop-conf-path | string | 否 | - | 为 'core-site.xml'、'hdfs-site.xml'、'hive-site.xml' 文件指定的加载路径。 [...] @@ -99,9 +99,9 @@ libfb303-xxx.jar | increment.scan-interval | long | 否 | 2000 | 增量扫描的间隔（毫秒），必须为正数 [...] | common-options | | 否 | - | 源插件通用参数，请参考 [源通用选项](../common-options/source-common-options.md) 详见。 [...] | query | String | 否 | - | 用于过滤或投影 Iceberg 数据的 select DML。这里使用固定占位表名 `table`，不要写真实表名，也不支持表别名。例如：`select * from table where f1 > 100`、`select fn from table where f1 > 100`。当前对 LIKE 语法的支持有限：LIKE 模式不应以 `%` 开头，例如 `select f1 from table where f2 like 'tom%'`。 [...] -| krb5_path | string | no | /etc/krb5.conf | `krb5.conf` 文件的路径，用于 Kerberos 认证。 | -| kerberos_principal | string | no | - | Kerberos 认证的 principal。 | -| kerberos_keytab_path | string | no | - | Kerberos 认证的 keytab 文件路径。 | +| krb5_path | string | 否 | /etc/krb5.conf | `krb5.conf` 文件的路径，用于 Kerberos 认证。 | +| kerberos_principal | string | 否 | - | Kerberos 认证的 principal。 | +| kerberos_keytab_path | string | 否 | - | Kerberos 认证的 keytab 文件路径。 | ## 源选项说明 @@ -155,6 +155,23 @@ sink { 一个 Iceberg Source 需要读取多张表时使用 `table_list`，不要同时配置 `table`。 +`table_list` 的每一项按独立配置解析，支持以下每表级别的键。注意：与根级别选项不同，这些键使用驼峰命名（camelCase）： + +| 每表键 | 类型 | 描述 | +|-------------------------|--------|----------------------------------------------------------------------------------------| +| namespace | string | 不生效：每一项都会被根级别 `namespace` 覆盖。如需读取其他数据库，请在 `table` 中使用限定名 `db.table`。 | +| table | string | Iceberg 表名。 | +| query | string | 该表的过滤 query，含义与根级别 `query` 选项相同。 | +| startSnapshotTimestamp | long | 自 Unix 纪元以来的毫秒时间戳；从该时间点最近的快照开始扫描变更。 | +| startSnapshotId | long | 从指定快照（不包含）开始扫描变更。 | +| endSnapshotId | long | 扫描变更直到指定快照（包含）。 | +| useSnapshotId | long | 批量模式下读取指定快照。 | +| useSnapshotTimestamp | long | 自 Unix 纪元以来的毫秒时间戳；批量模式下读取该时间点最近的快照。 | +| streamScanStrategy | enum | 该表的流式扫描策略。可选值：`FROM_LATEST_SNAPSHOT`（默认）、`TABLE_SCAN_THEN_INCREMENTAL`、`FROM_EARLIEST_SNAPSHOT`、`FROM_SNAPSHOT_ID`、`FROM_SNAPSHOT_TIMESTAMP`。 | +| splitSize | long | 该表的目标 split 大小（字节），对应 Iceberg `read.split.target-size`。 | +| splitLookback | int | 合并 split 时向前查看的 manifest 数量，对应 Iceberg `read.split.planning-lookback`。 | +| splitOpenFileCost | long | 打开一个文件的估算开销，用于合并 split，对应 Iceberg `read.split.openfile-cost`。 | + ```hocon source { Iceberg { diff --git a/docs/zh/connectors/source/LocalFile.md b/docs/zh/connectors/source/LocalFile.md index 97c146bbe53..18e14f9b9b6 100644 --- a/docs/zh/connectors/source/LocalFile.md +++ b/docs/zh/connectors/source/LocalFile.md @@ -84,6 +84,8 @@ import ChangeLog from '../changelog/connector-file-local.md'; | null_format | string | 否 | - | | binary_chunk_size | int | 否 | 1024 | | binary_complete_file_mode | boolean | 否 | false | +| markdown_rag_metadata_enabled | boolean | 否 | false | +| pdf_rag_metadata_enabled | boolean | 否 | false | | discovery_mode | string | 否 | once | | scan_interval | string | 否 | 10S | | start_mode | string | 否 | earliest | @@ -480,7 +482,7 @@ abc.* ### null_format [string] -仅在 file_format_type 为 text 时使用。 +仅在 file_format_type 为 `text` 或 `csv` 时使用。 null_format 定义哪些字符串可以表示为 null。 例如：`\N`
