This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13163-fae9ffa96cc9d0e3e0da90c3909b0ee173fd4ddd in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit f3f9baf39d4fecb901103cbf187c54119aeb3f3e Author: Jast <[email protected]> AuthorDate: Sat Nov 7 06:35:41 2026 +0000 [Docs] Fix JDBC dialect, transform multi-table and Zeta doc inaccuracies verified against source code (#13163) Co-authored-by: zhangshenghang <[email protected]> --- docs/en/connectors/sink/Aerospike.md | 6 +++--- docs/en/connectors/sink/GcsFile.md | 4 ++++ docs/en/connectors/source/HiveJdbc.md | 2 +- docs/en/connectors/source/Oracle.md | 9 +++++---- docs/en/connectors/source/Redshift.md | 2 +- docs/en/connectors/source/SqlServer.md | 2 +- docs/en/engines/zeta/benchmark.md | 2 +- docs/en/engines/zeta/download-seatunnel.md | 4 ++-- docs/en/engines/zeta/stain-trace.md | 4 ++-- docs/en/transforms/data-validator.md | 5 +++++ docs/en/transforms/define-sink-type.md | 3 +++ docs/en/transforms/dynamic-compile.md | 11 +++++++++++ docs/en/transforms/field-rename.md | 3 +++ docs/en/transforms/replace.md | 15 +++++++++++++++ docs/en/transforms/rowkind-extractor.md | 7 +++++++ docs/en/transforms/table-filter.md | 3 +++ docs/en/transforms/table-rename.md | 3 +++ docs/en/transforms/text-chunk.md | 14 ++++++++++++++ docs/zh/connectors/sink/Aerospike.md | 6 +++--- docs/zh/connectors/sink/GcsFile.md | 4 ++++ docs/zh/connectors/source/HiveJdbc.md | 4 ++-- docs/zh/connectors/source/Oracle.md | 9 +++++---- docs/zh/connectors/source/Redshift.md | 2 +- docs/zh/connectors/source/SqlServer.md | 2 +- docs/zh/engines/zeta/benchmark.md | 2 +- docs/zh/engines/zeta/download-seatunnel.md | 4 ++-- docs/zh/engines/zeta/stain-trace.md | 4 ++-- docs/zh/transforms/data-validator.md | 5 +++++ docs/zh/transforms/define-sink-type.md | 3 +++ docs/zh/transforms/dynamic-compile.md | 7 +++++++ docs/zh/transforms/field-rename.md | 3 +++ docs/zh/transforms/replace.md | 15 +++++++++++++++ docs/zh/transforms/rowkind-extractor.md | 7 +++++++ docs/zh/transforms/table-filter.md | 3 +++ docs/zh/transforms/table-rename.md | 3 +++ docs/zh/transforms/text-chunk.md | 10 ++++++++++ 36 files changed, 161 insertions(+), 31 deletions(-) diff --git a/docs/en/connectors/sink/Aerospike.md b/docs/en/connectors/sink/Aerospike.md index a1f25836d44..dee1828fc56 100644 --- a/docs/en/connectors/sink/Aerospike.md +++ b/docs/en/connectors/sink/Aerospike.md @@ -71,9 +71,9 @@ Note: ### data_format Options -- **map**: Store all non-key fields as a map in `bin_name` -- **string**: Store all non-key fields as a JSON string in `bin_name` -- **kv**: Store each non-key field as a separate bin. `bin_name` is not used +- **map**: Store the selected fields as a map in `bin_name` +- **string**: Store the selected fields as a JSON string in `bin_name` +- **kv**: Store each selected field as a separate bin. `bin_name` is not used When `data_format` is `map` or `string`, configure `bin_name`; otherwise the writer does not know which Aerospike bin should receive the packed row data. The `key` field must be present in the input schema because it is used as the Aerospike record key for every write. diff --git a/docs/en/connectors/sink/GcsFile.md b/docs/en/connectors/sink/GcsFile.md index e7fc89b1b91..7b3675f6221 100644 --- a/docs/en/connectors/sink/GcsFile.md +++ b/docs/en/connectors/sink/GcsFile.md @@ -88,6 +88,10 @@ The explicit `service_account_key_file` option takes precedence over the corresp | encoding | string | conditional | `UTF-8` | Encoding for text, CSV, JSON, and XML. | | enable_header_write | boolean | conditional | `false` | Writes a header for text or CSV output. | | compress_codec | string | no | `none` | Compression codec supported by the selected format. | +| date_format | string | no | `yyyy-MM-dd` | Date type format used when writing date values with `text`, `csv`, `excel`, or `xml` output. | +| datetime_format | string | no | `yyyy-MM-dd HH:mm:ss` | Datetime type format used when writing datetime values with `text`, `csv`, `excel`, or `xml` output. | +| time_format | string | no | `HH:mm:ss` | Time type format used when writing time values with `text`, `csv`, `excel`, or `xml` output. | +| multi_table_sink_replica | int | no | `1` | Number of writer replicas for each table in multi-table sink mode. Must be at least 1. | | common-options | | no | - | See [Sink Common Options](../common-options/sink-common-options.md). | The sink also accepts the format-specific file sink options documented by the corresponding diff --git a/docs/en/connectors/source/HiveJdbc.md b/docs/en/connectors/source/HiveJdbc.md index 4838c6d2b40..6e01dc7b720 100644 --- a/docs/en/connectors/source/HiveJdbc.md +++ b/docs/en/connectors/source/HiveJdbc.md @@ -78,7 +78,7 @@ authentication is supported. | connection_check_timeout_sec | Int | No | 30 | The time, in seconds, to wait for the database operation used to validate the connection to complete. | | socket_timeout_ms | Int | No | 86400000 | Socket timeout in milliseconds for reading data from the server. Set to `0` for no timeout. Tested with Hive 3.2.0+. For earlier versions, not yet verified. | | connect_timeout_ms | Int | No | 86400000 | Connection timeout in milliseconds for establishing a connection to the server. Set to `0` for no timeout. Tested with Hive 3.2.0+. For earlier versions, not yet verified. | -| partition_column | String | No | - | The column name for parallelism partition. Only supports numeric primary key columns. | +| partition_column | String | No | - | Column used to split data for parallel reading. Supports numeric, string and date columns; only one column can be configured. | | partition_lower_bound | String | No | - | The `partition_column` minimum value for the scan. If not set, SeaTunnel queries the database for the minimum value. | | partition_upper_bound | String | No | - | The `partition_column` maximum value for the scan. If not set, SeaTunnel queries the database for the maximum value. | | partition_num | Int | No | 10 | Number of partitions. Only positive integers are supported. Default value is 10. | diff --git a/docs/en/connectors/source/Oracle.md b/docs/en/connectors/source/Oracle.md index dff20ed4b83..8854b9e676d 100644 --- a/docs/en/connectors/source/Oracle.md +++ b/docs/en/connectors/source/Oracle.md @@ -74,7 +74,7 @@ Read external data source data through JDBC. | password | String | No | - | Connection instance password | | query | String | No | - | Query statement. Required when neither `table_path` nor `table_list` is configured. | | connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete | -| partition_column | String | No | - | The column name for parallelism's partition, only support numeric type,Only support numeric type primary key, and only can config one column. | +| partition_column | String | No | - | Column used to split data for parallel reading. Supports numeric and string columns; Oracle `DATE` and `TIMESTAMP` columns are read as `TIMESTAMP` and cannot be used. Only one column can be configured. | | partition_lower_bound | String | No | - | The partition_column min value for scan, if not set SeaTunnel will query database get min value. | | partition_upper_bound | String | No | - | The partition_column max value for scan, if not set SeaTunnel will query database get max value. | | partition_num | Int | No | 10 | The number of partition count, only support positive integer. default value is 10 | @@ -129,7 +129,8 @@ The JDBC Source connector supports parallel reading of data from tables. SeaTunn **Supported split data type:** * String * Number(int, bigint, decimal, ...) -* Date + +Oracle `DATE` and `TIMESTAMP` columns are mapped to SeaTunnel `TIMESTAMP` (see the type mapping above), which is not a supported split type. ### Options Related To Split @@ -336,7 +337,7 @@ sink { ### Streaming With Incremental ID Range -The Oracle Source connector is batch-oriented. Setting `job.mode = "STREAMING"` only enables checkpointing so the job can resume on failure; the source itself is bounded and reads the configured `[partition_lower_bound, partition_upper_bound)` range exactly once per job run. To pick up new rows repeatedly you must externally resubmit the job (for example on a schedule, with a sliding window), or use Oracle-CDC for continuous change capture. +The Oracle Source connector is batch-oriented. Setting `job.mode = "STREAMING"` only enables checkpointing so the job can resume on failure; the source itself is bounded and reads the configured `[partition_lower_bound, partition_upper_bound]` range exactly once per job run. To pick up new rows repeatedly you must externally resubmit the job (for example on a schedule, with a sliding window), or use Oracle-CDC for continuous change capture. ```hocon env { @@ -351,7 +352,7 @@ source { driver = "oracle.jdbc.OracleDriver" username = "root" password = "123456" - query = "SELECT * FROM ORDERS WHERE ORDER_ID >= ? AND ORDER_ID < ?" + query = "SELECT * FROM ORDERS" partition_column = "ORDER_ID" partition_lower_bound = 1 partition_upper_bound = 1000000 diff --git a/docs/en/connectors/source/Redshift.md b/docs/en/connectors/source/Redshift.md index 134fbe4b0fb..d78116aaeb6 100644 --- a/docs/en/connectors/source/Redshift.md +++ b/docs/en/connectors/source/Redshift.md @@ -56,7 +56,7 @@ reads via `table_list` are supported. | table_path | String | No | - | The fully-qualified table to read, for example `public.table2`. Useful as a shortcut for a single-table read. | | table_list | Array | No | - | List of tables to read. Each entry can override `table_path` and `query`. Enables multi-table reading and auto-split. | | connection_check_timeout_sec | Int | No | 30 | The time, in seconds, to wait for the database operation used to validate the connection to complete. | -| partition_column | String | No | - | The column name used to split the data for parallel reading. Only numeric primary key columns are supported. | +| partition_column | String | No | - | The column name used to split the data for parallel reading. Supports numeric, string and date columns; only one column can be configured. | | partition_lower_bound | String | No | - | The `partition_column` minimum value for the scan. If not set, SeaTunnel queries the database for the minimum value. | | partition_upper_bound | String | No | - | The `partition_column` maximum value for the scan. If not set, SeaTunnel queries the database for the maximum value. | | partition_num | Int | No | 10 | Number of partitions. Only positive integers are supported. Default value is 10. | diff --git a/docs/en/connectors/source/SqlServer.md b/docs/en/connectors/source/SqlServer.md index 486f8eeaa36..16268dd6fc5 100644 --- a/docs/en/connectors/source/SqlServer.md +++ b/docs/en/connectors/source/SqlServer.md @@ -82,7 +82,7 @@ parallel SeaTunnel split. | password | String | No | - | Connection instance password [...] | query | String | No | - | Query statement. Required when neither `table_path` nor `table_list` is configured. [...] | connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete [...] -| partition_column | String | No | - | The column name for parallelism's partition, only support numeric type. [...] +| partition_column | String | No | - | Column used to split data for parallel reading. Supports numeric, string and date columns; only one column can be configured. [...] | partition_lower_bound | String | No | - | The partition_column min value for scan, if not set SeaTunnel will query database get min value. [...] | partition_upper_bound | String | No | - | The partition_column max value for scan, if not set SeaTunnel will query database get max value. [...] | partition_num | Int | No | 10 | The number of partition count, only support positive integer. default value is 10 [...] diff --git a/docs/en/engines/zeta/benchmark.md b/docs/en/engines/zeta/benchmark.md index c3878159535..f67efac238e 100644 --- a/docs/en/engines/zeta/benchmark.md +++ b/docs/en/engines/zeta/benchmark.md @@ -255,7 +255,7 @@ over time: Baseline → Candidate → Candidate → Baseline ``` -Java 8 and Java 11 run this comparison separately, and the report summarizes two runs of each +Java 11 and Java 17 run this comparison separately, and the report summarizes two runs of each revision. After the run, confirm that the job reached JMH measurement and verify the SHAs, method, and diff --git a/docs/en/engines/zeta/download-seatunnel.md b/docs/en/engines/zeta/download-seatunnel.md index eca76398088..99fffa01e73 100644 --- a/docs/en/engines/zeta/download-seatunnel.md +++ b/docs/en/engines/zeta/download-seatunnel.md @@ -39,7 +39,7 @@ If you need a specific connector version, taking 3.1.0 as an example, you need t sh bin/install-plugin.sh 3.1.0 ``` -Usually you don't need all the connector plugins, so you can specify the plugins you need through configuring `config/plugin_config`, for example, if you only need the `connector-console` plugin, then you can modify the plugin.properties configuration file as follows. +Usually you don't need all the connector plugins, so you can specify the plugins you need through configuring `config/plugin_config`, for example, if you only need the `connector-console` plugin, then you can modify the plugin_config configuration file as follows. ```plugin_config --seatunnel-connectors-- @@ -56,7 +56,7 @@ connector-console --end-- ``` -You can find all supported connectors and the corresponding plugin_config configuration names under `${SEATUNNEL_HOME}/connectors/plugins-mapping.properties`. +You can find all supported connectors and the corresponding plugin_config configuration names under `${SEATUNNEL_HOME}/connectors/plugin-mapping.properties`. :::tip Tip diff --git a/docs/en/engines/zeta/stain-trace.md b/docs/en/engines/zeta/stain-trace.md index 91686b9be2a..deee540084b 100644 --- a/docs/en/engines/zeta/stain-trace.md +++ b/docs/en/engines/zeta/stain-trace.md @@ -8,7 +8,7 @@ StainTrace is SeaTunnel's data lineage and end-to-end performance tracing system - **Framework-level Implementation**: Works with all Connectors automatically, no connector code changes needed - **6 Basic Stages**: SOURCE_EMIT, QUEUE_IN, QUEUE_OUT, TRANSFORM_IN, TRANSFORM_OUT, SINK_WRITE_DONE (all instrumented; temporal order depends on pipeline topology) -- **Extended Stages**: 40+ fine-grained stage codes defined for future instrumentation (**not yet active — will not appear in trace files**) +- **Extended Stages**: 34 fine-grained stage codes defined for future instrumentation (**not yet active — will not appear in trace files**) - **Local File Storage**: Zero dependencies, OTLP JSON Lines format - **Performance Optimized**: < 2% overhead with proper sampling configuration @@ -70,7 +70,7 @@ seatunnel: engine: stain-trace-enabled: true stain-trace-sample-rate: 100000 # Sample 1 out of 100k records - stain-trace-file-base-path: /data/seatunnel/traces # Required for local trace files; no default + stain-trace-file-base-path: /tmp/seatunnel/traces # Required for local trace files; no default ``` ### 2. Enable in Job diff --git a/docs/en/transforms/data-validator.md b/docs/en/transforms/data-validator.md index e1cef97a01c..613a1f38c38 100644 --- a/docs/en/transforms/data-validator.md +++ b/docs/en/transforms/data-validator.md @@ -13,6 +13,9 @@ The DataValidator transform validates field values according to configured rules | row_error_handle_way| enum | no | FAIL | | row_error_handle_way.error_table | string | no | | | field_rules | array | yes (single-table mode) | | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | +| rule_match_mode | string | no | `FIRST_MATCH` | ### row_error_handle_way [enum] @@ -61,6 +64,8 @@ Array of field validation rules. Each rule defines validation criteria for a spe A single-table config must set `field_rules`. With `table_transform`, every entry declares its own `field_rules`; the transform level one is optional. +The `table_match_regex` option selects the tables the transform level rule applies to (default `.*`), `table_transform` holds complete per-table rules (`table_path` plus `field_rules`), and `rule_match_mode` (`FIRST_MATCH` or `ALL_MATCH`) controls how duplicate `table_path` entries are evaluated. See [Multi-Table Transform](transform-multi-table.md). + #### Field Rule Structure Each field rule contains: diff --git a/docs/en/transforms/define-sink-type.md b/docs/en/transforms/define-sink-type.md index 7f81c810d3b..75f7335fc9f 100644 --- a/docs/en/transforms/define-sink-type.md +++ b/docs/en/transforms/define-sink-type.md @@ -11,6 +11,9 @@ Used to define the storage type of sink field. This is effective when the savemo | name | type | required | default value | Description | |:-------:|---------------------------|----------|---------------|------------------------------------------------------------------------| | columns | list<map<string, string>> | yes | | The columns to be defined, the name and type of the column must be set | +| table_match_regex | string | no | `.*` | Regex matched against the upstream table path. The transform level rule only applies to matching tables. | +| table_transform | array | no | - | Per-table rules, see [Multi-Table Transform](transform-multi-table.md). Each entry contains `table_path` and the `columns` of that table. | +| rule_match_mode | string | no | `FIRST_MATCH` | How multiple `table_transform` entries with the same `table_path` are evaluated. | ## Examples diff --git a/docs/en/transforms/dynamic-compile.md b/docs/en/transforms/dynamic-compile.md index 72547259f2a..3a0ec33a036 100644 --- a/docs/en/transforms/dynamic-compile.md +++ b/docs/en/transforms/dynamic-compile.md @@ -22,6 +22,9 @@ If the conversion is too complex, it may affect performance | compile_language | Enum | yes | | | compile_pattern | Enum | no | SOURCE_CODE | | absolute_path | string | no | | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | +| rule_match_mode | string | no | `FIRST_MATCH` | ### common options [string] @@ -48,6 +51,14 @@ The absolute path of the Java, Groovy, or Scala source file on the server The source code. +### table_match_regex / table_transform / rule_match_mode + +Multi-table options. When the upstream source outputs several tables, one DynamicCompile transform +can handle all of them: `table_match_regex` selects the tables the transform level rule applies to +(default `.*`), `table_transform` holds complete per-table rules (`table_path` plus the options +above), and `rule_match_mode` (`FIRST_MATCH` or `ALL_MATCH`) controls how duplicate `table_path` +entries are evaluated. See [Multi-Table Transform](transform-multi-table.md). + #### Details about the source code In the source code, you must implement two method: diff --git a/docs/en/transforms/field-rename.md b/docs/en/transforms/field-rename.md index 7682b03c3d4..07f4cf2c076 100644 --- a/docs/en/transforms/field-rename.md +++ b/docs/en/transforms/field-rename.md @@ -15,6 +15,9 @@ FieldRename transform plugin for rename field name. | suffix | string | no | | The suffix to be added to the field name | | replacements_with_regex | array | no | | The array of replacement rules. Each rule is a map with `replace_from`, `replace_to`, and optional `is_regex` (default `true`). When `is_regex=false`, `replace_from` is treated as an exact field name (full match). | | specific | array | no | | Specific rename rules. Each rule is a map with `field_name` and `target_name`. When matched, it will rename the field directly and skip other rename rules. | +| table_match_regex | string | no | `.*` | Regex matched against the upstream table path. The transform level rule only applies to matching tables. | +| table_transform | array | no | - | Per-table rules, see [Multi-Table Transform](transform-multi-table.md). | +| rule_match_mode | string | no | `FIRST_MATCH` | How multiple `table_transform` entries with the same `table_path` are evaluated. | ## Examples diff --git a/docs/en/transforms/replace.md b/docs/en/transforms/replace.md index e50bf746b0e..aba22afa306 100644 --- a/docs/en/transforms/replace.md +++ b/docs/en/transforms/replace.md @@ -15,6 +15,9 @@ Examines string value in given fields and replaces substring of the string value | replacement | string | yes | - | | is_regex | boolean | no | false | | replace_first | boolean | no | false | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | +| rule_match_mode | string | no | `FIRST_MATCH` | ### replace_fields [array] @@ -44,6 +47,18 @@ Use regex for string match Whether replace the first match string. Only used when `is_regex = true`. +### table_match_regex [string] + +Regex matched against the upstream table path. The transform level rule (the options above outside of `table_transform`) only applies to tables whose path matches this regex. The default `.*` matches every table. + +### table_transform [array] + +A list of per-table rules. Each entry must contain `table_path` in the form `databaseName[.schemaName].tableName` (matched exactly) and a complete Replace rule for that table (`replace_fields`, `pattern`, `replacement`). Tables with an entry are not affected by the transform level rule. + +### rule_match_mode [string] + +Controls what happens when several `table_transform` entries have the same `table_path`: `FIRST_MATCH` applies only the first matching entry, `ALL_MATCH` applies all matching entries in order. If not set, duplicate `table_path` entries are rejected. + ### common options [string] Transform plugin common parameters, please refer to [Transform Plugin](common-options/common-options.md) for details diff --git a/docs/en/transforms/rowkind-extractor.md b/docs/en/transforms/rowkind-extractor.md index 40714bf20d4..c94b1e53e84 100644 --- a/docs/en/transforms/rowkind-extractor.md +++ b/docs/en/transforms/rowkind-extractor.md @@ -42,6 +42,9 @@ With `transform_type = FULL`, the field stores the full name instead: `row_kind= |-------------------|--------|----------|---------------|-------------| | custom_field_name | string | no | row_kind | The name of the new field used to store the original RowKind information | | transform_type | enum | no | SHORT | The output format of RowKind, options: SHORT (short format) or FULL (full format) | +| table_match_regex | string | no | `.*` | Regex matched against the upstream table path. The transform level rule only applies to matching tables. | +| table_transform | array | no | - | Per-table rules, see [Multi-Table Transform](transform-multi-table.md). | +| rule_match_mode | string | no | `FIRST_MATCH` | How multiple `table_transform` entries with the same `table_path` are evaluated. | ### custom_field_name [string] @@ -89,6 +92,10 @@ Specifies the output format of the RowKind field value. transform_type = FULL # Use full format ``` +### table_match_regex / table_transform / rule_match_mode + +Multi-table options. When the upstream source outputs several tables, one RowKindExtractor transform can apply this rule to all of them: `table_match_regex` selects the tables the transform level rule applies to (default `.*`), `table_transform` holds complete per-table rules (`table_path` plus `custom_field_name` / `transform_type`), and `rule_match_mode` (`FIRST_MATCH` or `ALL_MATCH`) controls how duplicate `table_path` entries are evaluated. See [Multi-Table Transform](transform-multi [...] + ## Complete Examples ### Example 1: Using Default Configuration (SHORT Format) diff --git a/docs/en/transforms/table-filter.md b/docs/en/transforms/table-filter.md index 31e2da30791..05237762c9f 100644 --- a/docs/en/transforms/table-filter.md +++ b/docs/en/transforms/table-filter.md @@ -14,6 +14,9 @@ TableFilter transform plugin for filter tables. | schema_pattern | string | no | | Specify schema filter pattern, the default value is null, which means no filtering. If you want to filter the schema name, please set it to a regular expression. | | table_pattern | string | no | | Specify table filter pattern, the default value is null, which means no filtering. If you want to filter the table name, please set it to a regular expression. | | pattern_mode | enum | no | INCLUDE | Specify pattern mode, the default value is INCLUDE, which means include the matched table. If you want to exclude the matched table, please set it to EXCLUDE. | +| table_match_regex | string | no | `.*` | Regex matched against the upstream table path. The transform level rule only applies to matching tables. | +| table_transform | array | no | - | Per-table rules, see [Multi-Table Transform](transform-multi-table.md). Each entry holds the filter patterns of a single table (`table_path` plus the options above). | +| rule_match_mode | string | no | `FIRST_MATCH` | How multiple `table_transform` entries with the same `table_path` are evaluated. | ## Examples diff --git a/docs/en/transforms/table-rename.md b/docs/en/transforms/table-rename.md index 636408a1967..00daabdd145 100644 --- a/docs/en/transforms/table-rename.md +++ b/docs/en/transforms/table-rename.md @@ -14,6 +14,9 @@ TableRename transform plugin for rename table name. | prefix | string | no | | The prefix to be added to the table name | | suffix | string | no | | The suffix to be added to the table name | | replacements_with_regex | array | no | | The array of replacement rules with regex. The replacement rule is a map with `replace_from` and `replace_to` fields. | +| table_match_regex | string | no | `.*` | Regex matched against the upstream table path. The transform level rule only applies to matching tables. | +| table_transform | array | no | - | Per-table rules, see [Multi-Table Transform](transform-multi-table.md). | +| rule_match_mode | string | no | `FIRST_MATCH` | How multiple `table_transform` entries with the same `table_path` are evaluated. | ## Examples diff --git a/docs/en/transforms/text-chunk.md b/docs/en/transforms/text-chunk.md index 815da88b36e..9aacf265664 100644 --- a/docs/en/transforms/text-chunk.md +++ b/docs/en/transforms/text-chunk.md @@ -19,6 +19,8 @@ chunk text and its 0-based sequence index within the source document. | overlap_size | int | no | 0 | | separators | array | no | ["\n\n", "\n", "。", "!", "?", ". ", " "] | | skip_empty_text | boolean| no | true | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | ### text_field [string] @@ -72,6 +74,18 @@ Controls how a row whose `text_field` is `null` or empty is handled. When `true` when `false` the row is passed through (`output_field` set to `null`, `chunk_index_field` set to `0`). Defaults to `true`. +### table_match_regex [string] + +Regex matched against the upstream table path. The transform level rule only applies to matching +tables. Defaults to `.*` (every table). + +### table_transform [array] + +A list of per-table rules, used when the upstream source outputs several tables. Each entry contains +`table_path` in the form `databaseName[.schemaName].tableName` plus a complete TextChunk rule for +that table (`text_field` and the optional chunking options). Tables with an entry are not affected +by the transform level rule. See [Multi-Table Transform](transform-multi-table.md). + ### common options [string] Transform plugin common parameters, please refer to [Transform Plugin](common-options/common-options.md) for details. diff --git a/docs/zh/connectors/sink/Aerospike.md b/docs/zh/connectors/sink/Aerospike.md index e77932360bd..58e0b099fea 100644 --- a/docs/zh/connectors/sink/Aerospike.md +++ b/docs/zh/connectors/sink/Aerospike.md @@ -71,9 +71,9 @@ import ChangeLog from '../changelog/connector-aerospike.md'; ### data_format 选项说明 -- **map**: 将所有非主键字段作为一个 map 存到 `bin_name` -- **string**: 将所有非主键字段作为 JSON 字符串存到 `bin_name` -- **kv**: 每个非主键字段存储为独立的 bin,此时不使用 `bin_name` +- **map**: 将选中的字段作为一个 map 存到 `bin_name` +- **string**: 将选中的字段作为 JSON 字符串存到 `bin_name` +- **kv**: 每个选中的字段存储为独立的 bin,此时不使用 `bin_name` 当 `data_format` 为 `map` 或 `string` 时,必须配置 `bin_name`,否则写入器无法判断打包后的整行数据应该写入哪个 Aerospike bin。 `key` 指定的字段必须存在于输入 schema 中,因为每次写入都会用它作为 Aerospike 记录主键。 diff --git a/docs/zh/connectors/sink/GcsFile.md b/docs/zh/connectors/sink/GcsFile.md index 08457493fed..0d989293f9f 100644 --- a/docs/zh/connectors/sink/GcsFile.md +++ b/docs/zh/connectors/sink/GcsFile.md @@ -83,6 +83,10 @@ Apache License 2.0,并以 Java 8 为目标版本。shaded GCS Hadoop 库已打 | encoding | string | 条件必填 | `UTF-8` | text、CSV、JSON 和 XML 的编码。 | | enable_header_write | boolean | 条件必填 | `false` | 为 text 或 CSV 输出写入表头。 | | compress_codec | string | 否 | `none` | 所选格式支持的压缩编码。 | +| date_format | string | 否 | `yyyy-MM-dd` | 使用 `text`、`csv`、`excel` 或 `xml` 格式写入 date 类型数据时使用的日期格式。 | +| datetime_format | string | 否 | `yyyy-MM-dd HH:mm:ss` | 使用 `text`、`csv`、`excel` 或 `xml` 格式写入 timestamp 类型数据时使用的时间格式。 | +| time_format | string | 否 | `HH:mm:ss` | 使用 `text`、`csv`、`excel` 或 `xml` 格式写入 time 类型数据时使用的时间格式。 | +| multi_table_sink_replica | int | 否 | `1` | 多表写入模式下每张表的 Sink writer 副本数,必须至少为 1。 | | common-options | | 否 | - | 参见 [Sink 通用配置](../common-options/sink-common-options.md)。 | Sink 也支持对应 SeaTunnel 文件格式定义的格式专用配置,包括 Parquet INT96 和 XML 元素配置。 diff --git a/docs/zh/connectors/source/HiveJdbc.md b/docs/zh/connectors/source/HiveJdbc.md index c837fe48bed..b706435bea4 100644 --- a/docs/zh/connectors/source/HiveJdbc.md +++ b/docs/zh/connectors/source/HiveJdbc.md @@ -60,7 +60,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | DECIMAL<br/>NUMERIC<br/>(未定义精度) | DECIMAL(38, 18) | | CHAR<br/>VARCHAR<br/>STRING | STRING | | DATE | DATE | -| DATETIME<br/>TIMESTAMP | TIMESTAMP | +| TIMESTAMP | TIMESTAMP | | BINARY<br/> ARRAY <br/>INTERVAL <br/>MAP <br/>STRUCT<br/>UNIONTYPE | Not supported yet | ## 源配置项 @@ -75,7 +75,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | connection_check_timeout_sec | Int | 否 | 30 | 等待用于验证连接的数据库操作完成的时间(秒)。 | | socket_timeout_ms | Int | 否 | 86400000 | 从服务器读取数据的 Socket 超时时间(毫秒)。设置为 `0` 表示无超时。已在 Hive 3.2.0+ 测试,更早版本暂未验证。 | | connect_timeout_ms | Int | 否 | 86400000 | 建立到服务器的连接超时时间(毫秒)。设置为 `0` 表示无超时。已在 Hive 3.2.0+ 测试,更早版本暂未验证。 | -| partition_column | String | 否 | - | 并行分区的列名,仅支持数值类型主键,且只能配置一列。 | +| partition_column | String | 否 | - | 用于并行分割的列名。支持数值、字符串和日期类型的列;只能配置一列。 | | partition_lower_bound | String | 否 | - | 扫描的分区列最小值。如果未设置,SeaTunnel 将查询数据库获取最小值。 | | partition_upper_bound | String | 否 | - | 扫描的分区列最大值。如果未设置,SeaTunnel 将查询数据库获取最大值。 | | partition_num | Int | 否 | 10 | 分区数量,仅支持正整数。默认值是 10。 | diff --git a/docs/zh/connectors/source/Oracle.md b/docs/zh/connectors/source/Oracle.md index a09d7070d80..321012f9ee9 100644 --- a/docs/zh/connectors/source/Oracle.md +++ b/docs/zh/connectors/source/Oracle.md @@ -74,7 +74,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | password | String | 否 | - | 连接实例密码 | | query | String | 否 | - | 查询语句。当未配置 `table_path` 和 `table_list` 时必填。 | | connection_check_timeout_sec | Int | 否 | 30 | 等待用于验证连接的数据库操作完成的时间(秒) | -| partition_column | String | 否 | - | 用于并行性分割的列名,仅支持数值类型,仅支持数值类型主键,只能配置一列。 | +| partition_column | String | 否 | - | 用于并行分割的列名。支持数值和字符串类型的列;Oracle 的 `DATE` 和 `TIMESTAMP` 列会被读取为 `TIMESTAMP`,不能作为分割列。只能配置一列。 | | partition_lower_bound | String | 否 | - | partition_column 的最小值用于扫描,如果未设置,SeaTunnel 将查询数据库获取最小值。 | | partition_upper_bound | String | 否 | - | partition_column 的最大值用于扫描,如果未设置,SeaTunnel 将查询数据库获取最大值。 | | partition_num | Int | 否 | 10 | 分割数量,仅支持正整数。默认值是 10。 | @@ -127,7 +127,8 @@ JDBC Source 连接器支持并行读取表数据。SeaTunnel 会按一定规则 **支持的分片数据类型:** - String - Number(int、bigint、decimal 等) -- Date + +Oracle 的 `DATE` 和 `TIMESTAMP` 列会映射为 SeaTunnel `TIMESTAMP`(见上方类型映射),该类型不在支持的分片数据类型中。 ### 与分片相关的选项 @@ -333,7 +334,7 @@ sink { ### 流式增量 ID 区间读取 -Oracle Source 本质上是一个批连接器。设置 `job.mode = "STREAMING"` 只用于开启 checkpoint 以便在失败时恢复作业;source 本身仍然是有界的,每次作业只会读取一次配置好的 `[partition_lower_bound, partition_upper_bound)` 区间。如需周期性地拉取新增数据,必须在外部重新提交作业(例如按计划滑动区间窗口),或改用 Oracle-CDC 做持续变更捕获。 +Oracle Source 本质上是一个批连接器。设置 `job.mode = "STREAMING"` 只用于开启 checkpoint 以便在失败时恢复作业;source 本身仍然是有界的,每次作业只会读取一次配置好的 `[partition_lower_bound, partition_upper_bound]` 区间。如需周期性地拉取新增数据,必须在外部重新提交作业(例如按计划滑动区间窗口),或改用 Oracle-CDC 做持续变更捕获。 ```hocon env { @@ -348,7 +349,7 @@ source { driver = "oracle.jdbc.OracleDriver" username = "root" password = "123456" - query = "SELECT * FROM ORDERS WHERE ORDER_ID >= ? AND ORDER_ID < ?" + query = "SELECT * FROM ORDERS" partition_column = "ORDER_ID" partition_lower_bound = 1 partition_upper_bound = 1000000 diff --git a/docs/zh/connectors/source/Redshift.md b/docs/zh/connectors/source/Redshift.md index 1c133473ea0..bdf39e3f3d4 100644 --- a/docs/zh/connectors/source/Redshift.md +++ b/docs/zh/connectors/source/Redshift.md @@ -56,7 +56,7 @@ Redshift 兼容 PostgreSQL,因此连接器可以读取 JDBC 用户有权访问 | table_path | String | 否 | - | 要读取的完整表名,例如 `public.table2`。单表读取时可直接使用。 | | table_list | Array | 否 | - | 要读取的表列表,每一项可以覆盖 `table_path` 和 `query`。用于开启多表读取与自动分片。 | | connection_check_timeout_sec | Int | 否 | 30 | 用于校验连接的数据库操作超时时间,单位秒。 | -| partition_column | String | 否 | - | 用于并行读取的拆分列,仅支持数值类型的主键列。 | +| partition_column | String | 否 | - | 用于并行读取的拆分列。支持数值、字符串和日期类型的列;只能配置一列。 | | partition_lower_bound | String | 否 | - | `partition_column` 的最小扫描值。未配置时连接器从数据库中查询最小值。 | | partition_upper_bound | String | 否 | - | `partition_column` 的最大扫描值。未配置时连接器从数据库中查询最大值。 | | partition_num | Int | 否 | 10 | 分片数量,仅支持正整数,默认值为 10。 | diff --git a/docs/zh/connectors/source/SqlServer.md b/docs/zh/connectors/source/SqlServer.md index 4e7543385da..68fe626e1b2 100644 --- a/docs/zh/connectors/source/SqlServer.md +++ b/docs/zh/connectors/source/SqlServer.md @@ -79,7 +79,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | password | String | 否 | - | 连接实例的密码。 | | query | String | 否 | - | 查询语句。当未配置 `table_path` 和 `table_list` 时必填。 | | connection_check_timeout_sec | Int | 否 | 30 | 等待用于验证连接的数据库操作完成的时间(秒)。 | -| partition_column | String | 否 | - | 用于并行度分区的列名,仅支持数值类型。 | +| partition_column | String | 否 | - | 用于并行分割的列名。支持数值、字符串和日期类型的列;只能配置一列。 | | partition_lower_bound | String | 否 | - | `partition_column` 扫描的最小值,如果未设置,SeaTunnel 将查询数据库获取最小值。 | | partition_upper_bound | String | 否 | - | `partition_column` 扫描的最大值,如果未设置,SeaTunnel 将查询数据库获取最大值。 | | partition_num | Int | 否 | 10 | 分区数量,仅支持正整数。默认值为 10。 | diff --git a/docs/zh/engines/zeta/benchmark.md b/docs/zh/engines/zeta/benchmark.md index 5f984b3a6a4..db03be067b2 100644 --- a/docs/zh/engines/zeta/benchmark.md +++ b/docs/zh/engines/zeta/benchmark.md @@ -241,7 +241,7 @@ Workflow 会在同一个 Worker 上按以下顺序交替运行,降低机器状 Baseline → Candidate → Candidate → Baseline ``` -Java 8 和 Java 11 分别执行这组对比,报告汇总每个版本的两轮结果。 +Java 11 和 Java 17 分别执行这组对比,报告汇总每个版本的两轮结果。 运行完成后,确认 Job 已执行到 JMH 测量,并核对 Summary 中的 SHA、方法和参数。各字段及 变化公式见[对比报告指标](#对比报告指标);结论不明确时应重新运行。原始 JMH JSON、标准化 diff --git a/docs/zh/engines/zeta/download-seatunnel.md b/docs/zh/engines/zeta/download-seatunnel.md index fbf1bc6c0af..e904f738d25 100644 --- a/docs/zh/engines/zeta/download-seatunnel.md +++ b/docs/zh/engines/zeta/download-seatunnel.md @@ -39,7 +39,7 @@ sh bin/install-plugin.sh 3.1.0 sh bin/install-plugin.sh 3.1.0 ``` -通常您并不需要所有的连接器插件,所以您可以通过配置`config/plugin_config`来指定您所需要的插件,例如,您只需要`connector-console`插件,那么您可以修改plugin.properties配置文件如下 +通常您并不需要所有的连接器插件,所以您可以通过配置`config/plugin_config`来指定您所需要的插件,例如,您只需要`connector-console`插件,那么您可以修改plugin_config配置文件如下 ```plugin_config --seatunnel-connectors-- @@ -56,7 +56,7 @@ connector-console --end-- ``` -您可以在`${SEATUNNEL_HOME}/connectors/plugins-mapping.properties`下找到所有支持的连接器和相应的plugin_config配置名称。 +您可以在`${SEATUNNEL_HOME}/connectors/plugin-mapping.properties`下找到所有支持的连接器和相应的plugin_config配置名称。 :::tip 提示 diff --git a/docs/zh/engines/zeta/stain-trace.md b/docs/zh/engines/zeta/stain-trace.md index 8e940fda0fc..9449ba6fd48 100644 --- a/docs/zh/engines/zeta/stain-trace.md +++ b/docs/zh/engines/zeta/stain-trace.md @@ -8,7 +8,7 @@ StainTrace 是 SeaTunnel 的数据血缘与端到端性能追踪系统,用于 - **框架级实现**:所有 Connector 自动支持,无需修改连接器代码 - **6 个基础阶段**:SOURCE_EMIT、QUEUE_IN、QUEUE_OUT、TRANSFORM_IN、TRANSFORM_OUT、SINK_WRITE_DONE(已全部落点;实际出现顺序取决于流水线拓扑) -- **扩展阶段**:40+ 个细粒度阶段代码已定义,用于未来精细化性能分析(**当前尚未落点,不会出现在追踪文件中**) +- **扩展阶段**:34 个细粒度阶段代码已定义,用于未来精细化性能分析(**当前尚未落点,不会出现在追踪文件中**) - **本地文件存储**:零依赖,OTLP JSON Lines 格式 - **性能优化**:合理采样配置下开销 < 2% @@ -70,7 +70,7 @@ seatunnel: engine: stain-trace-enabled: true stain-trace-sample-rate: 100000 # 每 10 万条采样 1 条 - stain-trace-file-base-path: /data/seatunnel/traces # 本地追踪文件目录,必须显式配置,默认不启用 + stain-trace-file-base-path: /tmp/seatunnel/traces # 本地追踪文件目录,必须显式配置,默认不启用 ``` ### 2. 启用任务级开关 diff --git a/docs/zh/transforms/data-validator.md b/docs/zh/transforms/data-validator.md index 9937ee2eda8..b01bdd0be38 100644 --- a/docs/zh/transforms/data-validator.md +++ b/docs/zh/transforms/data-validator.md @@ -13,6 +13,9 @@ DataValidator 转换插件会根据配置规则校验字段值,并按照指定 | row_error_handle_way| enum | 否 | FAIL | | row_error_handle_way.error_table | string | 否 | | | field_rules | array | 是(单表模式) | | +| table_match_regex | string | 否 | `.*` | +| table_transform | array | 否 | - | +| rule_match_mode | string | 否 | `FIRST_MATCH` | ### row_error_handle_way [enum] @@ -61,6 +64,8 @@ DataValidator 转换插件会根据配置规则校验字段值,并按照指定 单表配置必须设置 `field_rules`。使用 `table_transform` 时每个条目自己声明 `field_rules`,外层可选。 +`table_match_regex` 选择转换级别规则适用的表(默认 `.*`),`table_transform` 保存完整的单表规则(`table_path` 加上 `field_rules`),`rule_match_mode`(`FIRST_MATCH` 或 `ALL_MATCH`)控制相同 `table_path` 条目的求值方式。参见 [多表转换](transform-multi-table.md)。 + #### 字段规则结构 每个字段规则包含: diff --git a/docs/zh/transforms/define-sink-type.md b/docs/zh/transforms/define-sink-type.md index 1346e7e1a60..c574052cb53 100644 --- a/docs/zh/transforms/define-sink-type.md +++ b/docs/zh/transforms/define-sink-type.md @@ -11,6 +11,9 @@ DefineSinkType 转换插件用于定义 sink 字段的目标存储类型,适 | 参数名 | 类型 | 是否必填 | 默认值 | 说明 | |:------:|---------------------------|----------|--------|------| | columns | list<map<string, string>> | 是 | | 需要定义的列,必须为每一列指定名称和类型 | +| table_match_regex | string | 否 | `.*` | 与上游表路径进行匹配的正则表达式,转换级别规则只应用于匹配的表。 | +| table_transform | array | 否 | - | 针对单表的规则,参见 [多表转换](transform-multi-table.md)。每个条目包含 `table_path` 以及该表的 `columns`。 | +| rule_match_mode | string | 否 | `FIRST_MATCH` | 当多个 `table_transform` 条目的 `table_path` 相同时,决定它们的匹配方式。 | ## 示例 diff --git a/docs/zh/transforms/dynamic-compile.md b/docs/zh/transforms/dynamic-compile.md index a5fb41a6466..d7afbd881b5 100644 --- a/docs/zh/transforms/dynamic-compile.md +++ b/docs/zh/transforms/dynamic-compile.md @@ -22,6 +22,9 @@ DynamicCompile 转换插件提供一种可编程的方式来处理行,允许 | compile_language | Enum | yes | | | compile_pattern | Enum | no | SOURCE_CODE | | absolute_path | string | no | | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | +| rule_match_mode | string | no | `FIRST_MATCH` | ### 通用选项 [string] @@ -47,6 +50,10 @@ SOURCE_CODE,ABSOLUTE_PATH ### source_code [string] 源代码 +### table_match_regex / table_transform / rule_match_mode + +多表选项。当上游数据源输出多张表时,一个 DynamicCompile 转换即可处理所有这些表:`table_match_regex` 选择转换级别规则适用的表(默认 `.*`),`table_transform` 保存完整的单表规则(`table_path` 加上上述选项),`rule_match_mode`(`FIRST_MATCH` 或 `ALL_MATCH`)控制相同 `table_path` 条目的求值方式。参见 [多表转换](transform-multi-table.md)。 + #### 关于source_code 在代码中,你必须实现两个方法 - `Column[] getInlineOutputColumns(CatalogTable inputCatalogTable)` diff --git a/docs/zh/transforms/field-rename.md b/docs/zh/transforms/field-rename.md index c4b166ebb99..1c81c0accab 100644 --- a/docs/zh/transforms/field-rename.md +++ b/docs/zh/transforms/field-rename.md @@ -15,6 +15,9 @@ FieldRename 转换插件用于批量重命名字段名。 | suffix | string | 否 | | 追加到字段名后的后缀 | | replacements_with_regex | array | 否 | | 替换规则数组,元素为包含 `replace_from`、`replace_to` 以及可选 `is_regex`(默认 `true`)的映射;当 `is_regex=false` 时,`replace_from` 按字段名精确匹配(全匹配) | | specific | array | 否 | | 指定字段重命名规则,元素为包含 `field_name` 和 `target_name` 的映射;命中后会直接重命名并跳过其他规则 | +| table_match_regex | string | 否 | `.*` | 与上游表路径进行匹配的正则表达式,转换级别规则只应用于匹配的表 | +| table_transform | array | 否 | - | 针对单表的规则,参见 [多表转换](transform-multi-table.md)。 | +| rule_match_mode | string | 否 | `FIRST_MATCH` | 当多个 `table_transform` 条目的 `table_path` 相同时,决定它们的匹配方式。 | ## 示例 diff --git a/docs/zh/transforms/replace.md b/docs/zh/transforms/replace.md index 466e4dc35e2..824abea2a9e 100644 --- a/docs/zh/transforms/replace.md +++ b/docs/zh/transforms/replace.md @@ -15,6 +15,9 @@ | replacement | string | yes | - | | is_regex | boolean | no | false | | replace_first | boolean | no | false | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | +| rule_match_mode | string | no | `FIRST_MATCH` | ### replace_fields [array] @@ -44,6 +47,18 @@ 是否替换第一个匹配字符串。仅在 `is_regex = true` 时使用。 +### table_match_regex [string] + +与上游表路径进行匹配的正则表达式。转换级别规则(位于 `table_transform` 之外的上述选项)只会应用于路径匹配该正则表达式的表。默认值 `.*` 匹配所有表。 + +### table_transform [array] + +针对单表的规则列表。每个条目必须包含 `table_path`(格式为 `databaseName[.schemaName].tableName`,精确匹配),以及该表的一条完整的 Replace 规则(`replace_fields`、`pattern`、`replacement`)。有对应条目的表不会再受转换级别规则影响。 + +### rule_match_mode [string] + +控制当多个 `table_transform` 条目的 `table_path` 相同时的处理方式:`FIRST_MATCH` 只应用第一个匹配的条目,`ALL_MATCH` 按顺序应用所有匹配的条目。如果未设置,重复的 `table_path` 条目会被拒绝。 + ### common options [string] 转换插件的常见参数, 请参考 [Transform Plugin](common-options/common-options.md) 了解详情 diff --git a/docs/zh/transforms/rowkind-extractor.md b/docs/zh/transforms/rowkind-extractor.md index 04284f68ab3..1965211b6b6 100644 --- a/docs/zh/transforms/rowkind-extractor.md +++ b/docs/zh/transforms/rowkind-extractor.md @@ -42,6 +42,9 @@ RowKindExtractor 转换插件用于将 CDC(Change Data Capture)数据流改 |-------------------|--------|----------|---------------|------| | custom_field_name | string | 否 | row_kind | 新增字段的名称,用于存储原始的 RowKind 信息 | | transform_type | enum | 否 | SHORT | RowKind 的输出格式,可选值:SHORT(短格式)或 FULL(完整格式) | +| table_match_regex | string | 否 | `.*` | 与上游表路径进行匹配的正则表达式,转换级别规则只应用于匹配的表。 | +| table_transform | array | 否 | - | 针对单表的规则,参见 [多表转换](transform-multi-table.md)。 | +| rule_match_mode | string | 否 | `FIRST_MATCH` | 当多个 `table_transform` 条目的 `table_path` 相同时,决定它们的匹配方式。 | ### custom_field_name [string] @@ -89,6 +92,10 @@ custom_field_name = "operation_type" # 使用自定义字段名 transform_type = FULL # 使用完整格式 ``` +### table_match_regex / table_transform / rule_match_mode + +多表选项。当上游数据源输出多张表时,一个 RowKindExtractor 转换即可处理所有这些表:`table_match_regex` 选择转换级别规则适用的表(默认 `.*`),`table_transform` 保存完整的单表规则(`table_path` 加上 `custom_field_name` / `transform_type`),`rule_match_mode`(`FIRST_MATCH` 或 `ALL_MATCH`)控制相同 `table_path` 条目的求值方式。参见 [多表转换](transform-multi-table.md)。 + ## 完整示例 ### 示例 1:使用默认配置(SHORT 格式) diff --git a/docs/zh/transforms/table-filter.md b/docs/zh/transforms/table-filter.md index f6a8358f517..97bfaab58dd 100644 --- a/docs/zh/transforms/table-filter.md +++ b/docs/zh/transforms/table-filter.md @@ -14,6 +14,9 @@ TableFilter 转换插件用于按表名、库名或 schema 规则,正向或反 | schema_pattern | string | 否 | | schema 过滤规则。默认不过滤;如需过滤 schema 名称,请填写正则表达式。 | | table_pattern | string | 否 | | 表过滤规则。默认不过滤;如需过滤表名称,请填写正则表达式。 | | pattern_mode | enum | 否 | INCLUDE | 过滤模式。`INCLUDE` 表示保留匹配的表,`EXCLUDE` 表示排除匹配的表。 | +| table_match_regex | string | 否 | `.*` | 与上游表路径进行匹配的正则表达式,转换级别规则只应用于匹配的表。 | +| table_transform | array | 否 | - | 针对单表的规则,参见 [多表转换](transform-multi-table.md)。每个条目包含 `table_path` 以及该表的过滤规则(上述选项)。 | +| rule_match_mode | string | 否 | `FIRST_MATCH` | 当多个 `table_transform` 条目的 `table_path` 相同时,决定它们的匹配方式。 | ## 示例 diff --git a/docs/zh/transforms/table-rename.md b/docs/zh/transforms/table-rename.md index 9f0ff9bf0f8..3391f7cb652 100644 --- a/docs/zh/transforms/table-rename.md +++ b/docs/zh/transforms/table-rename.md @@ -14,6 +14,9 @@ TableRename 转换插件用于重命名表名。 | prefix | string | 否 | | 追加到表名前的前缀 | | suffix | string | 否 | | 追加到表名后的后缀 | | replacements_with_regex | array | 否 | | 正则替换规则数组,元素为包含 `replace_from`、`replace_to` 的映射,用于批量替换表名 | +| table_match_regex | string | 否 | `.*` | 与上游表路径进行匹配的正则表达式,转换级别规则只应用于匹配的表 | +| table_transform | array | 否 | - | 针对单表的规则,参见 [多表转换](transform-multi-table.md)。 | +| rule_match_mode | string | 否 | `FIRST_MATCH` | 当多个 `table_transform` 条目的 `table_path` 相同时,决定它们的匹配方式。 | ## 示例 diff --git a/docs/zh/transforms/text-chunk.md b/docs/zh/transforms/text-chunk.md index 2d03448eb3c..ce206396d24 100644 --- a/docs/zh/transforms/text-chunk.md +++ b/docs/zh/transforms/text-chunk.md @@ -17,6 +17,8 @@ | overlap_size | int | no | 0 | | separators | array | no | ["\n\n", "\n", "。", "!", "?", ". ", " "] | | skip_empty_text | boolean| no | true | +| table_match_regex | string | no | `.*` | +| table_transform | array | no | - | ### text_field [string] @@ -57,6 +59,14 @@ 控制 `text_field` 为 `null` 或空字符串时的行处理逻辑。为 `true` 时该行被丢弃,为 `false` 时该行透传 (`output_field` 置为 `null`、`chunk_index_field` 置为 `0`),默认值为 `true`。 +### table_match_regex [string] + +与上游表路径进行匹配的正则表达式,转换级别规则只应用于匹配的表。默认值 `.*` 匹配所有表。 + +### table_transform [array] + +针对单表的规则列表,用于上游数据源输出多张表的场景。每个条目包含 `table_path`(格式为 `databaseName[.schemaName].tableName`)以及该表的一条完整的 TextChunk 规则(`text_field` 及可选的分块选项)。有对应条目的表不会再受转换级别规则影响。参见 [多表转换](transform-multi-table.md)。 + ### common options [string] 转换插件的常见参数, 请参考 [Transform Plugin](common-options/common-options.md) 了解详情。
