nzw921rx opened a new pull request, #12804:
URL: https://github.com/apache/seatunnel/pull/12804
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
Close #12779, close #12780, close #12781.
`FactoryUtil#createAndPrepareMultiTableTransform` validates only the
transform-level options against
`factory.optionRule()`, while the real rule of a table is resolved later in
`AbstractMultiCatalogTransform`: `table_transform` entry (exact
`table_path`) > transform-level rule
(when the table matches `table_match_regex`) > unchanged. Since #11095 moved
the checks of the
affected transforms into the transform-level rule, a configuration that only
uses `table_transform`
was rejected before any table was resolved:
```
options: include_fields, exclude_fields
type: exclusive
constraint: exactly one option must be set, but none are configured
```
The same regression affected `Copy`/`FilterRowKind` (exclusive groups) and
15 transforms whose options
became `required(...)` at the transform level. This PR keeps the
transform-level contract for
single-table configurations while making the per-table path work again.
This PR:
- adds `TableTransformRuleValidator`, which validates **every**
`table_transform` entry as a complete
single-table rule, reports the offending `table_path`, requires a
non-blank `table_path` and
rejects a transform-level rule that is only partially configured;
- keeps the mandatory/exclusive contract by moving it into
`.conditionalRule(table_transform, defaultValue, tableRule)`: without
`table_transform` the
transform-level rule is mandatory, with `table_transform` it becomes
optional;
- resolves the effective rule per table (`entry` > transform-level rule >
unchanged) in
`AbstractMultiCatalogTransform`, so tables without a matching entry stay
unchanged when no
transform-level rule is configured;
- adds the missing multi-table wrapper of `DataValidator`, which declared
`table_transform` but only
ever built a single table;
- adds the optional `column_rule.sink_type` assertion to the Assert sink, so
transforms that declare
sink types (e.g. `DefineSinkType`) can be verified in E2E tests.
### Does this PR introduce _any_ user-facing change?
Yes.
* Since SeaTunnel 3.0.0 a valid multi-table configuration such as
```hocon
transform {
Filter {
table_transform = [{
table_path = "test.xyz"
include_fields = [name]
}]
}
}
```
was rejected at submission time. It is accepted again (2.3.x behaviour)
and `test.xyz` is filtered
while tables without an entry stay unchanged.
* Configurations that combine a transform-level rule with `table_transform`
keep their current
behaviour: entries take precedence, the transform-level rule still applies
to the tables matched
by `table_match_regex` (default all tables).
* Configurations are now validated more strictly where they were broken
before: a partially
configured transform-level rule and an incomplete `table_transform` entry
are rejected with a
message that points to the option or the table path.
* `DataValidator` supports multi-table validation (`table_transform`) as
declared.
* The Assert sink accepts an optional `column_rule.sink_type` and only
compares it when it is
declared.
Documentation: the Assert connector docs (en/zh) are updated in this PR. The
transform plugin docs
of the affected transforms (`Filter`, `Copy`, `FilterRowKind`,
`FieldMapper`, `Replace`, `Split`,
`Sql`, `Calcite`, `Metadata`, `JsonPath`, `DataValidator`, `DefineSinkType`,
`Python`,
`FieldEncrypt`, `TableMerge`, `RegexExtract`, `TextChunk`, `DynamicCompile`)
will be updated in a
follow-up PR.
### How was this patch tested?
Unit tests (`seatunnel-transforms-v2`):
* new/extended factory tests for every affected transform:
`table_transform`-only passes, an entry
without its mandatory option fails, an entry without `table_path` fails, a
partially configured
transform-level rule fails, and the single-table mode keeps requiring the
full rule
(`FilterFieldTransformFactoryTest`, `CopyFieldTransformFactoryTest`,
`FilterRowKindTransformFactoryTest`, `FieldMapperTransformFactoryTest`,
`ReplaceTransformFactoryTest`, `SplitTransformFactoryTest`,
`SQLTransformFactoryTest`,
`CalciteTransformFactoryTest`, `MetadataTransformFactoryTest`,
`JsonPathTransformFactoryTest`,
`DataValidatorTransformFactoryTest`, `DefineSinkTypeTransformFactoryTest`,
`FieldEncryptTransformFactoryTest`, `RegexExtractTransformFactoryTest`,
`TextChunkTransformFactoryTest`, `DynamicCompileTransformFactoryTest` and
the new
`TableMergeTransformFactoryTest`, `PythonTransformFactoryTest`);
* multi-catalog behaviour: `FilterFieldTransformTest`,
`CopyFieldMultiCatalogTransformTest`,
`FilterRowKindTransformFactoryTest` assert that a table without a matching
entry keeps its schema
and rows, and that a table with an entry gets its own rule;
* `DefineSinkTypeTransformFactoryTest` verifies the declared sink types of
both the transform-level
mode and the per-table mode (the Assert sink cannot compare sink types in
older versions).
```bash
./mvnw -pl seatunnel-transforms-v2 test
# Tests run: 1347, Failures: 0, Errors: 0, Skipped: 0
# PythonTransformTest is environment dependent (requires
# -Dseatunnel.transform.python.enabled=true and the python executable
allowlist)
```
E2E tests (new configurations, executed by CI containers):
* single-table mode, transform-level rule plus per-table rules and per-table
rules only for
`Filter`, `Copy`, `FilterRowKind`, `Split`, `TextChunk`, `FieldMapper`,
`FieldEncrypt`,
`RegexExtract`, `Replace`, `Sql`, `Calcite`, `Metadata`, `TableMerge`,
`DataValidator`,
`DefineSinkType`, `Python` and `DynamicCompile` (e.g.
`filter_transform_table_transform_only.conf`, `metadata_single_table.conf`,
`define_sink_type_multi_table.conf`, ...);
* `TestSplitIT#testSplitMultiTable` was pointing to a `RowkindExtractor`
configuration, it now
executes `split_transform_multi_table.conf`.
```bash
./mvnw -pl
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-1,seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2
-DskipTests test-compile
./mvnw -pl seatunnel-connectors-v2/connector-assert test
```
Maven `verify` was not executed locally; the E2E suites require
Docker/Testcontainers.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
