SEPURI-SAI-KRISHNA opened a new issue, #12979: URL: https://github.com/apache/seatunnel/issues/12979
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened This is the `ChainedMapTransform` remainder that #12938 deferred. Its author said on that PR that they would track it separately, so this issue is meant as that tracking record rather than a claim on the fix: the implementation is theirs if they want it. I measured the behaviour while reading #12938 and the numbers are below. `ChainedMapTransform` is reachable in a submitted job, though the condition is narrower than it first looks. `AbstractMultiCatalogMapTransform.composeTransforms` returns `new ChainedMapTransform(transforms)` only when one table resolves to more than one rule, and it is stored in the non-transient `transformMap`, which is serialized into the Zeta submission payload. That needs `rule_match_mode = ALL_MATCH` together with two or more `table_transform` entries on the same exact `table_path`. Without `rule_match_mode` configured, `AbstractMultiCatalogTransform` calls `rejectDuplicateTablePaths` and duplicate paths are rejected outright, and `FIRST_MATCH` keeps only the first entry. So the affected configurations are exactly the documented `ALL_MATCH` chaining case. #12804 added a field to it: ```java private final List<SeaTunnelTransform<SeaTunnelRow>> transforms; /** Extra table paths of every rule, rebuilt only by {@link #refreshExtraTablePaths()}. */ private final List<Set<String>> extraTablePaths = new ArrayList<>(); ``` `extraTablePaths` is non-transient with an inline initializer and is written only by `refreshExtraTablePaths()`, which runs from the constructor, `mapSchemaChangeEvent` and `setInputCatalogTables`. Java deserialization runs neither the constructor nor the field initializer. `map()` dereferences it once per row: ```java if (currentRow != null && extraTablePaths.get(i).contains(currentRow.getTableId())) { ``` Nothing rebuilds it on the normal path. `TransformFlowLifeCycle.received(...)` calls `setInputCatalogTables` only inside the `SchemaChangeEvent` branch and only when `i > 0`, so a job without DDL never reaches it, and the guard skips index 0, so the first transform in the pipeline never receives it. `AbstractMultiCatalogTransform.setInputCatalogTables` forwards to the existing inner transforms instead of rebuilding them. That refresh is not a recovery path either. `ChainedMapTransform.setInputCatalogTables` ends with `refreshExtraTablePaths()`, which starts with `extraTablePaths.clear()`, so on a deserialized instance it fails the same way. Measured: ``` setInputCatalogTables THREW java.lang.NullPointerException: Cannot invoke "java.util.List.clear()" because "this.extraTablePaths" is null ``` The class declares no `serialVersionUID`. **Measured.** Old class compiled from `02b156e45`, current from `dev` `886eb632f`, streams written and read in separate JVMs on JDK 17: | scenario | result | | --- | --- | | pre-#12804 implicit UID | `8769738606872706499` | | current `dev` implicit UID | `4659105814088680487` | | old stream read by current `dev` | `InvalidClassException`, loud and diagnosable | | old stream read by `dev` with that UID pinned | deserializes, then `map()` throws `NullPointerException` on the first row | | old stream read by `dev` with the field `transient` and rebuilt in `readObject` | `map()` returns the row | **Why the one-line fix used in #12938 is not sufficient here.** Pinning the historical UID alone lets the old stream through, the field is absent from it, the initializer does not run, and the first data row fails with: ``` java.lang.NullPointerException: Cannot invoke "java.util.List.get(int)" because "this.extraTablePaths" is null ``` No DDL is involved. That converts a startup failure into a crash in the data path, which is worse. **Two findings that may shorten the fix.** 1. Making the field `transient` restores `8769738606872706499` on its own, because the serialVersionUID computation skips private transient fields. No explicit pin and no `serialver` run are needed. The rebuild is the part that matters, in `readObject` or lazily in `map()`, and `refreshExtraTablePaths()` derives everything from `transforms`, which is itself serialized. 2. That shape stops streams archived between #12804 and the fix from loading: `4659105814088680487` against `8769738606872706499`. Pinning the current UID instead would keep those working and keep failing the older ones. It is a choice of which window to support, and worth deciding deliberately. Either way the window is snapshot-only and never existed in a release. `FileSourceState` in `connector-file-base` is a good precedent for the rebuild shape: it null-guards every added field in `readObject`. **Not proven.** No real Zeta job was submitted. The measurements above use stub transforms in place of real rules, so the reachability argument above is read from source rather than executed. `AbstractMultiCatalogTransformTest` exercises this class for schema-change and close behaviour but has no serialization coverage, which is why neither half was caught. ### SeaTunnel Version `dev` (3.0.1-SNAPSHOT), measured at `886eb632f`. ### SeaTunnel Config ```hocon transform { Copy { rule_match_mode = "ALL_MATCH" table_transform = [{ table_path = "test.xyz" src_field = "name" dest_field = "name2" }, { table_path = "test.xyz" src_field = "name2" dest_field = "name3" }] } } ``` This is the `ALL_MATCH` example from `docs/en/transforms/transform-multi-table.md`. Two entries resolving to one table is what produces a `ChainedMapTransform`. ### Running Command ```shell # submit the job above on a build that predates #12804, then restore or resubmit the # archived submission payload against current dev ./bin/seatunnel.sh --config config/chained.conf ``` ### Error Exception ```log java.io.InvalidClassException: org.apache.seatunnel.transform.common.ChainedMapTransform; local class incompatible: stream classdesc serialVersionUID = 8769738606872706499, local class serialVersionUID = 4659105814088680487 # and, once the historical UID is pinned without a rebuild: java.lang.NullPointerException: Cannot invoke "java.util.List.get(int)" because "this.extraTablePaths" is null ``` ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version JDK 17 (OpenJDK 17.0.20.1) ### Screenshots _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! I have the fix and the fixtures ready, but the author of #12938 called this out first and said they would track it. I will not open a PR unless they would rather not take it. ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
