hutiefang76 opened a new pull request, #11271:
URL: https://github.com/apache/seatunnel/pull/11271
### Purpose of this pull request
Fixes #11260.
When a multi-table CDC job is restored from a savepoint after one table is
removed from the current job configuration, the restored incremental split can
still carry state for the removed table. That stale state may later be used
while registering database history and schema, causing restore startup to fail
for a table that is no longer part of the job.
This patch keeps the fix focused on restored incremental-state pruning:
- prune removed tables from `IncrementalSplit.tableIds`
- prune matching entries from `completedSnapshotSplitInfos`
- prune removed-table `checkpointTables`
- prune removed-table `historyTableChanges`
- skip a restored incremental split if no tables remain after pruning
### Does this PR introduce _any_ user-facing change?
No. This only fixes restore behavior for CDC jobs whose table list changed
before restoring from a checkpoint/savepoint.
### How was this patch tested?
Added `IncrementalSplitTest` to verify that a removed table is pruned from
all restored incremental-state structures.
Tested locally with Java 8:
```bash
JAVA_HOME=$(/usr/libexec/java_home -v 1.8) PATH="$JAVA_HOME/bin:$PATH"
./mvnw -o -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base clean
test -Dtest=IncrementalSplitTest,IncrementalSplitStateTest -DskipITs
-Drat.skip=true -DfailIfNoTests=false
```
Result: 3 tests, 0 failures, 0 errors.
```bash
JAVA_HOME=$(/usr/libexec/java_home -v 1.8) PATH="$JAVA_HOME/bin:$PATH"
./mvnw -o -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base test
-Dtest=IncrementalSourceSplitReaderTest,IncrementalSourceStreamFetcherTest
-DskipITs -Drat.skip=true -DfailIfNoTests=false
```
Result: 5 tests, 0 failures, 0 errors.
Also ran:
```bash
git diff --check
```
No output.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
The checklist items above are not applicable because this patch does not add
a new connector, binary package, user-facing feature, documentation change, or
incompatible behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
