huyuanfeng2018 opened a new pull request, #11152:
URL: https://github.com/apache/paimon/pull/11152
### Purpose
Fixes #11138.
The writer coordinator refreshes its cached metadata when a checkpoint is
triggered, before the committer publishes that checkpoint's table snapshot. The
cache survives task recovery, so a restored primary-key writer can initialize
its sequence numbers from older files and silently lose replayed updates to
already committed rows.
Invalidate the metadata view on both global and subtask recovery, then
refresh the snapshot and cached commit identifiers on the existing request
executor before serving the next request. Track recovery generations so a
failed refresh is retried and a reset during an in-flight refresh is not lost.
Keep existing paged responses available to unaffected subtasks.
### Tests
- Reproduced on unmodified upstream master
`57233dbc18db4cefa45ab1447a44843368b1191e`: both recovery metadata regressions
fail, and the Flink 1.20.1 MiniCluster test returns version 4 instead of 5 with
the coordinator enabled. The coordinator-disabled control passes.
- The MiniCluster regression uses a checkpointed replayable source, manually
triggered checkpoints, an injected task failure, and a real Parquet primary-key
table. It verifies the restored source position and the final committed row.
- Unit coverage includes global/subtask recovery, reset before start, commit
identifier refresh, failed refresh retry, a second reset during refresh, and
continued consumption of in-flight paged responses.
Validation on Temurin JDK 8u512 and Flink 1.20.1:
- `WriteOperatorCoordinatorTest`: 7 passed.
- `TableWriteCoordinatorTest`: 7 passed.
- `WriterCoordinatorRecoveryITCase`: 2 passed (coordinator enabled/disabled).
- Checkstyle, Spotless, Maven enforcer checks, and `git diff --check` passed.
- The 28-module dependency reactor compiled and installed successfully with
tests skipped before the targeted test run.
```bash
mvn -pl paimon-flink/paimon-flink-common -Pflink1 \
-DwildcardSuites=none -Dflink.forkCount=1 \
-Dtest=WriteOperatorCoordinatorTest,TableWriteCoordinatorTest,WriterCoordinatorRecoveryITCase
test
```
The local container required supplying Mockito's Byte Buddy agent through
`-DextraJavaTestArgs=-javaagent:<byte-buddy-agent.jar>` because JVM dynamic
attachment is unavailable. No project configuration was changed for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]