sundapeng opened a new pull request, #9433: URL: https://github.com/apache/paimon/pull/9433
### Purpose `FlinkFormatTableDataStreamSinkTest` aside, the Flink format table sink currently gets `INSERT OVERWRITE` wrong in two ways. 1. **Parallel overwrite commits race.** `sinkFrom` leaves the sink at the default parallelism, so every subtask builds its own `BatchTableCommit` with overwrite semantics. Each of those commits deletes the target's existing files before publishing its own, so two subtasks can delete files the other just published. The result depends on interleaving. 2. **An overwrite that produced no rows silently does nothing.** `close()` only committed when `prepareCommit()` returned a non-empty list. For a normal insert that is correct. For an overwrite it is not: `INSERT OVERWRITE ... SELECT` with an empty result is supposed to replace the target with nothing, and the batch path (`FormatTableCommit`) already implements that. The streaming sink skipped the commit entirely and left the old data in place. This change: - sets parallelism to 1 when `overwrite` is true, so a single subtask owns the delete-then-publish sequence; - tracks `endOfInput` from `flush(boolean)` and commits when `overwrite && reachedEndOfInput`, even with no commit messages, so an empty overwrite clears its target; - keeps `abort()` aligned with the new condition, so the abort path only runs for a commit that was actually attempted. Serializing overwrite commits costs write throughput for overwrite jobs. That is deliberate: correctness of the replace semantics is not something parallelism can buy back, and non-overwrite inserts are unaffected. ### Tests `FlinkFormatTableDataStreamSinkTest` (new, 5 cases) covers overwrite parallelism pinning, non-overwrite parallelism being untouched, the empty-overwrite commit, the non-empty overwrite commit, and the abort path. ``` mvn -pl paimon-flink/paimon-flink-common -am -Pflink1 -Dsurefire.failIfNoSpecifiedTests=false -DwildcardSuites=none -Dtest=FlinkFormatTableDataStreamSinkTest test ``` Tests run: 5, Failures: 0, Errors: 0, Skipped: 0. `spotless:check` + `checkstyle:check` on the module pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
