zephyr45 opened a new pull request, #13060: URL: https://github.com/apache/seatunnel/pull/13060
### Purpose of this pull request Yes, I am contributing to Apache SeaTunnel. Fixes #13044. Thanks to @nzw921rx for the reproduction and root-cause investigation. Parallel Kafka sink writers currently reuse `<prefix>-<checkpoint>` transactional IDs when they share a configured prefix, so they fence each other. This change gives each writer a `<prefix>-<subtask>` namespace for transaction creation, checkpoint rotation and recovery cleanup. The original prefix remains in `KafkaSinkState`, preventing repeated restores from accumulating subtask suffixes. Recovery fences unfinished IDs in the current writer's namespace and also cleans the legacy shared namespace when restoring old state; restoring another writer's state does not authorize fencing that writer's current namespace. The change adds six writer regression tests, a real-broker batch test with two source partitions, two writers and 100 distinct records, and parallelism in the existing streaming restore test. The batch assertion checks both exact values and count through `read_committed`. ### Does this PR introduce _any_ user-facing change? Yes. An `EXACTLY_ONCE` Kafka sink can use parallel writers sharing one configured prefix without a transactional-ID collision. Transactional IDs now have the shape `<transaction_prefix>-<subtask>-<checkpoint>`. Users whose Kafka ACLs or monitoring depend on the old ID shape need to update them and retain access to old IDs while restoring old checkpoints. English and Chinese sink documentation and migration notes describe this change. No configuration keys, defaults, public API/SPI signatures, dependencies or checkpoint DTO fields are changed. Stop the previous job attempt before recovery and keep a distinct configured prefix for each concurrent job. ### How was this patch tested? Local runtime: Java 11 on Windows, Docker Linux containers, repository-pinned Kafka image `confluentinc/cp-kafka:7.0.9`, Zeta engine. Dependencies were installed beforehand; backend CI's UI and third-party-license-generation skip flags were used for the integration harness build. - Kafka connector unit tests: 138 tests, no failures/errors/skips, repeated successfully against the refreshed dependencies. Before the production fix, the focused regression run had three expected failures demonstrating writer/checkpoint and legacy-restore collisions. - Targeted Kafka integration run: parallel batch, parallel streaming restore and existing streaming exactly-once; 3 tests passed, none skipped. - Full `KafkaIT` Zeta suite: all 43 tests passed twice, including a repeat against the refreshed runtime, with no failures/errors/skips. Both runs include the parallel batch and streaming-restore scenarios. - Scoped `spotless:apply` and `git diff --check` passed. - `MarkdownTest,ChineseCharacterCheckTest`: four enabled checks passed; three repository/platform-disabled checks skipped. The refreshed Markdown run also passed its three enabled checks, with one Windows-disabled case skipped. - Repository-wide `verify` with tests skipped: all 329 modules passed, including distribution packaging (Java 11, two Maven threads, backend CI UI/license-generation skip flags). Full integration command (PowerShell, with `JAVA_HOME` set to Java 11): ```powershell $env:RUN_ENGINES = 'zeta' .\mvnw.cmd -B -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e '-DskipUT=true' '-DskipIT=false' '-Dskip.ui=true' '-Dlicense.skipAddThirdParty=true' '-Dit.test=KafkaIT' verify ``` Flink and Spark integration runs were not performed locally. The new batch test has no engine exclusions. Released-version control: tested the official 2.3.13 binary (verified with Apache's SHA-512) and matching Maven Kafka connector on Java 11 against the same Kafka image. A Kafka source supplied 100 distinct text records across two partitions (`max.poll.records=1`); the sink used `EXACTLY_ONCE`, a common configured prefix and one output partition. With parallelism 1, the job exited 0 and all 100 expected records were visible to `read_committed`. With parallelism 2, it reported `FINISHED`, 100 read and 100 written, and exited 0, but logged `ProducerFencedException` and only 50 distinct records were visible. Each run used an isolated sink topic and prefix. The [verified result is also recorded in the issue](https://github.com/apache/seatunnel/issues/13044#issuecomment-6500297286). This PR targets `dev`. ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md) * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [x] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. * [x] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) No new JAR dependencies or connector registrations are introduced; mapping, distribution, CI-label and plugin-config entries therefore need no changes. The existing Kafka E2E suite is updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
