SEZ9 commented on issue #13044: URL: https://github.com/apache/seatunnel/issues/13044#issuecomment-6514141234
Thanks @zephyr45, that reproduction is exactly what was needed. The parallelism 1 vs 2 comparison on 2.3.13 with per-run topics/prefixes rules out cross-run fencing, and the combination of `ProducerFencedException` in the logs, a `FINISHED` job with 100 read / 100 written, exit code 0, and only 50 distinct records visible to `read_committed` confirms the shared transactionalId problem, plus a silent data-loss symptom on top of it. Your plan sounds right: isolating each writer's transactionalId (so the id incorporates the subtask index, not just prefix + checkpointId), covering checkpoint rotation and recovery/abort including restoration of existing state, a two-writer unit regression, and a parallel exactly-once Kafka E2E test that verifies every expected record under `read_committed`. A few concrete asks for the PR: 1. Please make sure the recovery/abort path uses the same per-writer id scheme as the normal path, so a restarted writer aborts only its own open transactions and does not fence a sibling writer. 2. Please keep the E2E test at parallelism > 1 against a single-partition sink topic, since that is the layout that exposed the loss here. 3. In the PR description, please note whether the fenced writer's failure should surface as a job failure rather than `FINISHED`; if you think that belongs in a separate follow-up, just say so and we can track it separately. Please link the PR back to this issue when it is up and I will review it promptly. <!-- streview-comment:2149 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
