zephyr45 commented on PR #13060: URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6899295671
@SEZ9 I rechecked your latest questions against `7c272c5a` and reran the affected tests locally. - **F1/F2:** [Restore](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L243-L296) consumes the engine-assigned states; `validateRestoredStateOwner` and the modulo rejection are gone. [Parsing](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L362-L427) uses each state's own saved prefix, so historical mixed prefixes are accepted. The contiguous-assignment, mixed-prefix and repeated `3 → 4 → 3 → 4 → 3` regressions passed. - **F4:** Kafka [uses the stable index](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L103-L106), which is Spark's partition ID. Other sinks retain the task-attempt index. The [broker retry test](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaIT.java#L2399-L2460) passed with task IDs 99 and 100 sharing stable index 0: the abandoned record stayed invisible and `read_committed` progress resumed. - **F7:** The [option row and semantics](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/docs/en/connectors/sink/Kafka.md#L66-L108) describe the ID shape, Zeta/Flink stateful recovery and Spark's speculation restriction; ZH matches. **F3 still has a limit:** a writer with no retained state opens sequence 1, fencing its previous holder. Later abandoned transactions absent from checkpoint state can remain open until `transaction.timeout.ms`. The [fresh-writer path](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L319-L350) deliberately avoids scanning all history, which caused growing startup work across Spark micro-batches. Aligning with a checkpoint counter would not identify those missing transactions; Flink's adapter counters can also differ between writers. The Java 11 rerun passed 583 tests, including the three broker regressions, parallel Kafka batch jobs across Flink/Spark/Zeta, and Zeta's 3→2 restore. Real Spark 2.4 and 3.3 Doris 2PC tests also passed both batches; the old factories failed on batch two with `Label Already Exists`. Flink rescaling remains covered by model/adapter tests, not a real Flink rescale run. Is that documented F3 timeout boundary acceptable for this PR? If so, could you finish your re-review of this head? F5/F6 remains the agreed optimization follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
