zephyr45 commented on PR #13060: URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6656836630
@DanielLeens @SEZ9 The current head is [7c272c5a](https://github.com/zephyr45/seatunnel/commit/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5). The fork [Build 41960786230, attempt 2](https://github.com/zephyr45/seatunnel/actions/runs/41960786230/attempts/2) completed successfully. Attempt 1 had two setup/download errors: a Docker image download connection reset in Kudu on JDK 17, and a failed MySQL driver download in JDBC part 1 on JDK 11. I inspected both logs and reran only the failed jobs; both passed. Kafka integration jobs and all four unit-test OS/JDK combinations passed. The Spark fix in `9e4aad72` separates Kafka's retry-stable partition index from the historical task-attempt index used by other sinks, through the default `SinkWriter.Context#getStableSubtaskIndex()` method. Multi-table proxies preserve replica offsets. Kafka's namespace grammar, saved-state format and legacy fencing are unchanged. Doris retains its pre-PR retry behavior and the adapters' existing epoch handling. The latest commit fixes the two cosmetic review comments and changes no production code. Sorry for my incomplete earlier compatibility assessment. The [correction](https://github.com/apache/seatunnel/pull/13060#issuecomment-6613072913) remains accurate for the earlier head. With the fix, real Spark **3.3.0 and 2.4.6** streaming into the pinned Doris 2.1.0 image with 2PC enabled both commit the two consecutive batches. The original factories fail on batch two; independent SQL queries confirm the fixed runs contain both exact input rows. The production-Doris-label tests also fail against the original factories and pass with the fix. SelectDB has not been runtime-tested; its historical task-index behavior is restored in source. Here is the self-contained F1–F7 reconciliation: - **F1 — state redistribution/recovery revisions:** the connector consumes engine-assigned states without a modulo ownership check. The repeated Flink `3 -> 4 -> 3 -> 4 -> 3` regression failed on `351106062` and passed on `97516b977` and `b9ed7c129`; it also passed in the 18-test writer suite for the identity fix. The revision is read from the saved transaction ID by `parseRestoredTransaction`, incremented by `advanceWriterNamespace`, and persisted in the complete ID by `KafkaTransactionSender.snapshotState`. Recovery no longer relies on synchronized engine-local checkpoint counters. - **F2 — mixed generated prefixes:** every state is parsed relative to its own saved job prefix. `shouldRestoreMixedGeneratedPrefixesAfterScaleDown` passes. - **F3 — writers with no state:** opening sequence 1 fences its previous holder independently of engine checkpoint counters. `shouldNotWalkStatelessWriterTransactionHistory` and the broker bounded-initialization test pass. Later abandoned IDs absent from retained state still depend on Kafka's transaction timeout; that boundary is documented. - **F4 — Spark retries:** Kafka consumes the stable partition index, while other sinks retain the task index. Both adapter suites exercise retries and consecutive epochs. `KafkaIT.shouldFenceAbandonedAttemptWithStableIndex` uses different task IDs with one stable index and verifies abandoned data stays invisible while `read_committed` progress resumes. - **F5/F6 — legacy probing:** cross-writer legacy-prefix duplication remains an efficiency follow-up, not an implemented optimization. Safe deduplication needs per-prefix ownership and failover. Same-recovery restart-loop probe growth also remains a follow-up. - **F7 — documentation:** EN/ZH option rows describe the new ID shape, scope stateful recovery/rescaling to Zeta/Flink, and require `spark.speculation=false` for Spark. Migration, timeout boundaries and the separate Spark indexes are documented. Local Java 11 validation for the identity fix passed **573 targeted tests**: 411 API, 152 Kafka unit, 5 Spark adapter, 3 broker regressions and 2 packaged Spark/Kafka E2E. After the cosmetic commit, both complete Spark suites (5 tests), repository formatting and the full **329-module** `verify -DskipTests` passed again. The real Doris/Kafka runs were performed on the production-identical parent, not rerun for the cosmetic delta. The parser/prefix-contract documentation, removal of unreleased numeric namespace compatibility, and bounded expected-output wait in the batch test remain in place. The test-scoped Doris dependency is retained to exercise the production label generator; replacing it remains an optional follow-up. The Apache-side [PR signal](https://github.com/apache/seatunnel/actions/runs/41960794359) still reports `action_required`, and reviewer/maintainer acceptance remains pending. Those are separate from the successful fork build reported above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
