zephyr45 commented on PR #13060: URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6613072913
Correction to my earlier shared-Spark compatibility assessment at `b9ed7c1297d9e1ca517f171c688641363a9ab912`: **Spark 3.3 streaming into Doris with `sink.enable-2pc=true` fails on the second micro-batch. Please hold merging while this regression is resolved.** My earlier compatibility assessment was incomplete. The index-mapping test, Kafka batch E2E, and documentation update did not validate consecutive Doris streaming batches. I reproduced this with a real Spark 3.3.0 Structured Streaming query, Java 11, and the repository-pinned `apache/doris:doris-all-in-one-2.1.0` image (server reports `doris-2.1.0-rc11-91efb6a43d`). The query uses the production `SparkSinkInjector` and Doris connector, one partition, and two JSON files introduced in separate batches. | Factory used | Result | | --- | --- | | Current PR factory | First batch commits. Second batch and its retry fail with `Label Already Exists`. The query aborts with only the first row visible. | | Pre-PR factory from `5598ed7f`, with all other classes unchanged at the PR head | Both batches commit and both rows are visible. | I repeated the comparison on a fresh Doris instance, verified the loaded factory classes and build artifact hashes, and queried the persisted IDs and values. The control isolates the factory change. The shared factory now supplies the same `partitionId` across batches. Each fresh Doris writer receives empty state and starts its checkpoint counter at zero, generating the same initial `<prefix>_<table>_<job>_<partition>_1` label. Doris rejects that label because the previous batch has finished. The factory already discarded `epochId` before this PR, and changing the writer index exposes that lifecycle limitation. [Factory](https://github.com/zephyr45/seatunnel/blob/b9ed7c1297d9e1ca517f171c688641363a9ab912/seatunnel-translation/seatunnel-translation-spark/seatunnel-translation-spark-3.3/src/main/java/org/apache/seatunnel/translation/spark/sink/write/SeaTunnelSparkDataWriterFactory.java#L67-L90) The fix needs to preserve Kafka retry fencing while providing safe Doris identities across separate batches, with regression coverage. This runtime result is specifically for Spark 3.3 and Doris 2PC. Spark 2.4 has the analogous factory change, and SelectDB has a related label-reuse concern in source, but I have not runtime-confirmed those paths. The green CI result remains accurate. Its coverage did not detect this case. I am correcting the PR description to make this validation gap and merge blocker explicit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
