SEZ9 commented on issue #13286: URL: https://github.com/apache/seatunnel/issues/13286#issuecomment-6936967256
Thanks for the detailed report and reproduction (Zeta on `dev`, Cassandra 4.1.1, driver 4.14.0, a 17 MiB row above the 16 MiB mutation limit). A job that ends as FINISHED with only ids 1, 3 (`async_write = true`) or only id 1 (`async_write = false`) in the table is silent data loss, so this is worth fixing. I have not verified the `CassandraSinkWriter` analysis against `dev` myself, so the three causes still need to be confirmed during review. The direction you propose sounds reasonable to me: - wait for in-flight async writes when `batch_size` is reached, in `prepareCommit()` and in `close()`, and fail the writer on the first error; - send the buffered synchronous batch at a checkpoint, so a failure fails the checkpoint instead of surfacing after it; - close the session in `close()` even when the flush fails. The `CassandraIT` cases for a rejected row with `async_write` on and off, the unit tests for the checkpoint flush, and the `incompatible-changes.md` entry would all be good to include, since jobs that lose rows today will fail instead. Since you are willing to submit the PR, please go ahead, and we can discuss the details in review. <!-- streview-comment:2396 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
