rohankumardubey commented on issue #3635: URL: https://github.com/apache/iggy/issues/3635#issuecomment-5205588525
Thanks @hubcio , that makes the direction clear. Since backward compatibility isn’t required, I’ll base the proposal on changing the existing source FFI contract directly. The initial design will allow one in-flight batch and use this flow: 1. `poll()` returns the messages and staged state without advancing the cursor or deleting/marking rows. 2. The runtime sends the batch and waits for the producer confirmation. 3. After a successful send, the runtime persists the staged state. 4. The runtime ACKs the plugin. 5. The plugin commits its in-memory cursor and performs the staged deletes or marks. A send failure, NACK, or timeout will discard the staged work and cause the same batch to be read again. I’ll document each crash window and make sure it can result in redelivery or duplicates, but not missing rows. I’ll also cover the empty-batch state-save bug, the legacy server versus `server-ng` confirmation behavior from #3781, and how the acknowledgment handshake will limit the currently unbounded poll-to-send flow to one batch. For PostgreSQL CDC, I’ll look at replacing the destructive `pg_logical_slot_get_changes` polling with a peek-and-advance approach, so the slot is only consumed after acknowledgment. Once the design is approved, I’ll split the implementation into two PRs: 1. Source FFI and runtime acknowledgment support. 2. PostgreSQL migration with a deterministic kill-server regression test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
