rohankumardubey commented on issue #3635: URL: https://github.com/apache/iggy/issues/3635#issuecomment-5178076313
@hubcio , I reproduced the failure with `postgres_source`. I configured polling with `tracking_column = "id"`, stopped only the Iggy server, confirmed that its TCP port was refusing connections, and then inserted rows 1–3 into PostgreSQL. While Iggy remained offline, the connector state advanced to `tracking_offsets["test_messages"] = "3"` and `processed_rows = 3`. After restarting Iggy, the topic contained zero messages. Inserting row 4 afterward delivered only row 4, confirming that rows 1–3 had been skipped. One unexpected result is that the persisted runtime state also advanced while Iggy was offline, even though the current runtime saves state only after `producer.send()` reports success. Should that send/disconnection behavior be investigated as part of #3635, or tracked separately? For the FFI design, I propose retaining the existing source symbols for compatibility and adding an optional/versioned acknowledgment path with one in-flight batch. Source implementations such as PostgreSQL would stage cursor updates and destructive operations until the runtime acknowledges delivery. The exact ordering between Iggy delivery, plugin commit, and runtime state persistence needs agreement because it determines the remaining duplicate/data-loss windows. If this direction is acceptable, I can write the detailed API and failure-state sequence before making implementation changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
