tusharagrahari opened a new issue, #3635: URL: https://github.com/apache/iggy/issues/3635
### Bug description ## Description The connector runtime gives source plugins no way to know whether the messages they produced were actually delivered to Iggy. The runtime polls the source, sends the batch via the producer, and persists the source's saved state, but the send result is never fed back to the plugin, and offset advancement is not conditional on a successful send. This is the source-side counterpart of #2927 (sink `consume()` return value discarded) and #2928 (offsets committed before sink processing). Same root cause: no delivery acknowledgment across the runtime/plugin boundary, just in the opposite direction. Raised during review of the MySQL source (#3445), where it shows up twice: 1. In `delete` tracking mode, rows are deleted (or marked) in the source database as part of producing the batch before the runtime has sent anything. If the send fails, the row is gone from MySQL and never reached Iggy. Permanent data loss. 2. The tracking offset advances once the batch is handed to the runtime. A send failure doesn't roll it back, so the next poll starts past the failed batch. `postgres_source` has the same behavior — this is not specific to the MySQL connector. ## Impact At-least-once delivery isn't achievable for source connectors right now — same conclusion as #2928 on the sink side. For sources doing a destructive read (delete mode), this isn't "duplicates possible" anymore, it's actual data loss: the record's gone from the source DB and never landed in Iggy, and nothing gets logged about it. Affects any polling source that tracks its own position, not just the SQL connectors. ## Related - #2927 — sink consume() return value discarded (same gap, sink direction) - #2928 — offsets committed before sink processing - #2940 — replay-safe progress contract discussion - #3445 — MySQL polling source PR where this was flagged in review ### Affected area / component Connectors ### Deployment None ### Versions _No response_ ### Hardware / environment _No response_ ### Sample code _No response_ ### Logs _No response_ ### Iggy server config _No response_ ### Reproduction _No response_ ### Contribution - [ ] I'm willing to submit a pull request to fix this bug ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
