mattp5657 opened a new issue, #3847:
URL: https://github.com/apache/iggy/issues/3847
## Description
Neither the `Source` nor `Sink` trait gives a plugin a way to learn that a
batch was durably delivered before its read/write position advances, so
both sides can silently lose data:
- **Sinks:** the consumer-group offset commits when a batch is polled
(`AutoCommitWhen::PollingMessages`,
`core/connectors/runtime/src/sink.rs:522`), before `consume()` runs. A
failed downstream write still advances the read position. Shared runtime
code, so it affects all 13 sink plugins identically (`clickhouse_sink`,
`delta_sink`, `doris_sink`, `elasticsearch_sink`, `http_sink`,
`iceberg_sink`, `influxdb_sink`, `mongodb_sink`, `postgres_sink`,
`quickwit_sink`, `s3_sink`, `stdout_sink`, `surrealdb_sink`).
- **Sources:** `elasticsearch_source` and `influxdb_source` advance their
in-memory cursor during fetch, before `producer.send()` is confirmed
(`elasticsearch_source/src/lib.rs:434-435`,
`influxdb_source/src/lib.rs:855`). The runtime only persists state after
a successful send, so a crash is survivable — but a send failure without
a crash isn't: the plugin is never told, and the next successful poll
bakes the gap into persisted state permanently.
`random_source` is unaffected (no real external cursor). `postgres_source`
CDC mode has the same root cause but is tracked in a separate issue, since
its fix also touches the state-persistence contract.
Net effect: at-most-once delivery where at-least-once is expected, with no
error, duplicate, or gap signal.
## Affected area / component
Connectors
## Proposed solution
- **Sinks:** switch the consumer to `AutoCommit::Disabled` and call
`consumer.store_offset()` explicitly after `consume()` succeeds.
Runtime-only change, no plugin rebuild needed.
- **Sources:** add a confirmation hook to the `Source` trait so plugins
defer their cursor advance until the runtime confirms delivery. This is
an FFI/SDK surface change and needs its own design pass before
implementation.
## Alternatives considered
No response
## Contribution
I'm willing to submit a pull request to fix this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
