kriti-sc commented on issue #2927: URL: https://github.com/apache/iggy/issues/2927#issuecomment-4699245941
Been thinking about this and adjacent issues of DLQ and offset auto-commit (#2928). This actually requires a decision around responsibility -- **Essentially, a sink will fail and require some action. Who is going to be responsible for the action?** For ex. who is responsible for retries - sink or runtime? who is responsible for routing to DLQ - sink or runtime? Looked at it from this perspective, the sink should be responsible for what to do with messages on failure because sink has most context on the reason for the failure. Sinks has context on semantics it wants to maintain (at least once, at most once), what errors are retryable and what errors need to be routed to the DLQ. The sink can expose this information to the runtime, for ex. exposing a failure and a retryable flag. The downside is the FFI signature changing. Having the runtime supporting routing to DLQ would also require making the FFI boundary richer to expose messages that should go to the DLQ. Finally, uniformity becomes a concern when multiple sink implement the same behaviours. This can be achieved using utilities in the SDK such as the retry utility. Therefore, exposing sink errors to the runtime should remain "cosmetic" for the purpose of observability only. cc @hubcio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
