hubert-laserdata commented on issue #2540: URL: https://github.com/apache/iggy/issues/2540#issuecomment-4826149214
@realonbebeto single destination is the default model; iceberg's multi-table routing is the special case, not the template. the routing design in #3313 covers exactly this question, but source-side only. sink-side (one sink writing to N tables/indices) is explicitly out of scope there, parked under future as "sink-side multi-destination dispatch". it's a real gap though, and worth its own discussion rather than being improvised inside the redshift sink. one thing to flag up front so nobody treats it as a copy-paste of #3313: a source always writes to iggy, so the runtime owns a uniform Destination and does the dispatch. a sink writes to a backend-specific target (redshift table + s3 path, ES index, pg schema.table), so there's no universal destination type and the plugin has to own the physical write. a sink design would reuse the Route transform and the admission cap/allowlist, but move the dispatch seam plugin-side - and the commit()/state machinery from #3313 doesn't port, since sinks already have durable state in the consumer offset. so for the redshift sink right now: build it single-destination (consume one stream/topic, stage to one redshift target via s3, configurable iam/connection/bucket as the issue says). don't block on routing. if you'd like to drive it, opening a "per-message routing for sinks" discussion mirroring #3313 would be a great contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
