kfaraz commented on issue #18008: URL: https://github.com/apache/druid/issues/18008#issuecomment-2915281098
Had a brief discussion with @jtuglu-netflix on this. I agree with @abhishekrb19 in that separate supervisors would be a preferable approach for this. As the feature evolves, we would feel the need to have cluster-specific configs and keeping them in a single supervisor makes it more error prone and difficult to debug. (This also makes me wonder why we didn't do this when we wanted to implement multi-topic Kafka ingestion. Perhaps reading from multiple topics in a single supervisor was more straightforward. @abhishekagarwal87 would have more info on that.) Having multiple supervisors appending to the same datasource is not inherently wrong and is in fact the preferred approach. - We already support concurrent appends (the concurrent append and replace feature would come in handy here) - We would need to remove the constraint of 1-to-1 mapping from supervisor ID to datasource with back compat as pointed out by @abhishekrb19 - Persistence of datasource metadata would have to be against supervisor ID instead of datasource name - Realtime should probably just work as-is, or might need some minor tweak - Supervisor A would be completely agnostic of Supervisor B and they would also write to different segment IDs (thanks to the way segment IDs are built using the `taskAllocatorId`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
