kfaraz commented on issue #18008: URL: https://github.com/apache/druid/issues/18008#issuecomment-2915829752
> It was indeed simpler to extend the single supervisor to read from multiple topics in the same cluster. You also get the benefit of auto-discovering new topics against a regex and don't have to make any changes to supervisor for ingestion to continue. Yeah, the regex is a neat trick! > You can also choose to write a different supervisor altogether that knows how to consume from topics from multiple clusters, reusing what code you can reuse. The task logic will remain the same as it but the supervisor will be different. I suppose a composite supervisor (internally made up of one or more Kafka supervisors) could be doable. But it would be fairly complicated to orchestrate this supervisor and would lead to the same complications down the line as a single Kafka supervisor would, if it were to read from multiple clusters. With the multi-topic feature already in place, I think the recommended approach should be "one supervisor per Kafka cluster, regardless of the number of topics". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
