[jira] [Commented] (KAFKA-10219) KStream API support for multiple cluster broker
[ https://issues.apache.org/jira/browse/KAFKA-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148954#comment-17148954 ] Sachin Kurle commented on KAFKA-10219: -- I have simple use-case where I read off source topic from cluster A and write to target topic on cluster B I am not sure on how transaction going across multiple cluster affects the exactly once.. even in case of single cluster my kstream app deployed on host server have to make network call to kafka cluster change log topic to get transaction details and update the offset topic. May be I am not fully aware of the internal working of kafka stateful transaction in case of Exactly-once > KStream API support for multiple cluster broker > --- > > Key: KAFKA-10219 > URL: https://issues.apache.org/jira/browse/KAFKA-10219 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Sachin Kurle >Priority: Major > > we are trying to consume from cluster A broker from KStream api and produce > to cluster B broker.. we have configuration as boot strap server in consumer > and producer configuration but kstream api is picking randomly bootstrap > server cluster A or B -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10219) KStream API support for multiple cluster broker
[ https://issues.apache.org/jira/browse/KAFKA-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148818#comment-17148818 ] Boyang Chen commented on KAFKA-10219: - Thanks for the proposal, the use case you proposed is reasonable. However, we need to better clarify the feature we are going to introduce and the challenges we are facing, such as: 1. What does "multiple clusters" suggest? Do we support all input topics in cluster A and all output topics in cluster B, or a mixing of topics in random cluster A, B, C which needs to be automatically detected by Streams? 2. How do we allocate internal topics? Which cluster should the changelog/repartition topics go to, input topic cluster, or the output one? 3. How do we support Exactly-once? Right now the entire framework assumes a single cluster context. When switching to multiple cluster, we could no longer guarantee exactly-once because we may spam our transaction across multiple clusters, and we don't have a centralized coordinator to track the progress. > KStream API support for multiple cluster broker > --- > > Key: KAFKA-10219 > URL: https://issues.apache.org/jira/browse/KAFKA-10219 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Sachin Kurle >Priority: Major > > we are trying to consume from cluster A broker from KStream api and produce > to cluster B broker.. we have configuration as boot strap server in consumer > and producer configuration but kstream api is picking randomly bootstrap > server cluster A or B -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10219) KStream API support for multiple cluster broker
[
https://issues.apache.org/jira/browse/KAFKA-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148407#comment-17148407
]
Bruno Cadonna commented on KAFKA-10219:
---
[~sachinkurle] The current Kafka Streams documentation states
??Kafka Streams applications can only communicate with a single Kafka cluster
specified by this config value.??
The mentioned config is {{bootstrap.servers.}}
See
[https://kafka.apache.org/25/documentation/streams/developer-guide/config-streams.html#bootstrap-servers]
> KStream API support for multiple cluster broker
> ---
>
> Key: KAFKA-10219
> URL: https://issues.apache.org/jira/browse/KAFKA-10219
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Sachin Kurle
>Priority: Major
>
> we are trying to consume from cluster A broker from KStream api and produce
> to cluster B broker.. we have configuration as boot strap server in consumer
> and producer configuration but kstream api is picking randomly bootstrap
> server cluster A or B
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
