[
https://issues.apache.org/jira/browse/KAFKA-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100399#comment-16100399
]
Ewen Cheslack-Postava commented on KAFKA-5635:
----------------------------------------------
bq. Kafka connect currently uses kafka clients which directly connect to kafka
brokers.
bq. In a use case, wherein I have many kafka connect [producers] running
remotely its a challenge to configure broker information on every connect agent.
It doesn't seem like using a REST proxy would help here, you still need to
configure *that* for all workers. The common solution to simplify this is to
use a load balancer with a single, fixed address as your bootstrap servers.
Then you have a single location where you need to maintain a list of broker
addresses.
bq. Also, in case of IP change [upgrade or cluster re-creation], we need to
update every remote connect configuration.
See the above solution, which also addresses this concern.
bq. If kafka connect source connectors talk to ReST endpoint then client is
unaware of broker details. This way we can transparently upgrade / re-create
kafka cluster as long as ReST endpoint remains same.
This would be unlikely to be implemented as part of the core Kafka project
since REST proxies are 3rd party. There is no standard REST proxy API for Kafka
that we could rely on the protocol for.
> KIP-181 Kafka-Connect integrate with kafka ReST Proxy
> -----------------------------------------------------
>
> Key: KAFKA-5635
> URL: https://issues.apache.org/jira/browse/KAFKA-5635
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Dhananjay Patkar
> Labels: features, newbie
>
> Kafka connect currently uses kafka clients which directly connect to kafka
> brokers.
> In a use case, wherein I have many kafka connect [producers] running remotely
> its a challenge to configure broker information on every connect agent.
> Also, in case of IP change [upgrade or cluster re-creation], we need to
> update every remote connect configuration.
> If kafka connect source connectors talk to ReST endpoint then client is
> unaware of broker details. This way we can transparently upgrade / re-create
> kafka cluster as long as ReST endpoint remains same.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
