Hello,

We would like to expose a Kafka cluster running on one network to clients
that are running on other networks without having to have full routing
between the two networks. In this case these networks are in different AWS
accounts but the concept applies more widely. We would like to access Kafka
over a single (or very few) host names.

In addition we would like to filter incoming messages to enforce some level
of data quality and also impose some access control.

A solution we are looking into is to provide a Kafka protocol level proxy
that presents to clients as a single node Kafka cluster holding all the
topics and partitions of the cluster behind it. This proxy would be able to
operate in a load balanced cluster behind a single DNS entry and would also
be able to intercept and filter/alter messages as they passed through.

The advantages we see in this approach over the HTTP proxy is that it
presents the Kafka protocol whilst also meaning that we can use a typical
TCP level load balancer that it is easy to route connections to. This means
that we continue to use native Kafka clients.

Does anything like this already exist? Does anybody think it would useful?
Does anybody know of any reason it would be impossible (or a bad idea) to
do?

James Grant

Developer - Expedia Group

Reply via email to