Hello, We would like to expose a Kafka cluster running on one network to clients that are running on other networks without having to have full routing between the two networks. In this case these networks are in different AWS accounts but the concept applies more widely. We would like to access Kafka over a single (or very few) host names.
In addition we would like to filter incoming messages to enforce some level of data quality and also impose some access control. A solution we are looking into is to provide a Kafka protocol level proxy that presents to clients as a single node Kafka cluster holding all the topics and partitions of the cluster behind it. This proxy would be able to operate in a load balanced cluster behind a single DNS entry and would also be able to intercept and filter/alter messages as they passed through. The advantages we see in this approach over the HTTP proxy is that it presents the Kafka protocol whilst also meaning that we can use a typical TCP level load balancer that it is easy to route connections to. This means that we continue to use native Kafka clients. Does anything like this already exist? Does anybody think it would useful? Does anybody know of any reason it would be impossible (or a bad idea) to do? James Grant Developer - Expedia Group
