For go I would seriously consider using a pure go client, for the reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
There is now a wealth of pure go Kafka clients to choose from: - https://github.com/Shopify/sarama <https://github.com/Shopify/sarama> - https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/kafka-go/> - https://github.com/optiopay/kafka <https://github.com/optiopay/kafka> The functionality you are looking for is built into sarama-cluster, the package that provides consumer group functionality on top of sarama: https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75> If you for some reason don’t want to use sarama-cluster, you can implement it yourself quite easily: Every now and then, as metadata is being refreshed, just cycle through the list of topics and determine if any of the topics are worth subscribing to: https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448 <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448> For each of the clients these are the relevant methods to use to grab the list of topics: - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <https://godoc.org/github.com/Shopify/sarama#Client> - kafka-go: ReadPartitions(): https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions> - optiopay/kafka: Metadata(): https://godoc.org/github.com/optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/optiopay/kafka#Broker.Metadata> -Robin > On 25 Mar 2018, at 09:46, Gary Taylor <garydavidtay...@gmail.com> wrote: > > Hi, > This is a fairly generic question but has some specifics too > > Ill ask the specific first - I am trying to use golang to talk to kafka and > it works, but a fairly important part of my application is to subscribe to > information in many topics where the topic is matched server side and will > include new topics added since the subscribe. I have read that kafka can do > this, but I cannot see any way of asking it to do it via golang apart from > with the library https://github.com/confluentinc/confluent-kafka-go > <https://github.com/confluentinc/confluent-kafka-go> - which uses a library > called librdkafka which I would have to compile and I am struggling as I am > using an arm processor (long story behind that one - but I don’t think this > solution will ever work with an arm processor). All other libraries seem to > not offer this. Is this because of some underlying restriction in kafka or > do you think it is going to be a case of keep hunting until I find one that > does ? > > Then the more generic question - I get the feeling that Scala and Java are > first class citizens when it comes to using kafka. Whilst I love Scala - I > am learning golang and am more generally a ruby developer. Am I likely go > get a ‘second class’ service with these other languages - with restrictions > around zookeeper for example (I think I read something about having to have > knowledge of which partition to subscribe to - where zookeeper would normally > track this but there is no API for it) ? > > Many Thanks > > Gary Taylor
