Hi Robin, Maybe this is where I have been getting confused - when I read the docs about the regex support for topics - I probably made a big assumption it was a server side thing. Is this not the case ? would explain why Ive not found much support for it lol
Cheers Gary > On 26 Mar 2018, at 16:18, Dmitriy Vsekhvalnov <dvsekhval...@gmail.com> wrote: > > Hi Gary, > > don't have experience with other go libs (they seems to be way younger), > but Sarama is quite low level, which is both at same time powerful and to > some extent more complicated to work with. > > With pure Sarama client you have to implement wildcard (or pattern based) > topic subscription yourself. In all clients in all languages it is usually > as simple as: > > - every X seconds (minutes, e.tc.) refresh metadata (see > Client.RefreshMetadata(..)) > - diff new metadata with old metadata = removed topics/added topics > - stop consuming for removed topics > - start consuming for new topics > > > > On Mon, Mar 26, 2018 at 11:58 AM, Gary Taylor <gary.tay...@hismessages.com> > wrote: > >> Hi Robin - thanks for that. I started off with sarama, but found it did >> not (seem to) support regex matching of the topics - I tried all sorts of >> ‘regex as strings’ as the topic - but it has to be a string according to >> the interface. >> >> I will dig deeper though and look at those other clients too >> >> Cheers >> >> Gary >> >>> On 26 Mar 2018, at 09:54, Robin Bartholdson < >> robin.barthold...@formulate.se> wrote: >>> >>> For go I would seriously consider using a pure go client, for the >> reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go < >> https://dave.cheney.net/2016/01/18/cgo-is-not-go>. >>> >>> There is now a wealth of pure go Kafka clients to choose from: >>> >>> - https://github.com/Shopify/sarama <https://github.com/Shopify/sarama> >>> - https://github.com/segmentio/kafka-go/ <https://github.com/segmentio/ >> kafka-go/> >>> - https://github.com/optiopay/kafka <https://github.com/optiopay/kafka> >>> >>> The functionality you are looking for is built into sarama-cluster, the >> package that provides consumer group functionality on top of sarama: >>> >>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 < >> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75> >>> >>> If you for some reason don’t want to use sarama-cluster, you can >> implement it yourself quite easily: Every now and then, as metadata is >> being refreshed, just cycle through the list of topics and determine if any >> of the topics are worth subscribing to: >>> >>> https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448 >> <https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448> >>> >>> For each of the clients these are the relevant methods to use to grab >> the list of topics: >>> >>> - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client < >> https://godoc.org/github.com/Shopify/sarama#Client> >>> - kafka-go: ReadPartitions(): https://godoc.org/github.com/ >> segmentio/kafka-go#Conn.ReadPartitions <https://godoc.org/github.com/ >> segmentio/kafka-go#Conn.ReadPartitions> >>> - optiopay/kafka: Metadata(): https://godoc.org/github.com/ >> optiopay/kafka#Broker.Metadata <https://godoc.org/github.com/ >> optiopay/kafka#Broker.Metadata> >>> >>> -Robin >>> >>> >>>> On 25 Mar 2018, at 09:46, Gary Taylor <garydavidtay...@gmail.com> >> wrote: >>>> >>>> Hi, >>>> This is a fairly generic question but has some specifics too >>>> >>>> Ill ask the specific first - I am trying to use golang to talk to kafka >> and it works, but a fairly important part of my application is to subscribe >> to information in many topics where the topic is matched server side and >> will include new topics added since the subscribe. I have read that kafka >> can do this, but I cannot see any way of asking it to do it via golang >> apart from with the library https://github.com/ >> confluentinc/confluent-kafka-go <https://github.com/ >> confluentinc/confluent-kafka-go> - which uses a library called librdkafka >> which I would have to compile and I am struggling as I am using an arm >> processor (long story behind that one - but I don’t think this solution >> will ever work with an arm processor). All other libraries seem to not >> offer this. Is this because of some underlying restriction in kafka or do >> you think it is going to be a case of keep hunting until I find one that >> does ? >>>> >>>> Then the more generic question - I get the feeling that Scala and Java >> are first class citizens when it comes to using kafka. Whilst I love Scala >> - I am learning golang and am more generally a ruby developer. Am I likely >> go get a ‘second class’ service with these other languages - with >> restrictions around zookeeper for example (I think I read something about >> having to have knowledge of which partition to subscribe to - where >> zookeeper would normally track this but there is no API for it) ? >>>> >>>> Many Thanks >>>> >>>> Gary Taylor >>> >> >>
