Re: Developing with kafka and other non jvm languages

[Gary Taylor]

Hi Robin,
   Maybe this is where I have been getting confused - when I read the docs 
about the regex support for topics - I probably made a big assumption it was a 
server side thing.  Is this not the case ? would explain why Ive not found much 
support for it lol

Cheers

Gary

> On 26 Mar 2018, at 16:18, Dmitriy Vsekhvalnov  wrote:
> 
> Hi Gary,
> 
> don't have experience with other go libs (they seems to be way younger),
> but Sarama is quite low level, which is both at same time powerful and to
> some extent more complicated to work with.
> 
> With pure Sarama client you have to implement wildcard (or pattern based)
> topic subscription yourself. In all clients in all languages it is usually
> as simple as:
> 
> - every X seconds (minutes, e.tc.) refresh metadata (see
> Client.RefreshMetadata(..))
> - diff new metadata with old metadata = removed topics/added topics
> - stop consuming for removed topics
> - start consuming for new topics
> 
> 
> 
> On Mon, Mar 26, 2018 at 11:58 AM, Gary Taylor 
> wrote:
> 
>> Hi Robin - thanks for that.  I started off with sarama, but found it did
>> not (seem to) support regex matching of the topics - I tried all sorts of
>> ‘regex as strings’ as the topic - but it has to be a string according to
>> the interface.
>> 
>> I will dig deeper though and look at those other clients too
>> 
>> Cheers
>> 
>> Gary
>> 
>>> On 26 Mar 2018, at 09:54, Robin Bartholdson <
>> robin.barthold...@formulate.se> wrote:
>>> 
>>> For go I would seriously consider using a pure go client, for the
>> reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <
>> https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
>>> 
>>> There is now a wealth of pure go Kafka clients to choose from:
>>> 
>>> - https://github.com/Shopify/sarama 
>>> - https://github.com/segmentio/kafka-go/ > kafka-go/>
>>> - https://github.com/optiopay/kafka 
>>> 
>>> The functionality you are looking for is built into sarama-cluster, the
>> package that provides consumer group functionality on top of sarama:
>>> 
>>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <
>> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
>>> 
>>> If you for some reason don’t want to use sarama-cluster, you can
>> implement it yourself quite easily: Every now and then, as metadata is
>> being refreshed, just cycle through the list of topics and determine if any
>> of the topics are worth subscribing to:
>>> 
>>> https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448
>> 
>>> 
>>> For each of the clients these are the relevant methods to use to grab
>> the list of topics:
>>> 
>>> - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <
>> https://godoc.org/github.com/Shopify/sarama#Client>
>>> - kafka-go: ReadPartitions(): https://godoc.org/github.com/
>> segmentio/kafka-go#Conn.ReadPartitions > segmentio/kafka-go#Conn.ReadPartitions>
>>> - optiopay/kafka: Metadata(): https://godoc.org/github.com/
>> optiopay/kafka#Broker.Metadata > optiopay/kafka#Broker.Metadata>
>>> 
>>> -Robin
>>> 
>>> 
 On 25 Mar 2018, at 09:46, Gary Taylor 
>> wrote:
 
 Hi,
 This is a fairly generic question but has some specifics too
 
 Ill ask the specific first - I am trying to use golang to talk to kafka
>> and it works, but a fairly important part of my application is to subscribe
>> to information in many topics where the topic is matched server side and
>> will include new topics added since the subscribe.  I have read that kafka
>> can do this, but I cannot see any way of asking it to do it via golang
>> apart from with the library https://github.com/
>> confluentinc/confluent-kafka-go > confluentinc/confluent-kafka-go> - which uses a library called librdkafka
>> which I would have to compile and I am struggling as I am using an arm
>> processor (long story behind that one - but I don’t think this solution
>> will ever work with an arm processor).  All other libraries seem to not
>> offer this.  Is this because of some underlying restriction in kafka or do
>> you think it is going to be a case of keep hunting until I find one that
>> does ?
 
 Then the more generic question - I get the feeling that Scala and Java
>> are first class citizens when it comes to using kafka.  Whilst I love Scala
>> - I am learning golang and am more generally a ruby developer.  Am I likely
>> go get a ‘second class’ service with these other languages - with
>> restrictions around zookeeper for example (I think I read something about
>> having to have knowledge of which partition to subscribe to - where
>> zookeeper would normally track this 

Re: Developing with kafka and other non jvm languages

[Dmitriy Vsekhvalnov]

Hi Gary,

don't have experience with other go libs (they seems to be way younger),
but Sarama is quite low level, which is both at same time powerful and to
some extent more complicated to work with.

With pure Sarama client you have to implement wildcard (or pattern based)
topic subscription yourself. In all clients in all languages it is usually
as simple as:

 - every X seconds (minutes, e.tc.) refresh metadata (see
Client.RefreshMetadata(..))
 - diff new metadata with old metadata = removed topics/added topics
 - stop consuming for removed topics
 - start consuming for new topics



On Mon, Mar 26, 2018 at 11:58 AM, Gary Taylor 
wrote:

> Hi Robin - thanks for that.  I started off with sarama, but found it did
> not (seem to) support regex matching of the topics - I tried all sorts of
> ‘regex as strings’ as the topic - but it has to be a string according to
> the interface.
>
> I will dig deeper though and look at those other clients too
>
> Cheers
>
> Gary
>
> > On 26 Mar 2018, at 09:54, Robin Bartholdson <
> robin.barthold...@formulate.se> wrote:
> >
> > For go I would seriously consider using a pure go client, for the
> reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go <
> https://dave.cheney.net/2016/01/18/cgo-is-not-go>.
> >
> > There is now a wealth of pure go Kafka clients to choose from:
> >
> > - https://github.com/Shopify/sarama 
> > - https://github.com/segmentio/kafka-go/  kafka-go/>
> > - https://github.com/optiopay/kafka 
> >
> > The functionality you are looking for is built into sarama-cluster, the
> package that provides consumer group functionality on top of sarama:
> >
> > https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 <
> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75>
> >
> > If you for some reason don’t want to use sarama-cluster, you can
> implement it yourself quite easily: Every now and then, as metadata is
> being refreshed, just cycle through the list of topics and determine if any
> of the topics are worth subscribing to:
> >
> > https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448
> 
> >
> > For each of the clients these are the relevant methods to use to grab
> the list of topics:
> >
> > - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client <
> https://godoc.org/github.com/Shopify/sarama#Client>
> > - kafka-go: ReadPartitions(): https://godoc.org/github.com/
> segmentio/kafka-go#Conn.ReadPartitions  segmentio/kafka-go#Conn.ReadPartitions>
> > - optiopay/kafka: Metadata(): https://godoc.org/github.com/
> optiopay/kafka#Broker.Metadata  optiopay/kafka#Broker.Metadata>
> >
> > -Robin
> >
> >
> >> On 25 Mar 2018, at 09:46, Gary Taylor 
> wrote:
> >>
> >> Hi,
> >> This is a fairly generic question but has some specifics too
> >>
> >> Ill ask the specific first - I am trying to use golang to talk to kafka
> and it works, but a fairly important part of my application is to subscribe
> to information in many topics where the topic is matched server side and
> will include new topics added since the subscribe.  I have read that kafka
> can do this, but I cannot see any way of asking it to do it via golang
> apart from with the library https://github.com/
> confluentinc/confluent-kafka-go  confluentinc/confluent-kafka-go> - which uses a library called librdkafka
> which I would have to compile and I am struggling as I am using an arm
> processor (long story behind that one - but I don’t think this solution
> will ever work with an arm processor).  All other libraries seem to not
> offer this.  Is this because of some underlying restriction in kafka or do
> you think it is going to be a case of keep hunting until I find one that
> does ?
> >>
> >> Then the more generic question - I get the feeling that Scala and Java
> are first class citizens when it comes to using kafka.  Whilst I love Scala
> - I am learning golang and am more generally a ruby developer.  Am I likely
> go get a ‘second class’ service with these other languages - with
> restrictions around zookeeper for example (I think I read something about
> having to have knowledge of which partition to subscribe to - where
> zookeeper would normally track this but there is no API for it) ?
> >>
> >> Many Thanks
> >>
> >> Gary Taylor
> >
>
>

Re: Developing with kafka and other non jvm languages

[Gary Taylor]

Hi Robin - thanks for that.  I started off with sarama, but found it did not 
(seem to) support regex matching of the topics - I tried all sorts of ‘regex as 
strings’ as the topic - but it has to be a string according to the interface.

I will dig deeper though and look at those other clients too

Cheers

Gary

> On 26 Mar 2018, at 09:54, Robin Bartholdson  
> wrote:
> 
> For go I would seriously consider using a pure go client, for the reasons 
> listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go 
> .
> 
> There is now a wealth of pure go Kafka clients to choose from:
> 
> - https://github.com/Shopify/sarama 
> - https://github.com/segmentio/kafka-go/ 
> 
> - https://github.com/optiopay/kafka 
> 
> The functionality you are looking for is built into sarama-cluster, the 
> package that provides consumer group functionality on top of sarama:
> 
> https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 
> 
> 
> If you for some reason don’t want to use sarama-cluster, you can implement it 
> yourself quite easily: Every now and then, as metadata is being refreshed, 
> just cycle through the list of topics and determine if any of the topics are 
> worth subscribing to:
> 
> https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448 
> 
> 
> For each of the clients these are the relevant methods to use to grab the 
> list of topics:
> 
> - sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client 
> 
> - kafka-go: ReadPartitions(): 
> https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions 
> 
> - optiopay/kafka: Metadata(): 
> https://godoc.org/github.com/optiopay/kafka#Broker.Metadata 
> 
> 
> -Robin
> 
> 
>> On 25 Mar 2018, at 09:46, Gary Taylor  wrote:
>> 
>> Hi,
>> This is a fairly generic question but has some specifics too
>> 
>> Ill ask the specific first - I am trying to use golang to talk to kafka and 
>> it works, but a fairly important part of my application is to subscribe to 
>> information in many topics where the topic is matched server side and will 
>> include new topics added since the subscribe.  I have read that kafka can do 
>> this, but I cannot see any way of asking it to do it via golang apart from 
>> with the library https://github.com/confluentinc/confluent-kafka-go 
>>  - which uses a library 
>> called librdkafka which I would have to compile and I am struggling as I am 
>> using an arm processor (long story behind that one - but I don’t think this 
>> solution will ever work with an arm processor).  All other libraries seem to 
>> not offer this.  Is this because of some underlying restriction in kafka or 
>> do you think it is going to be a case of keep hunting until I find one that 
>> does ?
>> 
>> Then the more generic question - I get the feeling that Scala and Java are 
>> first class citizens when it comes to using kafka.  Whilst I love Scala - I 
>> am learning golang and am more generally a ruby developer.  Am I likely go 
>> get a ‘second class’ service with these other languages - with restrictions 
>> around zookeeper for example (I think I read something about having to have 
>> knowledge of which partition to subscribe to - where zookeeper would 
>> normally track this but there is no API for it) ?
>> 
>> Many Thanks
>> 
>> Gary Taylor
> 

Re: Developing with kafka and other non jvm languages

[Robin Bartholdson]

For go I would seriously consider using a pure go client, for the reasons 
listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go 
.

There is now a wealth of pure go Kafka clients to choose from:

- https://github.com/Shopify/sarama 
- https://github.com/segmentio/kafka-go/ 

- https://github.com/optiopay/kafka 

The functionality you are looking for is built into sarama-cluster, the package 
that provides consumer group functionality on top of sarama:

https://github.com/bsm/sarama-cluster/blob/master/config.go#L69-L75 


If you for some reason don’t want to use sarama-cluster, you can implement it 
yourself quite easily: Every now and then, as metadata is being refreshed, just 
cycle through the list of topics and determine if any of the topics are worth 
subscribing to:

https://github.com/bsm/sarama-cluster/blob/master/consumer.go#L421-L448 


For each of the clients these are the relevant methods to use to grab the list 
of topics:

- sarama: Topics(): https://godoc.org/github.com/Shopify/sarama#Client 

- kafka-go: ReadPartitions(): 
https://godoc.org/github.com/segmentio/kafka-go#Conn.ReadPartitions 

- optiopay/kafka: Metadata(): 
https://godoc.org/github.com/optiopay/kafka#Broker.Metadata 


-Robin


> On 25 Mar 2018, at 09:46, Gary Taylor  wrote:
> 
> Hi,
>  This is a fairly generic question but has some specifics too
> 
> Ill ask the specific first - I am trying to use golang to talk to kafka and 
> it works, but a fairly important part of my application is to subscribe to 
> information in many topics where the topic is matched server side and will 
> include new topics added since the subscribe.  I have read that kafka can do 
> this, but I cannot see any way of asking it to do it via golang apart from 
> with the library https://github.com/confluentinc/confluent-kafka-go 
>  - which uses a library 
> called librdkafka which I would have to compile and I am struggling as I am 
> using an arm processor (long story behind that one - but I don’t think this 
> solution will ever work with an arm processor).  All other libraries seem to 
> not offer this.  Is this because of some underlying restriction in kafka or 
> do you think it is going to be a case of keep hunting until I find one that 
> does ?
> 
> Then the more generic question - I get the feeling that Scala and Java are 
> first class citizens when it comes to using kafka.  Whilst I love Scala - I 
> am learning golang and am more generally a ruby developer.  Am I likely go 
> get a ‘second class’ service with these other languages - with restrictions 
> around zookeeper for example (I think I read something about having to have 
> knowledge of which partition to subscribe to - where zookeeper would normally 
> track this but there is no API for it) ?
> 
> Many Thanks
> 
> Gary Taylor

Re: Developing with kafka and other non jvm languages

[Gary Taylor]

Hi Matthias,

  Thanks for the reply.  Ill start hunting around to see what I can find with 
regards to clients.  I had just looked at the 2 most popular ones but you never 
know, the features I want may be in another library

Cheers

Gary

> On 25 Mar 2018, at 20:24, Matthias J. Sax  wrote:
> 
> Gary,
> 
> The Apache Kafka project itself, only maintains Java clients -- thus,
> your are right that those are the primary and best supported clients.
> (Kafka used to have Scala clients, but those are all deprecated now and
> will be removed eventually.)
> 
> Clients in other languages are not part of Apache Kafka project itself
> but developed by third parties. Thus, the quality and feature set of
> those clients can vary. A list of known third party clients is
> maintained in the Kafka Wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Clients
> 
> If a client does not support a feature, it's a client limitation only.
> The Kafka protocol is a binary protocol and thus it's up to the client
> to (fully) implement the protocol -- the broker does not even know in
> which language a client is written. (cf
> https://kafka.apache.org/protocol) Thus, you should open feature
> requests for clients to support pattern subscription if it's missing and
> you need it.
> 
> If you need help to get librdkafka to compile on arm, you might want to
> reach out to on Github (https://github.com/edenhill/librdkafka) or
> Confluent mailing list
> (https://groups.google.com/forum/#!forum/confluent-platform) or
> Confluent Slack (https://launchpass.com/confluentcommunity).
> 
> Hope this helps!
> 
> 
> -Matthias
> 
> 
> 
> 
> On 3/25/18 12:46 AM, Gary Taylor wrote:
>> Hi,
>>  This is a fairly generic question but has some specifics too
>> 
>> Ill ask the specific first - I am trying to use golang to talk to kafka and 
>> it works, but a fairly important part of my application is to subscribe to 
>> information in many topics where the topic is matched server side and will 
>> include new topics added since the subscribe.  I have read that kafka can do 
>> this, but I cannot see any way of asking it to do it via golang apart from 
>> with the library https://github.com/confluentinc/confluent-kafka-go 
>>  - which uses a library 
>> called librdkafka which I would have to compile and I am struggling as I am 
>> using an arm processor (long story behind that one - but I don’t think this 
>> solution will ever work with an arm processor).  All other libraries seem to 
>> not offer this.  Is this because of some underlying restriction in kafka or 
>> do you think it is going to be a case of keep hunting until I find one that 
>> does ?
>> 
>> Then the more generic question - I get the feeling that Scala and Java are 
>> first class citizens when it comes to using kafka.  Whilst I love Scala - I 
>> am learning golang and am more generally a ruby developer.  Am I likely go 
>> get a ‘second class’ service with these other languages - with restrictions 
>> around zookeeper for example (I think I read something about having to have 
>> knowledge of which partition to subscribe to - where zookeeper would 
>> normally track this but there is no API for it) ?
>> 
>> Many Thanks
>> 
>> Gary Taylor
>> 
> 

Re: Developing with kafka and other non jvm languages

[Matthias J. Sax]

Gary,

The Apache Kafka project itself, only maintains Java clients -- thus,
your are right that those are the primary and best supported clients.
(Kafka used to have Scala clients, but those are all deprecated now and
will be removed eventually.)

Clients in other languages are not part of Apache Kafka project itself
but developed by third parties. Thus, the quality and feature set of
those clients can vary. A list of known third party clients is
maintained in the Kafka Wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Clients

If a client does not support a feature, it's a client limitation only.
The Kafka protocol is a binary protocol and thus it's up to the client
to (fully) implement the protocol -- the broker does not even know in
which language a client is written. (cf
https://kafka.apache.org/protocol) Thus, you should open feature
requests for clients to support pattern subscription if it's missing and
you need it.

If you need help to get librdkafka to compile on arm, you might want to
reach out to on Github (https://github.com/edenhill/librdkafka) or
Confluent mailing list
(https://groups.google.com/forum/#!forum/confluent-platform) or
Confluent Slack (https://launchpass.com/confluentcommunity).

Hope this helps!


-Matthias




On 3/25/18 12:46 AM, Gary Taylor wrote:
> Hi,
>   This is a fairly generic question but has some specifics too
> 
> Ill ask the specific first - I am trying to use golang to talk to kafka and 
> it works, but a fairly important part of my application is to subscribe to 
> information in many topics where the topic is matched server side and will 
> include new topics added since the subscribe.  I have read that kafka can do 
> this, but I cannot see any way of asking it to do it via golang apart from 
> with the library https://github.com/confluentinc/confluent-kafka-go 
>  - which uses a library 
> called librdkafka which I would have to compile and I am struggling as I am 
> using an arm processor (long story behind that one - but I don’t think this 
> solution will ever work with an arm processor).  All other libraries seem to 
> not offer this.  Is this because of some underlying restriction in kafka or 
> do you think it is going to be a case of keep hunting until I find one that 
> does ?
> 
> Then the more generic question - I get the feeling that Scala and Java are 
> first class citizens when it comes to using kafka.  Whilst I love Scala - I 
> am learning golang and am more generally a ruby developer.  Am I likely go 
> get a ‘second class’ service with these other languages - with restrictions 
> around zookeeper for example (I think I read something about having to have 
> knowledge of which partition to subscribe to - where zookeeper would normally 
> track this but there is no API for it) ?
> 
> Many Thanks
> 
> Gary Taylor
> 



signature.asc
Description: OpenPGP digital signature