Re: Developing with kafka and other non jvm languages

Hi Matthias, Thanks for the reply. Ill start hunting around to see what I can find with regards to clients. I had just looked at the 2 most popular ones but you never know, the features I want may be in another library Cheers Gary > On 25 Mar 2018, at 20:24, Matthias J. Sax

Re: Developing with kafka and other non jvm languages

Hi Robin - thanks for that. I started off with sarama, but found it did not (seem to) support regex matching of the topics - I tried all sorts of ‘regex as strings’ as the topic - but it has to be a string according to the interface. I will dig deeper though and look at those other clients too

Re: Developing with kafka and other non jvm languages

For go I would seriously consider using a pure go client, for the reasons listed in https://dave.cheney.net/2016/01/18/cgo-is-not-go . There is now a wealth of pure go Kafka clients to choose from: - https://github.com/Shopify/sarama

Re: Kafka Mirrormaker issue

I’ve had similar problems, but I don’t have an explanation for ya :/ On Sun, Mar 25, 2018 at 12:19 PM, Siva A wrote: > Hi, > > We have 3 nodes Kafka cluster(0.10.0.1) and its mirroring the data from > another 3 node cluster of same Kafka version. > Both the clusters

RE: Kafka & ZooKeeper IDs

Hi Vigensh, There is no restriction on ID. It can be any unique number. broker.id=0 broker.id=1 broker.id=2 Thanks Uttam -Original Message- From: Vignesh [mailto:davidviki...@gmail.com] Sent: Monday, March 26, 2018 10:31 AM To: users@kafka.apache.org Subject: Kafka &

Apache Kafka / Spark Integration - Exception - The server disconnected before a response was received.

Hi: I am working with spark 2.2.1 and spark kafka 0.10 client integration with Kafka brokers using 0.11. I get the exception - org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received - when the application is trying to write to a topic. This

Re: Kafka & ZooKeeper IDs

Hi Anand, Thanks.. I was asking about linux/unix ids these kafka & zookeeper processes are running. On Mon, Mar 26, 2018 at 11:16 AM, Anand, Uttam wrote: > Hi Vigensh, > > > > There is no restriction on ID. It can be any unique number. > > > > broker.id=0 > > broker.id=1 >

Re: Apache Kafka / Spark Integration - Exception - The server disconnected before a response was received.

Can you post the stack trace for NetworkException (pastebin) ? Please also check the broker logs to see if there was some clue around the time this happened. Thanks On Mon, Mar 26, 2018 at 9:30 AM, M Singh wrote: > Hi: > I am working with spark 2.2.1 and spark

Fwd: ListOffsets parameters

Hi users, I'm the maintainer of the PyKafka library and I'm working on improving its support for the ListOffsets API. Some questions: Kafka version: 1.0.0 I'm using this documentation for reference.

Re: Apache Kafka / Spark Integration - Exception - The server disconnected before a response was received.

Hi Ted: Here is the exception trace (Note - The exception is occuring in the kafka spark writer class). I will try to check broker logs.  Is there anything specific I should look for ? Driver stacktrace: at

Re: Developing with kafka and other non jvm languages

Hi Gary, don't have experience with other go libs (they seems to be way younger), but Sarama is quite low level, which is both at same time powerful and to some extent more complicated to work with. With pure Sarama client you have to implement wildcard (or pattern based) topic subscription

Am I using the state store appropriately?

​We've started experimenting with Kafka to see if it can be used to aggregate our application data. I think our use case is a match for Kafka streams, but we aren't sure if we are using the tool correctly. The proof of concept we've built seems to be working as designed, I'm not sure that we are

Kafka & ZooKeeper IDs

Hello Kafka Users, I'm setting up 3 node kafka cluster in RHEL I don't see a documentation for what ID these services should be running under. I'm creating unique Id's for Kafka & Zookeeper .. each on their unique nodes. Does it has any restriction or any mandatory IDs it should be

Re: Am I using the state store appropriately?

Hi Jason, Your design sounds right to me (I presume you are using PAPI not the Streams DSL), that you are reading in one stream, calling transform() on the stream in which an state store is associated with the operator. Since your update logic seems to be only key-dependent and hence can be

Kafka replication

Hi all! Ok I was playing those days with replication for a specific purpose:having a failover solution with 2 physical servers. So the idea is to have 2 zookeeper (ZK) nodes on 1 box and 1 ZP on the 2nd node. Then 3 Kafka brokers on each nodes (so 6 in total). I want to have a fail over issue in

Re: Kafka & ZooKeeper IDs

Hi Christophe, No, not the PIDs... UNIX IDs. Are they(kafka & zookeeper) suppose to run under root ? or can i define my own Ids? Thanks, Vignesh On Mon, Mar 26, 2018 at 4:09 PM, Christophe Schmitz < christo...@instaclustr.com> wrote: > Hi Vignesh, > > Are you talking about the PID (process

Re: Kafka & ZooKeeper IDs

Hi Vignesh I am not sure what you mean by UNIX IDs is. Do you mean User ID (UID) which is the ID (number) of a given user? If so, there is no real requirement. You can run as root, or you can run as a user you create, provided that it has read and write access to a set of directories (data, logs

Re: Kafka replication

Hi Romain, Zookeeper needs to achieve quorum to function. This means that a strict majority of nodes need to be alive. If your box1 dies, you lose 2 zookeeper services, with only one left. Your system is already offline. So you would need to spread your zookeeper services across 3 physical nodes.

Re: Kafka & ZooKeeper IDs

Hi Vignesh, Are you talking about the PID (process ID)? If so, those are automatically allocated by the kernel and you don't have to worry about. Cheers, Christophe On 27 March 2018 at 03:29, Vignesh wrote: > Hi Anand, Thanks.. > I was asking about linux/unix ids