Re: Fail fast a broker

2018-06-08 Thread Jacob Sheck
What do you mean by "The issue appears when one of the brokers starts being impacted by environmental issues within the server it's running into (for whatever reason)"? You should consider Kafka to be a first tier service, it shouldn't be deployed on shared resources. There are a lot of opinions

Re: Pet Project - Web based consumer for Kafka

2018-01-08 Thread Jacob Sheck
Is this tool similar to this: https://github.com/Landoop/kafka-topics-ui ? On Mon, Jan 8, 2018 at 7:09 AM Stephen P wrote: > Hello all! > > I've been working on a pet project over the last couple of months building > out a web based interface for consuming from Kafka

Re: what are common ways to convert info on a web site into a log entry?

2018-01-15 Thread Jacob Sheck
You will need to find or create something to accomplish this. Topics in Kafka primarily act as queues. If you search the web for more information about "kafka connector http" you will find a few projects that do this. You could also take a look at the Confluent S3 Connector as an example of how

Re: Is Kafka Streams right for me ?

2018-03-13 Thread Jacob Sheck
Sameer when you say that you need to "consume from and produce to a topic" to me that seems like a good fit for Kafka Streams. Streaming your data out of Kafka for a transform and back in has some fundamental costs and operational challenges involved. Are the events in your stream stateless? If

Re: Is Kafka Streams right for me ?

2018-03-13 Thread Jacob Sheck
d output stream. Basically the source of > this pipeline can be anything and the output can be anything to. > > On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck <shec0...@gmail.com> wrote: > > > Sameer when you say that you need to "consume from and produce to a > topic&

Re: delivery semantics?

2018-04-05 Thread Jacob Sheck
Your question seems to be confusing the producer and consumer responsibilities in message delivery durability. Neha from Confluent wrote a blog post about this which should help your understanding of the how the Kafka clients handle this.

Re: Closing a KafkaConsumer triggers a rebalance in KafkaConsumers subscribed to other topics. Is this the expected behavior?

2018-04-03 Thread Jacob Sheck
I believe you have answered your own question. Having the system enforce which topics each consumer in a consumer group needs to consume messages from may be restricting, and possibly expensive. I have seen this brought up a few times and discussed it with a colleague. I am not clear as to what

Re: zookeeper connection issue

2018-06-28 Thread Jacob Sheck
Does the container system used in your Rancher environment have persistence configured for the brokers? Are you are using ephemeral storage? On Thu, Jun 28, 2018 at 7:39 AM Karthick Kumar wrote: > Hi, > > I'm using Kafka cluster on three different servers, Recently my servers > went down when