When a broker down, Producer LOST messages!

Hi all, I have a kafka cluster with 3 nodes: node1, node2, node3 *kafka version is 0.8.2.1, which I can not change!* A Producer writes msg to kafka, its code framework is like this in pseudo-code: Properties props = new Properties();props.put("bootstrap.servers",

Re: When a broker down, Producer LOST messages!

Hi, Thank you for you reply. It looks like the attachment is fine. I have a replication-factor of 3. The producer can not even get the METADATA from a broker(when it chooses a dead broker), let alone sending messages and wait for all brokers to ACK. Best regards! 2018-03-01 17:21 GMT+08:00

Re: When a broker down, Producer LOST messages!

Hi, The log is not attached. I'm assuming your topic has a replication factor greater than 1 so that it is available from another Broker if the partition leader fails. Try adding props.put("acks", "all"); to your producer and run your experiment again. If you configured your topic to have

RE: difference between 2 options

Sorry Andras, the the delay of my response. Ok I correctly understood for the deletion thank to your explanation. however, for recovery point I wanted to ask you, the concept's logic: For example I have one recovery-point-offset-checkpoint in topic-0 If the broker crashed, and restarted:

Re: [VOTE] 1.1.0 RC0

Thanks Jason. Assuming the system tests pass i'll cut RC1 tomorrow. Thanks, Damian On Thu, 1 Mar 2018 at 19:10 Jason Gustafson wrote: > The fix has been merged to 1.1. > > Thanks, > Jason > > On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy wrote: > > >

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

Hi, Damian, It would also be useful to include KAFKA-6111, which prevents deleteLogDirEventNotifications path to be deleted correctly from Zookeeper. The patch should be committed later today. Thanks, Jun On Thu, Mar 1, 2018 at 1:47 PM, Damian Guy wrote: > Thanks Jason.

Re: RE: Zookeeper and Kafka JMX metrics

Hi Adrien Thank you for the response and your time. I had looked at the MBeans exposed by zookeeper. I am trying to use this MBeans to port the metrics to UI tools. It is pretty straight forward with kafka. But I am not getting any sample code for zookeeper.  ThanksArunkumar Pichaimuthu, PMP

Re: [VOTE] 1.1.0 RC0

The fix has been merged to 1.1. Thanks, Jason On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy wrote: > Hi Jason, > > Ok - thanks. Let me know how you get on. > > Cheers, > Damian > > On Wed, 28 Feb 2018 at 19:23 Jason Gustafson wrote: > > > Hey Damian, >

Re: Hardware Guidance

Hi Adrien, Without asking the author directly I can't give the exact answer but I would interpret that as per broker. Kafka will make use of as much hardware as you give it so it's not uncommon to see many CPU cores and lots or RAM per broker. That being said it's completely down to your use

Re: Hardware Guidance

I'm quite sure it's per broker (it's a standard way to provide recommendation on node sizes in systems like Kafka), but you should definitely read it in the context of the data size and traffic the cluster has to handle. I didn't read the presentation, so not sure if it contains such information

Detecting end of distributed batch processing within a streaming graph

Hello all, This question is not strictly related to Kafka but rather to a streaming design using Kafka. Hope it still stays within the scope of this list. I would like to distribute the processing of a monolithic batch within a streaming DAG, so with multiple parallel branches, each branch

Re: Hardware Guidance

It's per broker. Usually you run with 4-6GB of java heap. The rest is used as disk cache and it's more that 64GB seems like a sweet spot between memory cost and performance. /svante 2018-03-01 18:30 GMT+01:00 Michal Michalski : > I'm quite sure it's per broker (it's

RE: Hardware Guidance

Thank all, okay it per broker. As you can say it's really depend of the use case. Even if for me it seems huge, it will really depend on the use in each IS and the needed throughput to carry out a project. More again, thank all, Adrien De : Svante

Re: which Kafka StateStore could I use ?

Hello Jie, Just to understand your problem better, are you referring "db" for an external storage engine outside Kafka Streams, and you are asking how to only send one record per aggregation key (assuming you are doing some aggregations with Streams' statestore) to that end storage engine?

答复: which Kafka StateStore could I use ?

Yes .but the DB’s Concurrent quantity is the limitation. Now I can process 600 records/second And I want enhance it 发送自 Windows 10 版邮件应用 发件人: Guozhang Wang 发送时间: 2018年3月2日 2:59 收件人:

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

KAFKA-6111 is now merged to 1.1 branch. Thanks, Jun On Thu, Mar 1, 2018 at 2:50 PM, Jun Rao wrote: > Hi, Damian, > > It would also be useful to include KAFKA-6111, which prevents > deleteLogDirEventNotifications path to be deleted correctly from > Zookeeper. The patch

Burrow V3 - going down with memory issues

We are running burrow version3 for kafka consumer lag monitoring, Issue is burrow service is getting down frequently which is every 1hr, Command used to start the burrow service # nohup $GOPATH/bin/Burrow --config-dir /opt/work/src/github.com/ linkedin/Burrow/config 1>&2 & We are seeing the

Re: Consumer group intermittently can not read any records from a cluster with 3 nodes that has one node down

Hi Check the __consumer_offsets topics replication. If it's set to one that's the issue. Increase the replication of the topic. Thanks Siva On Feb 21, 2018 1:35 PM, "Sandor Murakozi" wrote: > hi Behrang, > I recommend you to check out some docs that explain how partitions

Hardware Guidance

Hi all, on the slide 5 in the following link: https://fr.slideshare.net/HadoopSummit/apache-kafka-best-practices/1 The "Memory" mentions that "24GB+ (for small) and 64GB+ (for large)" Kafka Brokers but is it 24 or 64 GB spread over all brokers ? Or 24 GB for example for each broker ?