Re: Batch processing with Kafka Streams with at-least-once semantics

2018-10-12 Thread Tomoyuki Saito
Hello Guozhang, Thank you for your reply. > setting to "0" will actually mean to commit every time. Hum, I somehow misunderstood the code. Now I understand that is true. > You should actually set it to Long.MAX_VALUE to indicate "not commit regularly by intervals" I see. I'd consider taking

Kafka on Eclipse Setup

2018-10-12 Thread James Kwan
Has anyone successfully set up an eclipse environment for Kafka on Windows? I have tried many times with different options but still have build issues with stream project. There are always some libraries missing. The steps I used 1) Run git command in command prompt to download the Kafka.

Re: Kafka UnknownProducerIdException

2018-10-12 Thread Matthias J. Sax
This error is related to Kafka EOS feature. I am not familiar with Golden Gate though. The error can happen, if the state of a transactional producer is lost. The state is stored in the topic itself, thus, it could be lost due to long truncation (for example, if a producer does not send data for

Re: Kafka Connect herder timeout when getting position for newly created offsets partition.

2018-10-12 Thread Daniel Wilson
To anyone else struggling with this. It looks to have been caused by some default values for the offsets.topic.replication.factor and transaction.state.log.replication.factor properties. They default to 3 it seems. Since a replication factor higher than the number of nodes in your cluster doesn't

Streams SessionStore - bug or documentation error?

2018-10-12 Thread Thomas Becker
I've been experimenting with the streams SessionStore and found some behavior that contradicts the javadoc. Specifically: I have a SessionStore, and put() a session with key K1 and session of time T0-T5. I then call findSessions("K1", T2, T4) and it comes back empty. I would expect the session

Re: Batch processing with Kafka Streams with at-least-once semantics

2018-10-12 Thread Guozhang Wang
Hello Tomoyuki, 1. Seems a good use case for Streams. 2. You should actually set it to Long.MAX_VALUE to indicate "not commit regularly by intervals", setting to "0" will actually mean to commit every time. Then you can leverage on ProcessorContext.commit() to manually commit after the batch is

Re: How to manage Kafka quota configuration with java code?

2018-10-12 Thread SJTU胡昊坤
My kafka version is 1.1.0, so it has two versions of notification format. Anyway, the problem is solved by restarting the broker. I guess the listening thread of broker might be shutdown during my debugging. Anyway, Thanks! 赖剑清 于2018年10月12日周五 下午5:32写道: > Hi, > > I use kafka 0.10.0.0, the entity

RE: How to manage Kafka quota configuration with java code?

2018-10-12 Thread 赖剑清
Hi, I use kafka 0.10.0.0, the entity type enum may be different with yours. However, in the DynamicConfigManager.ConfigChangedNotificationHandler I found that the right format of a config change notification should be as: {"version": 1, "entity_type": "topic/client", "entity_name":

New increased partitions could not be rebalance, until stop all consumers and start them

2018-10-12 Thread ????????????
Hi Kafka team, I meet a strange thing about Kafka rebalance. If I increase partitions of a topic which subscribed by some java consumers(in same one group), there is no rebalance occur. Furthermore, if I start a new consumer (or stop one) to cause a rebalance, the increased partitions could

Re: [ANNOUNCE] New Committer: Manikumar Reddy

2018-10-12 Thread Edoardo Comar
Well done Manikumar ! -- Edoardo Comar IBM Event Streams IBM UK Ltd, Hursley Park, SO21 2JN From: "Matthias J. Sax" To: dev Cc: users Date: 11/10/2018 23:41 Subject:Re: [ANNOUNCE] New Committer: Manikumar Reddy

Re: [ANNOUNCE] New Committer: Manikumar Reddy

2018-10-12 Thread Viktor Somogyi-Vass
Congratulations Manikumar, well deserved! On Fri, 12 Oct 2018, 06:30 Andras Beni, wrote: > Congratulations, Manikumar! > > Srinivas Reddy ezt írta (időpont: 2018. okt. > 12., P 3:00): > > > Congratulations Mani. We'll deserved  > > > > - > > Srinivas > > > > - Typed on tiny keys. pls ignore