Re: State flush & recovery during failures

2018-09-20 Thread Matthias J. Sax
1. The record will be re-read, but the state would not be re-build (ie, no undo of step (2). Thus, on re-processing you would add the record again, and you would "over count" in step (3) -- trigger would still fire I assume. 2. I assume, by "forward" you mean writing to an output topic: If you

Re: Kafka stream issue : Deleting obsolete state directory

2018-09-20 Thread Bhavesh Patel
Hi Bill, Yes and thanks for your response. Many thanks, Bhavesh Patel On Thu, Sep 20, 2018, 21:31 Bill Bejeck wrote: > Hi Bhavesh, > > I haven't forgotten about you, just got sidetracked with some other > issues. > > Just to be clear you are using Kafka Streams version 2.0 correct? > > I'll

Re: Kafka stream issue : Deleting obsolete state directory

2018-09-20 Thread Bill Bejeck
Hi Bhavesh, I haven't forgotten about you, just got sidetracked with some other issues. Just to be clear you are using Kafka Streams version 2.0 correct? I'll take a look in the next day or so. Thanks, Bill On Thu, Sep 20, 2018 at 2:31 PM Bhavesh Patel wrote: > Hi, > Is it possible to get h

Re: Kafka stream issue : Deleting obsolete state directory

2018-09-20 Thread Bhavesh Patel
Hi, Is it possible to get h lp with below issue? Many Thanks, Bhavesh Patel On Wed, Sep 19, 2018, 09:13 Bhavesh Patel wrote: > Hi Bill, > Apologies, I was trying to attached image. > Please find below the error log, > > 2018-09-18 09:26:09.112 INFO 1 --- [5-CleanupThread] >

Kafka_2.12-1.1.0 config

2018-09-20 Thread sarath reddy
Hi Team, We are trying to configure Kafka to produce larger messages, Below are the configs:- Server.properties message.max.bytes=1 replica.fetch.max.bytes=10001 Producer.properties max.request.size=1 compression.type=gzip Consumer.properties

Re: Kafka_2.12-1.1.0 config

2018-09-20 Thread sarath reddy
++pushkar On Thu 20 Sep, 2018, 16:20 sarath reddy, wrote: > Hi Team, > > We are trying to configure Kafka to produce larger messages, > > Below are the configs:- > > Server.properties > > message.max.bytes=1 > replica.fetch.max.bytes=10001 > > Producer.properties > >

Problems trying to make kafka 'rack-aware'

2018-09-20 Thread Bryan Duggan
Hi, I have a kafka cluster consisting of 3 brokers across 3 different AWS availability zones.  It hosts several topics, each of which has a replication factor of 3. The cluster is currently not 'rack-aware'. I am trying to do the following;     - add 3 additional brokers (one in each of the

Re: How to enable Kafka Authentication as on-demand feature and later make mandatory feature?

2018-09-20 Thread Manikumar
One option is to enable multiple listeners (PLAINTEXT, SASL_PLAINTEXT) on brokers and slowly migrate the clients/services. On Wed, Sep 19, 2018 at 10:38 PM Ashwin Sinha wrote: > Hi, > > We have tried setting up Kafka Authentication by SASL/PLAIN >