Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Yes, it's noticeably faster in my test environment, under load. I'm going to apply the default value (100) and see how it behaves in production. Thanks, Johan On Tue, Oct 17, 2017 at 1:58 PM Guozhang Wang wrote: > This is only for my diagnosing your issue. > > I suspect that due to your very sm

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

This is only for my diagnosing your issue. I suspect that due to your very small traffic (5 records per sec) and long polling parameter, the restoration process was blocked on waiting for data. More specifically, in KAFKA-5152 which is merged in 0.11.0.1, a thread may be restoring for some tasks w

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Ok, I will try that. Is there any documentation as to what exactly this does, in general? Just want to safe guard against side effects. Also, do you recommend 0 as a production setting (default is 100ms, I think), or is this just to test this out/diagnose? The only description I can find is this:

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Johan, Could you please try change the following configs and try again? properties.put(StreamsConfig.POLL_MS_CONFIG, 0); Guozhang On Tue, Oct 17, 2017 at 12:34 PM, Johan Genberg wrote: > Hi, > > The trace log is here (tried to attach to email but it was rejected): > > https://gist.github.co

Re: Can I commit the same offset? Can I back up?

> For my own nefarious reasons, I generally run with enable_auto_commit > == False in my consumers. I'm just using Kafka's default commit > capability, nothing more sophisticated so far. Is it okay to commit > out-of-sequence, effectively allowing me to back up and replay > messages? For example, i

Can I commit the same offset? Can I back up?

I'm a pretty wet-behind-the-ears Kafka user, but I really like it. I was up and running with the basics in just a few minutes (I use kafka-python - dunno if that makes a difference). For my own nefarious reasons, I generally run with enable_auto_commit == False in my consumers. I'm just using Kafk

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Hi, The trace log is here (tried to attach to email but it was rejected): https://gist.github.com/trickleup/fe0c095df65b8ae10906ea6774804b54 The logs are from a test environment, but I was able to somewhat reproduce the issue, and with 5 or so messages per second on average. The production syste

KTable Tombstone and expiry of records in Session Window

A few questions here First environment: Scala, Kafka 0.11.0.0 I have a KTable[Windowed[String], MyType] This is then transformed to a stream via .toStream[String]((k, _) => k.key()) The KTable is a result of a reduce operation with a SessionWindow InactvitiyGap defined as 10 minutes. Retention (u

Re: Can you please subscribe me

Hello Nikihil, It is a self-service. Guozhang On Tue, Oct 17, 2017 at 10:36 AM, Nikhil Deore wrote: > Thank you, > Nikhil > -- -- Guozhang

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Hello Johan, Could you list the following information in addition to your topology? 1. Your config settings if you have any overwrites, especially consumer's " poll.ms" and "max.poll.records", and "num.threads". 2. Your expected incoming data rate (messages per sec) at normal processing phase.

Fwd: Can you please subscribe me in this project

Hi, I want to learn and contribute to this project, Please subscribe me in. Thanks, Nikhil

Re: [VOTE] 1.0.0 RC1

Thanks Ismael for the tip. I missed it in the Readme page ( https://github.com/apache/kafka#running-a-task-on-a-particular-version-of-scala-either-211x-or-212x ) --Vahid From: Ismael Juma To: d...@kafka.apache.org Cc: Kafka Users Date: 10/16/2017 06:50 PM Subject:Re: [VOTE

Can you please subscribe me in this project

Hi, I want to learn and contribute to this project, Please subscribe me in. Thanks, Nikhil

Can you please subscribe me

Thank you, Nikhil

[VOTE] 1.0.0 RC2

Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 1.0.0. The main PRs that gets merged in after RC1 are the following: https://github.com/apache/kafka/commit/dc6bfa553e73ffccd1e604963e076c78d8ddcd69 It's worth noting that starting in thi

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Just to clarify, in 0.11.0.1 we have optimized the restoration process so it is actually expected to be faster than 0.10.2.1, so what you observed is indeed very weird. We'd need some more information like log entries to see what's happening there. Guozhang On Tue, Oct 17, 2017 at 2:50 AM, Damia

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Hi, Thank you for responding so quickly. This is the topology. I've simplified it a bit, but these are the steps it goes through, not sure if that is helpful. I'll try to get some logs too in a bit. *KStream eventStream = builder.stream(* * topic.keySerde(),* * topic.valueSerde(),* * topic.nam

Re: general query.

Kafka's own consumer is here: 2 3 > bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning Source: https://kafka.apache.org/quickstart On Oct 17, 2017 8:25 AM, "Dhawan Gajendran" wrote: Kafka comes with a cli consumer which you could use to retrieve mess

Re: Kafka 0.9.0.1 Direct buffer memory OOM

Can you take a look at KAFKA-5470 ? On Tue, Oct 17, 2017 at 2:32 AM, 杨文 wrote: > Hi Kafka Users, > We are using kafka 0.9.0.1 and frequently we have seen below exception > which causes the broker > to die.We even increased the MaxDirectMemory to 1G but still see this. > 2017-02-16 00:55:57,750]

Re: general query.

Kafka comes with a cli consumer which you could use to retrieve messages from your topic. https://github.com/edenhill/kafkacat/blob/master/README.md Please keep in mind this is more for testing purposes than for production. On Oct 17, 2017 8:20 AM, "Jaswinder Singh" wrote: > Hello, > > My name

Kafka 0.9.0.1 Direct buffer memory OOM

Hi Kafka Users, We are using kafka 0.9.0.1 and frequently we have seen below exception which causes the broker to die.We even increased the MaxDirectMemory to 1G but still see this. 2017-02-16 00:55:57,750] ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError

general query.

Hello, My name is Jaswiner Singh. I am new to kafka and have a questio to ask. Is there any way to see the producer data in kafka? for example=> if i use filebeat to send logs to kafka so can i see that logs in kafka ..is there any way to do this. thanks and regards Jaswinder singh

Re: Kafka connect rebalance problem

Thank you for the reply. I have 7 kafka s3 connectors. Each connector has been configured with 4 tasks. However, I am trying to add more tasks for a few s3 connectors. When I try to change the configuration of these connectors using the rest API, I see the worker rebalance error thrown from the R

Re: KIP-99 streams global ktable - slowly changing dimension type 2 supported?

Hi Chris, You can only join on the key of the table, so i don't think this would work as is. Also, the global table is updated in a different thread and there is no guarantee that it would have been updated before the purchase. Perhaps you could do it by making the key of the product table versio

Re: Custom converter with Kafka Connect ?

HI Philip, thanks for the reply ! This issue looks like it might match my problem indeed. I'll look into it. Jehan On 16 October 2017 at 20:53, Philip Schmitt wrote: > Hi Jehan, > > I've run into the same issue last week and also got a "class could not be > found" error. > > Konstantine Karan

Re: KIP-99 streams global ktable - slowly changing dimension type 2 supported?

Thanks, Guozhang. I've been thinking about the following approach: https://imgur.com/a/pP92Z Does this approach make sense? A key consideration will be that the product dimension table updates are processed and added to kafka before the corresponding purchase transaction record is processed.

Re: [VOTE] 1.0.0 RC1

Hi Ghouzang, We have indeed started our performance testing at Heroku for RC1. However, we are more than happy to retest once RC2 is available, especially given larger amounts of time to do so. Thanks Tom Crayford Heroku Kafka On Tue, Oct 17, 2017 at 2:50 AM, Ismael Juma wrote: > If you don't

Re: Kafka Streams 0.11.0.1 Rebalancing Delay

Hi Johan, Do you have any logs? The state store restoration changed significantly in 0.11.0.1. If you could get some logs at trace level, that would be useful. Also if you could provide your topology (removing anything proprietary/sensitive). Thanks, Damian On Tue, 17 Oct 2017 at 05:55 Johan Gen

Re: Kafka connect rebalance problem

bq. There are 7 connectors configured with each connector configured with 8 tasks (Averaging about 4 tasks per connector) Pardon. I don't quite understand the above setup. Can you describe in more detail ? Which version of connector are you using ? Cheers On Mon, Oct 16, 2017 at 11:30 PM, Dhawa