[VOTE] 2.0.1 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.0.1. This is a bug fix release closing 49 tickets: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1 Release notes for the 2.0.1 release:

Re: The limit on the number of consumers in a group.

I always have more number of partitions than the number of consumers. When I face "hangs" with 150 consumers, the number of partitions was 300. 2018년 10월 26일 (금) 오전 11:09, Liam Clarke 님이 작성: > How many partitions? > > On Fri, 26 Oct. 2018, 2:52 pm Dominic Kim, wrote: > > > Dear all. > > > > Is

Re: The limit on the number of consumers in a group.

How many partitions? On Fri, 26 Oct. 2018, 2:52 pm Dominic Kim, wrote: > Dear all. > > Is there any limit on the number of consumers in a group? > I want to utilize about 300 or more consumers in a group, but rebalancing > hangs and never get finished. > When I invoke only 130~140 consumers in

The limit on the number of consumers in a group.

Dear all. Is there any limit on the number of consumers in a group? I want to utilize about 300 or more consumers in a group, but rebalancing hangs and never get finished. When I invoke only 130~140 consumers in a group, it works fine. Buf from 150 consumers, rebalancing is never stopped. I am

Re: Consumer Pause & Scheduled Resume

Code Snippet Without continuous polling == public class OfferItemImageScheduler { @Scheduled(cron = "0 0/2 * * * ?") void startStreaming() { kafkaConsumer.resume(kafkaConsumer.assignment()); offerItemImageConsumer.streamMessages(kafkaConsumer);

Re: Consumer Pause & Scheduled Resume

Hi Manoj/Matthias, My requirement is that to run the consumer daily once , stream the messages and pause when i am encountering a few empty fetches . I am planning to run two consumers and pausing the consumption based on the empty fetches for a topic with 4 partitions . To avoid the consumer

Re: Consumer Pause & Scheduled Resume

One item to be aware with pause and resume - is that it applies to partitions currently assigned to the consumer. But partitions can get revoked or additional partitions can get assigned to consumer. With reassigned , you might be expecting the consumer to be paused but suddenly start getting

Re: Consumer Pause & Scheduled Resume

That is correct: clients are not thread safe. You can use an `AtomicBoolean needToResume` that you share over both threads and that is initially false. In your scheduled method, you set the variable to true. In your main consumer, each time before you call poll(), you check if the variable is

Re: running kafka streams inside kafka connect

Streams is not designed to be run inside Connect, and this won't work. What you can do is, to import the data via connect into a "staging topic" and then read this "staging topic" with a Kafka Streams application and apply the transformations etc to write the data into the actual target topics.

Re: Converting a Stream to a Table - groupBy/reduce vs. stream.to/builder.table

Patrik, `null` values in a KStream don't have delete semantics (it's not a changelog stream). That's why we drop them in the KStream#reduce implemenation. If you want to explicitly remove results for a key from the result KTable, your `Reducer#apply()` implementation must return `null` -- the

Re: running kafka streams inside kafka connect

Thank you Ryanne for the answer, My question is can I run Streams app inside Connect, what I mean is since Streams is a library, can I build a custom Source connector and use Streams in there . or using Streams in transforms?? We want to containerize Kafka connect as in wrap it in a docker, and

Re: running kafka streams inside kafka connect

Dhurandar, definitely! Connect and Streams are both agnostic to how their workers are run. They aren't really platforms per se. You just need to spin up one or more workers and they do their thing. So a Streams app doesn't run "inside" Connect, but you can certainly have Connect and Streams

Re: Consumer Pause & Scheduled Resume

Thanks Matthias. I am facing the issue when i am trying to call the resume from the scheduled method . Was getting exception that Kafka Consumer is not safe for multi threaded access . I am trying to see how can call pause and resume on the same thread. There will be only one thread running for

running kafka streams inside kafka connect

We have a requirement to do Single row transformations, basic joins, deduping and routing from the source to the destination Kafka topics. We want to use Kafka-connect as the platform which is running Kafka stream inside. Has anyone used Kafkastreams inside Kafka connect ?? Is this pattern ok

Converting a Stream to a Table - groupBy/reduce vs. stream.to/builder.table

Hello Recently we noticed a lot of warning messages in the logs which pointed to this method (we are running 2.0): KStreamReduce public void process(final K key, final V value) { // If the key or value is null we don't need to proceed if (key == null || value == null) {

How to turn-off Time-Based index files in Kafka?

Hi Users, How to turn off time-based index files in Kafka (.timeindex files). Tried searching for it in topic and broker configs documentation but could not find it. -- *Ashwin Sinha *| Data Engineer ashwin.si...@go-mmt.com | 9452075361

Stream Metrics - Memory Analysis

Hello During the analysis of JVM memory two possible issues were shown which I would like to bring to your attention: 1) Duplicate strings Top findings: string_content="stream-processor-node-metrics" count="534,277" string_content="processor-node-id" count="148,437"