Re: KIP-122: Add a tool to Reset Consumer Group Offsets

2017-02-23 Thread Matthias J. Sax
There is a voting thread on dev list. Please put your vote there. Thx. -Matthias On 2/23/17 8:15 PM, Mahendra Kariya wrote: > +1 for such a tool. It would be of great help in a lot of use cases. > > On Thu, Feb 23, 2017 at 11:44 PM, Matthias J. Sax > wrote: > >> \cc

Re: Re: KIP-122: Add a tool to Reset Consumer Group Offsets

2017-02-23 Thread Mahendra Kariya
+1 for such a tool. It would be of great help in a lot of use cases. On Thu, Feb 23, 2017 at 11:44 PM, Matthias J. Sax wrote: > \cc from dev > > > Forwarded Message > Subject: Re: KIP-122: Add a tool to Reset Consumer Group Offsets > Date: Thu, 23 Feb

Re: Frequently shrink/expand ISR on one broker

2017-02-23 Thread Jun MA
Digging into on of the case that Broker 1 kicked Broker 6 out of ISR, I found a weird consequences that Broker 1 firstly decided to remove Broker 6 from ISR out of no reason, then Broker 6 saw an connection disconnected exception on ReplicaFetcherThread-0-1. On Broker 1: [2017-02-23

Re: Question about messages in __consumer_offsets topic

2017-02-23 Thread James Cheng
Yup, this got fixed in 0.10.2 https://issues.apache.org/jira/browse/KAFKA-2000 -James > On Feb 23, 2017, at 11:10 AM, Jeff Widman wrote: > > The topic deletion only triggers tombstone on brokers >= 0.10.2, correct? I >

Overriding JAAS config for connector

2017-02-23 Thread Stephen Durfey
Now that 0.10.2.0 is out, I was looking forward to checking out the inclusion of KIP-85 . I have a potential need for multi-tenancy in a single kafka connect instance, and wanted to be able to

Re: Question about messages in __consumer_offsets topic

2017-02-23 Thread Jeff Widman
The topic deletion only triggers tombstone on brokers >= 0.10.2, correct? I thought there was an outstanding bug report for this in lower versions... On Wed, Feb 22, 2017 at 6:17 PM, Hans Jespersen wrote: > The __consumer_offsets topic should also get a tombstone message as

Re: Creating topic partitions automatically using python

2017-02-23 Thread Jeff Widman
This is probably a better fit for the pykafka issue tracker. AFAIK, there's no public kafka API for creating partitions right now, so pykafka is likely hacking around this by calling out to internal Java APIs, so it will be brittle On Thu, Feb 23, 2017 at 5:13 AM, VIVEK KUMAR MISHRA 13BIT0066 <

kafka streams locking issue in 0.10.20.0

2017-02-23 Thread Ara Ebrahimi
Hi, After upgrading to 0.10.20.0 I got this: Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was

Pattern to create Task with dependencies (DI)

2017-02-23 Thread Petr Novak
Hello, it seems that KConnect take control over creating task instance and requires no-arg constructor. What is the recommended pattern when I need to create a task which has dependency e.g. on some db client and I want to be able to pass in mock in tests, preferable through constructor? In Java

Creating topic partitions automatically using python

2017-02-23 Thread VIVEK KUMAR MISHRA 13BIT0066
Hi All, I am trying to create partitions automatically using python script instead of doing the same through terminal. I am using partition class of pykafka package to do so and kafka_2.11-0.9.0.0,but I am not getting the desired output, only one partition is getting created. Please do suggest

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

2017-02-23 Thread Jorge Esteban Quilcate Otoya
@Matthias about the point 9: What about keeping only the --topic option, and support this format: `--topic t1:0,1,2 --topic t2 --topic t3:2` In this case topics t1, t2, and t3 will be selected: topic t1 with partitions 0,1 and 2; topic t2 with all its partitions; and topic t3, with only

Re: Simple data-driven app design using Kafka

2017-02-23 Thread Michael Noll
Pete, have you looked at Kafka's Streams API yet? There are many examples available in the `kafka-streams` folder at https://github.com/confluentinc/examples. The simplest example of "Do sth to a new data record as soon as it arrives" might be the MapFunctionLambdaExample. You can create

How to change the logging level or logging place of KafkaServerStable?

2017-02-23 Thread 이산하
Hi all Kafka users. I'm sanha, new to Kafka. I have a question to ask Kafka experts (hopefully you!). I'm making my own system with Maven. I've added a Kafka source in my project, using Kafka version 0.10.1.0. To test it, I added a unit test using KafkaServerStable to pass some data locally.