Re: [Discussion] New Consumer API / Protocol

2015-06-15 Thread Guozhang Wang
Yeah I think it is better to discuss these points in the KIP meeting, or it may become a long thread. Let's do that this Tuesday. Guozhang On Thu, Jun 11, 2015 at 5:48 PM, Jun Rao j...@confluent.io wrote: Guozhang, Perhaps we can discuss this in our KIP hangout next week? Thanks, Jun

Re: [Discussion] New Consumer API / Protocol

2015-06-11 Thread Jun Rao
Guozhang, Perhaps we can discuss this in our KIP hangout next week? Thanks, Jun On Tue, Jun 9, 2015 at 1:12 PM, Guozhang Wang wangg...@gmail.com wrote: This email is to kick-off some discussion around the changes we want to make on the new consumer APIs as well as their semantics. Here are

[Discussion] New Consumer API / Protocol

2015-06-09 Thread Guozhang Wang
This email is to kick-off some discussion around the changes we want to make on the new consumer APIs as well as their semantics. Here are a not-comprehensive list of items in my mind: 1. Poll(timeout): current definition of timeout states The time, in milliseconds, spent waiting in poll if data

Re: [Discussion] New Consumer API / Protocol

2015-06-09 Thread Jiangjie Qin
Thanks for kicking off this discussion, Guozhang. We might also want to discuss the API to expose the high watermark. Some discussion has been there in KAFKA-2076. Thanks, Jiangjie (Becket) Qin On 6/9/15, 1:12 PM, Guozhang Wang wangg...@gmail.com wrote: This email is to kick-off some

Re: [Discussion] New Consumer API / Protocol

2015-06-09 Thread Jason Gustafson
Just a minor correction, but #2 is KAFKA-2168 and #3 is KAFKA-2123. For #1, I think there should be some minimal effort to making the poll respect the timeout (with the understanding that spurious wakeups can happen). I think this really just means calling NetworkClient.poll() in a loop and

Re: [Discussion] New Consumer API / Protocol

2015-06-09 Thread Jay Kreps
My two cents: Overall I think our focus as far as extensions go should be on (1) things which would be incompatible if changed later and (2) operationalization. There are lots of new apis that could be nice to have, but I think if adding them later will just be an api addition we should hold off