Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
props.put("bootstrap.servers", "psaq1-wc.sys.comcast.net: > 61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616, > psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3 > -ab.sys.comcast.net:61617"); >

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
roup Metadata Manager on Broker 4]: > Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator. > GroupMetadataManager) > [2016-09-01 02:13:04,928] INFO [Group Metadata Manager on Broker 4]: > Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator. > GroupMetadataManager) > [20

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
nding if 3 nodes > are up and message already replicated why it's trying to fetch the data > from failed node. Can you please explain bit details how it works. Thanks > for your response. > > -Original Message- > From: Jason Gustafson [mailto:ja...@confluent.io] >

Re: PartitionAssignor / Sort members per subscription time before assigning partitions

2016-09-01 Thread Jason Gustafson
Hi Florian, I'm not totally sure I understand the problem. The consumer id consists of the clientId configured by the user with a UUID appended to it. If the clientId has not been passed in configuration, we use "consumer-{n}" for it where n is incremented for every new consumer instance. Is the p

Re: [DISCUSS] Remove beta label from the new Java consumer

2016-09-01 Thread Jason Gustafson
change). > > > > > > Ismael > > > > > > On Mon, Aug 29, 2016 at 7:37 PM, Jay Kreps wrote: > > > > > >> +1 I talk to a lot of kafka users, and I would say > 75% of people > doing > > >> new things are on the new consumer des

Re: [ANNOUNCE] New committer: Jason Gustafson

2016-09-06 Thread Jason Gustafson
gt; > > > > > --Vahid > > > > > > > > > > > > > > > > From: Neha Narkhede > > > > To: "dev@kafka.apache.org" , > > > > "us...@kafka.apache.org" > > > > Cc: "priv...@ka

Re: [DISCUSS] Time-based releases for Apache Kafka

2016-09-06 Thread Jason Gustafson
Hey All, It sounds like the general consensus is in favor of time-based releases. We can continue the discussion about LTS, but I wanted to go ahead and get things moving forward by volunteering to manage the next release, which is currently slated for October. If that sounds OK, I'll draft a rele

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-09-07 Thread Jason Gustafson
Hey Becket, Thanks for the KIP. As I understand, the intention is to preserve the current behavior with a timestamp of -1 indicating latest timestamp and -2 indicating earliest timestamp. So users can query these offsets using the offsetsForTimes API if they know the magic values. I'm wondering if

Re: [VOTE] KIP-78 Cluster Id (second attempt)

2016-09-07 Thread Jason Gustafson
+1 and thanks for the excellent write-up! On Wed, Sep 7, 2016 at 10:41 AM, Neha Narkhede wrote: > +1 (binding) > > On Wed, Sep 7, 2016 at 9:49 AM Grant Henke wrote: > > > +1 (non-binding) > > > > On Wed, Sep 7, 2016 at 6:55 AM, Rajini Sivaram < > > rajinisiva...@googlemail.com > > > wrote: > >

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-09-07 Thread Jason Gustafson
e > redundant to earliestOffsets() and latestOffsets(). > > Personally I prefer option (2) because of the conciseness and it seems > intuitive enough. But I am open to option (1) as well. > > Thanks, > > Jiangjie (Becket) Qin > > On Wed, Sep 7, 2016 at 11:06 AM, Jason

[DISCUSS] Kafka 0.10.1.0 Release Plan

2016-09-09 Thread Jason Gustafson
Hi All, I've volunteered to be release manager for the upcoming 0.10.1 release and would like to propose the following timeline: Feature Freeze (Sep. 19): The 0.10.1 release branch will be created. Code Freeze (Oct. 3): The first RC will go out. Final Release (~Oct. 17): Assuming no blocking issu

Re: [DISCUSS] Kafka 0.10.1.0 Release Plan

2016-09-11 Thread Jason Gustafson
ut > > together for 0.10.0: > > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0 > > > > Also, you merged KIP-70 recently so that can be moved to the completed > > section. > > > > Ismael > > > > On Fri, Sep 9, 2

Re: [DISCUSS] Kafka 0.10.1.0 Release Plan

2016-09-12 Thread Jason Gustafson
ions, I will begin a vote on this release plan. Thanks, Jason On Sun, Sep 11, 2016 at 10:05 AM, Jason Gustafson wrote: > Hey Rajini, > > We took a long look at KIP-55 and decided that the time needed to review, > stabilize, and add system testing might not be sufficient. Usually a > s

[VOTE] 0.10.1 Release Plan

2016-09-13 Thread Jason Gustafson
Hi All, I'd like to start a vote on the release plan documented here: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1. I went ahead and included KIP-55 since Jun said it may have a chance, but note that in-progress KIPs will only be included if they are ready by the feature f

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-09-13 Thread Jason Gustafson
Hey Becket, I looked at the most recent version and it's looking good to me. One very minor question on naming: do you think OffsetAndTimestamp would be a more consistent name than TimestampOffset given some of our other naming (e.g. OffsetAndMetadata)? I put the "Offset" first since the name "off

Re: [VOTE] KIP-54: Sticky Partition Assignment Strategy

2016-09-13 Thread Jason Gustafson
Thanks for the KIP. +1 from me. On Tue, Sep 13, 2016 at 12:05 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi all, > > Thanks for providing feedback on this KIP so far. > The KIP was discussed during the KIP meeting today and there doesn't seem > to be any unaddressed issue at this

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-09-13 Thread Jason Gustafson
buse. > > Thanks, > > Jiangjie (Becket) Qin > > > > > > On Tue, Sep 13, 2016 at 3:08 PM, Jason Gustafson > wrote: > > > Hey Becket, > > > > I looked at the most recent version and it's looking good to me. One very > > minor question

Re: [VOTE] KIP-79 - ListOffsetRequest v1 and search by timestamp methods in new consumer.

2016-09-13 Thread Jason Gustafson
+1 and thanks for the great proposal! On Fri, Sep 9, 2016 at 4:38 PM, Becket Qin wrote: > Hi all, > > I'd like to start the voting for KIP-79 > > In short we propose to : > 1. add a ListOffsetRequest/ListOffsetResponse v1, and > 2. add earliestOffsts(), latestOffsets() and offsetForTime() method

Re: [VOTE] 0.10.1 Release Plan

2016-09-16 Thread Jason Gustafson
Hi All, Looks like this vote has passed with 8 binding and 4 non-binding votes. I'll send a progress update this afternoon as we race for the Monday feature freeze. Thanks, Jason On Wed, Sep 14, 2016 at 11:08 PM, Neha Narkhede wrote: > +1 (binding) > > On Tue, Sep 13, 2016 at 6:58 PM Becket Qi

[UPDATE] 0.10.1 Release Progress

2016-09-16 Thread Jason Gustafson
Hi All, Thanks everyone for the hard work! Here's an update on the remaining KIPs that we are hoping to include: KIP-78 (clusterId): Review is basically complete. Assuming no problems emerge, Ismael is planning to merge today. KIP-74 (max fetch size): Review is nearing completion, just a few mino

Re: [VOTE] KIP-79 - ListOffsetRequest v1 and search by timestamp methods in new consumer.

2016-09-19 Thread Jason Gustafson
+1 on Jun's suggestion to use "beginning" and "end". The term "latest" is misleading since the last message in the log may not have the largest timestamp. On Mon, Sep 19, 2016 at 9:49 AM, Jun Rao wrote: > Hi, Jiangjie, > > Thanks for the proposal. Looks good to me overall. Just a couple of minor

Re: [UPDATE] 0.10.1 Release Progress

2016-09-19 Thread Jason Gustafson
Thanks everyone for the hard work! The 0.10.1 release branch has been created. We're now entering the stabilization phase of this release which means we'll focus on bug fixes and testing. -Jason On Fri, Sep 16, 2016 at 5:00 PM, Jason Gustafson wrote: > Hi All, > > Thanks ev

Re: it this a bug? - message disorder in async send mode -- 0.9.0 java client sdk InFlightRequests

2016-09-26 Thread Jason Gustafson
Hi there, The Kafka server implements head of line request blocking, which means that it will only handle one request a time from a given socket. That means that the responses will always be returned in the same order as the requests were sent. -Jason On Sat, Sep 24, 2016 at 1:19 AM, 一生有你 wrote

Re: [UPDATE] 0.10.1 Release Progress

2016-09-27 Thread Jason Gustafson
at 2:43 PM, Becket Qin wrote: > > > Awesome! > > > > On Mon, Sep 19, 2016 at 11:42 PM, Neha Narkhede > wrote: > > > > > Nice! > > > On Mon, Sep 19, 2016 at 11:33 PM Ismael Juma > wrote: > > > > > > > Well done everyone. :) >

Re: [UPDATE] 0.10.1 Release Progress

2016-10-03 Thread Jason Gustafson
> Thanks Jason! > > On Tue, Sep 27, 2016 at 5:38 PM, Ismael Juma wrote: > > > Thanks for the update Jason. :) > > > > Ismael > > > > On Wed, Sep 28, 2016 at 1:28 AM, Jason Gustafson > > wrote: > > > > > Hi All, > > > >

[VOTE] 0.10.1.0 RC0

2016-10-04 Thread Jason Gustafson
Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 0.10.1.0. This is a major release that includes great new features including throttled replication, secure quotas, time-based log searching, and queryable state for Kafka Streams. A full li

Re: [VOTE] 0.10.1.0 RC0

2016-10-04 Thread Jason Gustafson
One clarification: this is a minor release, not a major one. -Jason On Tue, Oct 4, 2016 at 4:01 PM, Jason Gustafson wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 0.10.1.0. This is > a major release that in

Re: Snazzy new look to our website

2016-10-04 Thread Jason Gustafson
Huge improvement. Thanks Derrick and Gwen! On Tue, Oct 4, 2016 at 5:54 PM, Becket Qin wrote: > Much fancier now :) > > On Tue, Oct 4, 2016 at 5:51 PM, Ali Akhtar wrote: > > > Just noticed this on pulling up the documentation. Oh yeah! This new look > > is fantastic. > > > > On Wed, Oct 5, 2016

Re: Snazzy new look to our website

2016-10-06 Thread Jason Gustafson
ither from within the website or a google link - it > always takes me to the top of the Kafka 0.10 documentation page. I can't > figure out how to get to the javadoc. > > Thanks, Jonathan > > On Tue, Oct 4, 2016 at 6:46 PM Jason Gustafson wrote: > > > Huge improvemen

Re: [VOTE] 0.10.1.0 RC0

2016-10-07 Thread Jason Gustafson
a.lang.OutOfMemoryError: Map failed > > at sun.nio.ch.FileChannelImpl.map0(Native Method) > > ... 29 more > > > > This issue seems to break the broker and I have to clear out the logs so > I > > can bring the broker back up again. > > > > >

Re: [VOTE] 0.10.1.0 RC0

2016-10-07 Thread Jason Gustafson
> > I suggest not having a "Fix version" set for issues that don't fix anything > (it's not part of any release really). Yeah, good call. On Fri, Oct 7, 2016 at 8:59 AM, Ismael Juma wrote: > On Fri, Oct 7, 2016 at 4:56 PM, Jason Gustafson > wrote: > >

Re: [VOTE] 0.10.1.0 RC0

2016-10-07 Thread Jason Gustafson
I'll submit a patch for the trivial changes in the quick start. > Do you recommend adding Windows version of commands along with the current > commands? > > I'll also open a JIRA for the new consumer issue. > > --Vahid > > > > From: Jason Gustafson > T

[VOTE] 0.10.1.0 RC1

2016-10-10 Thread Jason Gustafson
Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 0.10.1.0. This is a minor release that includes great new features including throttled replication, secure quotas, time-based log searching, and queryable state for Kafka Streams. A full l

Re: [VOTE] 0.10.1.0 RC1

2016-10-10 Thread Jason Gustafson
The documentation is mostly fixed now: http://kafka.apache.org/ 0101/documentation.html. Thanks to Derrick Or for all the help. Let me know if anyone notices any additional problems. -Jason On Mon, Oct 10, 2016 at 1:10 PM, Jason Gustafson wrote: > Hello Kafka users, developers and cli

Re: Request to get added to contributor list

2016-10-11 Thread Jason Gustafson
Done. Thanks for contributing! -Jason On Tue, Oct 11, 2016 at 3:25 AM, HuXi wrote: > Hi, can I get added to the contributor list? I 'd like to take crack at > some issues. Thank you. My JIRA id: huxi_2b >

Re: [VOTE] 0.10.1.0 RC1

2016-10-12 Thread Jason Gustafson
FYI: I'm cutting another RC this morning due to https://issues.apache.org/jira/browse/KAFKA-4290. Hopefully this is the last! -Jason On Mon, Oct 10, 2016 at 8:20 PM, Jason Gustafson wrote: > The documentation is mostly fixed now: http://kafka.apache.org/0 > 101/documentation.html

[VOTE] 0.10.1.0 RC2

2016-10-12 Thread Jason Gustafson
Hello Kafka users, developers and client-developers, One more RC for 0.10.1.0. I think we're getting close! Release plan: https://cwiki.apache.org/confluence/display/KAFKA/Rele ase+Plan+0.10.1. Release notes for the 0.10.1.0 release: http://home.apache.org/~jgus/kafka-0.10.1.0-rc2/RELEASE_NOTES.

Re: [kafka-clients] [VOTE] 0.10.1.0 RC2

2016-10-13 Thread Jason Gustafson
> > Thanks, > Damian > > On Wed, 12 Oct 2016 at 20:05 Dana Powers wrote: > >> +1; all kafka-python integration tests pass. >> >> -Dana >> >> >> On Wed, Oct 12, 2016 at 10:41 AM, Jason Gustafson >> wrote: >> > Hello Kafka

Re: [VOTE] 0.10.1.0 RC2

2016-10-13 Thread Jason Gustafson
he stopped broker (0), records are > returned. If the leader is broker 1 or 2 I don't run into this issue. If I > use the old consumer I don't run into the issue either. I have been able > to reproduce this consistently on all three OS's above. > > --Vahid > >

Re: [VOTE] 0.10.1.0 RC2

2016-10-14 Thread Jason Gustafson
05-27-apache-kafka-010- > evaluating-performance-in-distributed-systems/). > > We see no issues at all with them on RC2. > > On Thu, Oct 13, 2016 at 11:04 PM, Jason Gustafson > wrote: > > > Thanks Vahid, I'll see if I can reproduce the problem you're seeing

[VOTE] 0.10.1.0 RC3

2016-10-14 Thread Jason Gustafson
Hello Kafka users, developers and client-developers, One more RC for 0.10.1.0. We're hoping this is the final one so that we can meet the release target date of Oct. 17 (Monday). Please let me know as soon as possible if you find any major problems. Release plan: https://cwiki.apache.org/confluen

Re: [VOTE] 0.10.1.0 RC3

2016-10-15 Thread Jason Gustafson
, Jason Gustafson wrote: > Hello Kafka users, developers and client-developers, > > One more RC for 0.10.1.0. We're hoping this is the final one so that we > can meet the release target date of Oct. 17 (Monday). Please let me know as > soon as possible if you find any major proble

Re: [VOTE] 0.10.1.0 RC3

2016-10-19 Thread Jason Gustafson
+1 from myself too. The vote passes with 9 +1 votes and no 0 or -1 votes. +1 votes PMC Members: * Gwen Shapira * Jun Rao * Neha Narkhede Committers: * Ismael Juma * Jason Gustafson Community: * Eno Thereska * Manikumar Reddy * Dana Powers * Magnus Edenhill 0 votes * No votes -1 votes * No

Re: [ANNOUNCE] Apache Kafka 0.10.1.0 Released

2016-10-20 Thread Jason Gustafson
Had the wrong address for dev and users (haven't sent from this account before). On Thu, Oct 20, 2016 at 11:05 AM, Jason Gustafson wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.1.0. This is a feature release which includes the complet

Re: [ANNOUNCE] Apache Kafka 0.10.1.0 Released

2016-10-20 Thread Jason Gustafson
> > > Guozhang > > > > > > On Thu, Oct 20, 2016 at 11:12 AM, Jason Gustafson > wrote: > > > > > Had the wrong address for dev and users (haven't sent from this account > > > before). > > > > > > On Thu, Oct 20, 2016 at 11:05 A

Re: [DISCUSS] KIP-80: Kafka REST Server

2016-10-20 Thread Jason Gustafson
I've been a little reluctant to wade into this discussion since I've spent a lot of my own time on Confluent's kafka-rest project myself. Seems obvious I would hope that time is not wasted and the project succeeds, right? I think it's the same for a lot of others who have submitted patches to it. A

Re: [DISCUSS] KIP-81: Max in-flight fetches

2016-10-26 Thread Jason Gustafson
Hey Mickael, Thanks for picking this up and sorry for the late comment. In the proposed changes section, you have the following: Update Fetcher.java to check the number of existing in-flight fetches (this > is already tracked by numInFlightFetches) before initiating new fetch > requests in create

Re: [VOTE] KIP-85: Dynamic JAAS configuration for Kafka clients

2016-10-27 Thread Jason Gustafson
Thanks for the KIP! +1 On Wed, Oct 26, 2016 at 3:23 PM, Gwen Shapira wrote: > Woohoo! +1 (binding) > > On Wed, Oct 26, 2016 at 5:26 PM, Rajini Sivaram < > rajinisiva...@googlemail.com> wrote: > > > I would like to initiate the voting process for KIP-85: Dynamic JAAS > > configuration for Kafka C

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

2016-10-31 Thread Jason Gustafson
Great work, Becket! On Mon, Oct 31, 2016 at 10:54 AM, Onur Karaman < okara...@linkedin.com.invalid> wrote: > Congrats Becket! > > On Mon, Oct 31, 2016 at 10:35 AM, Joel Koshy wrote: > > > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > > committer and we are pleased to

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

2019-08-15 Thread Jason Gustafson
t this? > > +1 (binding) > > best, > Colin > > > On Mon, Aug 12, 2019, at 10:57, Guozhang Wang wrote: > > +1 (binding). > > > > Thanks Jason! > > > > On Wed, Aug 7, 2019 at 11:18 AM Jason Gustafson > wrote: > > > > > Hi All, >

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

2019-08-20 Thread Jason Gustafson
, I'm actually fine to name it > more > > > generally and leave a note that at the moment its value is defined as > the > > > zk version. > > > > > > > > > Guozhang > > > > > > > > > On Mon, Aug 12, 2019 at 2:22 PM Ja

[VOTE] KIP-352: Distinguish URPs caused by reassignment

2019-08-20 Thread Jason Gustafson
Hi All, I'd like to start a vote on KIP-352, which is a follow-up to KIP-455 to fix a long-known shortcoming of URP reporting and to improve reassignment monitoring: https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment . Note that I have added one n

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

2019-08-20 Thread Jason Gustafson
+1 Thanks for the KIP! -Jason On Thu, Aug 15, 2019 at 4:01 AM Jakub Scholz wrote: > +1 (non-binding) > > Jakub > > On Sat, Aug 10, 2019 at 8:34 PM Stanislav Kozlovski < > stanis...@confluent.io> > wrote: > > > Awesome KIP, +1 (non-binding) > > > > Thanks, > > Stanislav > > > > On Fri, Aug 9, 20

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

2019-08-20 Thread Jason Gustafson
ng AlterIsrRequest from a broker to the controller (at least for a > given partition) in order to set CurrentZkVersion properly? > > Jun > > On Tue, Aug 20, 2019 at 10:07 AM Jason Gustafson > wrote: > > > I'm going to close this vote. The final result is +5 with 3 bind

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

2019-08-21 Thread Jason Gustafson
-1. > 2. Epoch and producerId are provided >a) the provided producerId/epoch matches the current producerId/epoch: > i) if the epoch is not exhausted, bump the epoch > ii) if the epoch is exhausted, create a new PID with epoch=0 >b) the provided producerId/epoch m

Re: [VOTE] KIP-360: Improve handling of unknown producer when using EOS

2019-08-21 Thread Jason Gustafson
e exposed to users) for > future > >> developers, how about rename it to "required_epoch" and if it is set to > >> "-1" it means "not required, hence not checks"? > >> > >> Guozhang > >> > >> On T

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-08-23 Thread Jason Gustafson
Hi Arjun, Thanks for the KIP. Do we really need a JMX-based API? Is there literally anyone in the world that wants to use JMX if they don't have to? I thought one of the major motivations of KIP-412 was how much of a pain JMX is. Thanks, Jason On Mon, Aug 19, 2019 at 5:28 PM Arjun Satish wrote:

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

2019-08-23 Thread Jason Gustafson
, and calculate the URPs based > on that. +1 for this. (I assume Jason will update the KIP...) > > best, > Colin > > > > > > Taking that into account, +1 from me! (non-binding) > > > > On Fri, Aug 23, 2019 at 7:00 PM Colin McCabe wrote: > > > &g

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-08-23 Thread Jason Gustafson
lly turn off in Kafka. Now that we have a proper API with support in the AdminClient, we can deprecate and eventually remove the JMX endpoint. Thanks, Jason On Fri, Aug 23, 2019 at 10:49 AM Jason Gustafson wrote: > Hi Arjun, > > Thanks for the KIP. Do we really need a JMX-based API? Is

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

2019-08-26 Thread Jason Gustafson
Closing this vote. The final result is +9 with 4 binding votes. @Satish Sorry, I missed your question above. Good point about updating documentation. I will create a separate jira to make sure this gets done. -Jason On Fri, Aug 23, 2019 at 11:23 AM Jason Gustafson wrote: > Thanks Stan, g

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-08-26 Thread Jason Gustafson
to it. It will > give brokers multiple ways of changing log levels. and there is still a > consistent way of achieving cross component goals of the KIP. > > Best, > > > On Fri, Aug 23, 2019 at 4:12 PM Jason Gustafson > wrote: > > > Let me elaborate a little bit. W

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

2019-08-27 Thread Jason Gustafson
rted; some others (INVALID_PRODUCER_EPOCH) would cause the producer to > enter the FATAL_ERROR state, plus it would cause all future txns to be > aborted. > > Is that right? > > > Guozhang > > > On Wed, Aug 21, 2019 at 3:52 PM Matthias J. Sax > wrote: > > >

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-04 Thread Jason Gustafson
Hi Colin, Just a couple questions. 1. I think we discussed that we would do a lazy version bump of all protocols in order to get flexible version support. Can you add that to the KIP? 2. The doc mentions a bump to the request and response header formats to version 1. Currently there is no formal

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-06 Thread Jason Gustafson
+1 Thanks Colin. This is really going to help with compatibility. -Jason On Wed, Sep 4, 2019 at 1:34 PM Colin McCabe wrote: > On Wed, Sep 4, 2019, at 13:01, Jason Gustafson wrote: > > Hi Colin, > > > > Just a couple questions. > > > > 1. I think we discusse

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-09-06 Thread Jason Gustafson
ew endpoint. > > If this works with you all, I can update the KIP. Please let me know what > you think. > > Thanks everyone. > > Best, > > On Thu, Aug 29, 2019 at 10:14 AM Colin McCabe wrote: > > > On Mon, Aug 26, 2019, at 14:03, Jason Gustafson wrote: > &

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2019-09-09 Thread Jason Gustafson
+1 Thanks for the KIP. Just a couple comments below: 1. Kafka APIs traditionally leave off `get` from API names. How about `groupMetadata` instead of `getMetadata`? 2. I am guessing memberId and groupInstanceId should be nullable in the TxnOffsetCommit schema? 3. Just to clarify on the upgrade pro

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2019-09-09 Thread Jason Gustafson
;). Anyway, I'm satisfied if we just document the recommended approach and explain the risk if users don't follow it. -Jason On Mon, Sep 9, 2019 at 12:52 PM Boyang Chen wrote: > Thanks Jason for the vote! > > On Mon, Sep 9, 2019 at 12:07 PM Jason Gustafson > wrote: >

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-10 Thread Jason Gustafson
Hi Guozhang, I think the motivation for the new API makes sense. I've wanted something like this in the past. That said, do you think there is a substantial benefit from deprecating the old API? I can still see it being convenient in some cases and it's no real cost to maintain. Also, just a mino

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-10 Thread Jason Gustafson
breaking compatibility just for aesthetics. -Jason On Tue, Sep 10, 2019 at 9:41 AM Guozhang Wang wrote: > Thanks Jason! > > On Tue, Sep 10, 2019 at 9:07 AM Jason Gustafson > wrote: > > > Hi Guozhang, > > > > I think the motivation for the new API makes sense. I've wa

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-12 Thread Jason Gustafson
gt; > > On Tue, Sep 10, 2019 at 10:18 AM Matthias J. Sax < > > > matth...@confluent.io> > > > > > > wrote: > > > > > > > > > > > >> Thanks for the KIP Guozhang. > > > > > >> > > > > > >>> Another reason i

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-09-12 Thread Jason Gustafson
. I'll also modify the kip to make > > > these clear. > > > > > > On Fri, Sep 6, 2019 at 4:01 PM Jason Gustafson > wrote: > > > > > >> Hi Arjun, > > >> > > >> The updated KIP looks good. Just a couple questions: &g

[DISCUSS] KIP-524: Allow users to choose config source when describing configs

2019-09-13 Thread Jason Gustafson
Hi All, I have a minor KIP to improve the config tool: https://cwiki.apache.org/confluence/display/KAFKA/KIP-524%3A+Allow+users+to+choose+config+source+when+describing+configs. Let me know what you think. -Jason

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

2019-09-13 Thread Jason Gustafson
Hi Kevin, This looks reasonable to me. I'd also +1 Radai's suggestion if you're willing. Something like an idle ratio for the consumer would be helpful. Thanks, Jason On Fri, Sep 13, 2019 at 10:08 AM radai wrote: > while youre at it another metric that we have found to be useful is % > time sp

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-09-13 Thread Jason Gustafson
Late here, but I am a big +1. Great to see this finally happening. -Jason On Fri, Sep 13, 2019 at 11:43 AM Colin McCabe wrote: > Hi all, > > With 3 binding +1 votes from Gwen Shapira, Ismael Juma, and Bill Bejeck > and 5 non-binding +1 votes from Tom Bentley, Ron Dagostino, David Jacot, > Dhruv

Re: [DISCUSS] KIP-524: Allow users to choose config source when describing configs

2019-09-16 Thread Jason Gustafson
rote: > > > Ah, great idea. > > > > On Fri, Sep 13, 2019 at 8:47 AM Jason Gustafson > > wrote: > > > > > > Hi All, > > > > > > I have a minor KIP to improve the config tool: > > > > > > https://cwiki.apache.org/

Re: [DISCUSS] KIP-515: Reorganize checkpoint system in log cleaner to be per partition

2019-09-19 Thread Jason Gustafson
Hi Richard, Just reposting my comment from the JIRA: The underlying problem here also impacts the cleaning of transaction markers. We use the same delete horizon in order to tell when it is safe to remove the marker. If all the data from a transaction has been cleaned and the delete horizon has p

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

2019-09-20 Thread Jason Gustafson
rios. > > > > > > > > > > I have added this to the KIP: > > > > > * - poll-idle-ratio*: The fraction of time the consumer spent > waiting > > > for > > > > > the user to process records from poll. > > > > > > >

Re: [VOTE] KIP-517: Add consumer metrics to observe user poll behavior

2019-09-20 Thread Jason Gustafson
+1 Thanks! On Thu, Sep 19, 2019 at 11:22 PM Tom Bentley wrote: > +1 (non-binding). > > On Fri, Sep 20, 2019 at 7:00 AM Maulin Vasavada > > wrote: > > > +1 (non-binding). Thanks for the KIP. > > > > On Thu, Sep 19, 2019 at 10:38 PM Manikumar > > wrote: > > > > > +1 (binding), Thanks for the KIP

Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-20 Thread Jason Gustafson
+1 from me. This is a clever solution. Kind of a pity we couldn't work flexible version support into the response, but I understand why it is difficult. One minor nitpick: the INVALID_REQUEST error already exists. Are you intending to reuse it? Thanks, Jason On Fri, Sep 20, 2019 at 3:50 PM Konst

Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-23 Thread Jason Gustafson
Thanks David for the clarification. That sounds good. On Mon, Sep 23, 2019 at 12:35 AM David Jacot wrote: > Hi all, > > The vote has passed with +3 binding votes (Colin McCabe, Gwen Shapira, > Jason Gustafson) and +3 non binding votes (Mickael Maison, Konstantine > Karantasis

[VOTE] KIP-524: Allow users to choose config source when describing configs

2019-09-24 Thread Jason Gustafson
Hi All, I'm starting a vote for KIP-524, which is a small change to the config tool: https://cwiki.apache.org/confluence/display/KAFKA/KIP-524%3A+Allow+users+to+choose+config+source+when+describing+configs . +1 from me. Thanks, Jason

Re: [DISCUSS] KIP-515: Reorganize checkpoint system in log cleaner to be per partition

2019-09-24 Thread Jason Gustafson
o I created this KIP only > with resolving the problem regarding tombstones. > Whats your thoughts? If the problem regarding transaction markers is a > little too complex, then we can we just leave it out of the KIP and fix the > tombstones issue. > > Cheers, > Richard >

Re: [VOTE] 2.3.1 RC0

2019-09-24 Thread Jason Gustafson
Hi David, Thanks for running the release. I think we should consider getting this bug fixed: https://issues.apache.org/jira/browse/KAFKA-8896. The impact of this bug is that consumer groups cannot commit offsets or rebalance. The patch should be ready shortly. Thanks, Jason On Fri, Sep 13, 201

[DISCUSS] KIP-533: Add default api timeout to AdminClient

2019-10-09 Thread Jason Gustafson
Hi All, I wrote a short KIP to address a longstanding issue with timeout behavior in the AdminClient: https://cwiki.apache.org/confluence/display/KAFKA/KIP-533%3A+Add+default+api+timeout+to+AdminClient . Take a look and let me know what you think. Thanks, Jason

Re: [VOTE] KIP-524: Allow users to choose config source when describing configs

2019-10-09 Thread Jason Gustafson
gt; > > +1 (binding) > > > > > > Regards, > > > > > > Rajini > > > > > > On Wed, Sep 25, 2019 at 4:41 PM Colin McCabe > wrote: > > > > > > > Looks good. +1 (binding) > > > > > > > &g

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

2019-10-09 Thread Jason Gustafson
; exception hierarchy. > >>> >> > >>> >> > >>> >> -Matthias > >>> >> > >>> >> On 8/8/18 2:57 AM, Stanislav Kozlovski wrote: > >>> >> >> If you are inheriting from SerializationException, your derived > >>&g

[DISCUSS] KIP-537: Increase default zookeeper session timeout

2019-10-15 Thread Jason Gustafson
Hi All, I have a short KIP to raise the default zk session timeout: https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout. Please take a look. Thanks, Jason

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

2019-10-16 Thread Jason Gustafson
+1. Thanks Richard. On Wed, Oct 16, 2019 at 10:04 AM Richard Yu wrote: > Hi all, > > Want to try to get this KIP wrapped up. So it would be great if we can get > some votes. > > Cheers, > Richard > > On Tue, Oct 15, 2019 at 12:58 PM Jun Rao wrote: > > > Hi, Richard, > > > > Thanks for the updat

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

2019-10-17 Thread Jason Gustafson
batch (and practically the deltas will be all negative)? > > > > > > If that's case, could we do some back of the envelope calculation on > > what's > > > the possible smallest case of deltas? Note that since we use varInt for > >

Re: [DISCUSS] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

2019-10-21 Thread Jason Gustafson
+1 Looks reasonable to me. Tracking this per-processor is probably not that useful, but I agree it simplifies the implementation. -Jason On Wed, Oct 16, 2019 at 11:58 AM Colin McCabe wrote: > Hi all, > > I wrote a short KIP about adding a metric to track the number of open > connections with a

[VOTE] KIP-537: Increase default zookeeper session timeout

2019-10-21 Thread Jason Gustafson
I'd like to start a vote for KIP-537: https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout . +1 from me Thanks, Jason

Re: [VOTE] 2.3.1 RC2

2019-10-22 Thread Jason Gustafson
+1 I ran the basic quickstart on the 2.12 artifact and verified signatures/checksums. I also looked over the release notes. I see that KAFKA-8950 is included, so maybe they just need to be refreshed. Thanks for running the release! -Jason On Fri, Oct 18, 2019 at 5:23 AM David Arthur wrote: >

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

2019-10-22 Thread Jason Gustafson
+1 Thanks Colin. On Tue, Oct 22, 2019 at 12:42 AM Tom Bentley wrote: > +1 (non-binding). Thanks! > > On Tue, Oct 22, 2019 at 1:03 AM Gwen Shapira wrote: > > > +1 > > > > Thanks for leading this :) > > > > On Mon, Oct 21, 2019 at 3:43 PM Colin McCabe wrote: > > > > > > Hi all, > > > > > > I'd l

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

2019-10-24 Thread Jason Gustafson
gt; Thanks, > > Harsha > > > > On Tue, Oct 22, 2019 at 3:20 PM Colin McCabe wrote: > > > > > +1 (binding). > > > > > > Thanks, Jason. > > > > > > best, > > > Colin > > > > > > > > > On Mon, Oct 21

Re: [VOTE] KIP-543: Expand ConfigCommand's non-ZK functionality

2019-11-07 Thread Jason Gustafson
+1 Looking forward to this. I love the quality-of-life changes. On Thu, Nov 7, 2019 at 7:00 PM Manikumar wrote: > +1 (binding), Thanks for the KIP. > > > > On Fri, Nov 8, 2019 at 8:14 AM Gwen Shapira wrote: > > > +1 (binding) > > > > Thank you. > > > > On Thu, Nov 7, 2019, 8:41 AM Brian Byrne

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

2019-11-13 Thread Jason Gustafson
Ni Noa, Thanks for the KIP. I agree with the concerns about Metadata bloat. In fact, I have wanted a BrokerStatus API for a while now. Perhaps this is a good excuse to introduce it. I was thinking something like this: BrokerStatusRequest => BrokerId BrokerStatusResponse => Listeners => [Listen

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

2019-11-13 Thread Jason Gustafson
Sorry, that should be "Hi Noa" On Wed, Nov 13, 2019 at 8:43 AM Jason Gustafson wrote: > Ni Noa, > > Thanks for the KIP. I agree with the concerns about Metadata bloat. In > fact, I have wanted a BrokerStatus API for a while now. Perhaps this is a > good excuse to int

Re: [DISCUSS] KIP-533: Add default api timeout to AdminClient

2019-11-13 Thread Jason Gustafson
2:01 PM Ismael Juma wrote: > Thanks for the KIP, this makes sense to me. It will be good to align the > AdminClient better with the consumer and producer when it comes to > timeouts. > > Ismael > > On Wed, Oct 9, 2019 at 12:06 PM Jason Gustafson > wrote: > > >

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

2019-11-13 Thread Jason Gustafson
Hi Satish, Thanks for the KIP. I'm wondering how much of this problem can be addressed just by increasing the replication max lag? That was one of the purposes of KIP-537 (the default increased from 10s to 30s). Also, the new configurations seem quite low level. I think they will be hard for user

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

2020-07-13 Thread Jason Gustafson
; > > > > > > > >> > > > > > > And I have the following questions about the Proposal: > > > > >> > > > &

<    3   4   5   6   7   8   9   10   11   12   >