Build failed in Jenkins: kafka-trunk-jdk11 #768

See Changes: [matthias] KIP-476: Add new getAdmin method to KafkaClientSupplier (#7162) -- [...truncated 2.59 MB...] org.apache.kafka.streams.test.OutputVerifierTest >

[jira] [Resolved] (KAFKA-8325) Remove from the incomplete set failed. This should be impossible

[ https://issues.apache.org/jira/browse/KAFKA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-8325. Fix Version/s: 2.3.1 Resolution: Fixed > Remove from the incomplete set failed. This

Re: [VOTE] KIP-440: Extend Connect Converter to support headers

Thanks everyone! Closing the vote with +7 (+3 binding). On 2019/08/21 13:41:34, Bill Bejeck wrote: > Thanks for the KIP! This looks like a valuable addition. > > +1(binding) > > -Bill > > On Mon, Aug 5, 2019 at 6:15 PM Ryanne Dolan wrote: > > > +1, non-binding > > > > Ryanne > > > > On Mon

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Thanks, Colin. The changes you made to the KIP related to the bridge release help make it clearer. I still have some confusion about the phrase "The rolling upgrade from the bridge release will take several steps." This made me think you are talking about moving from the bridge release to some ot

Re: KIP Creation permission

Perfect. Done. On 8/21/19 6:06 PM, Renuka M wrote: > Yes.. username: rmetukuru > Email: renumetuk...@gmail.com > > Thanks > Renuka M > > On Wed, Aug 21, 2019 at 5:58 PM Matthias J. Sax > wrote: > >> There is no user with this name. >> >> Did you create an account? If yes, we need your wiki id

Build failed in Jenkins: kafka-trunk-jdk8 #3863

See Changes: [github] KAFKA-8594: Add version 2.3 to Streams system tests (#7131) -- [...truncated 5.90 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldNo

Re: KIP Creation permission

Yes.. username: rmetukuru Email: renumetuk...@gmail.com Thanks Renuka M On Wed, Aug 21, 2019 at 5:58 PM Matthias J. Sax wrote: > There is no user with this name. > > Did you create an account? If yes, we need your wiki id (ie, user name) > to grant permissions. > > > -Matthias > > On 8/21/19 5:

Re: KIP Creation permission

There is no user with this name. Did you create an account? If yes, we need your wiki id (ie, user name) to grant permissions. -Matthias On 8/21/19 5:51 PM, Renuka M wrote: > renumetuk...@gmail.com > > Thanks > Renuka M > > On Wed, Aug 21, 2019 at 5:28 PM Matthias J. Sax > wrote: > >> What

Re: KIP Creation permission

renumetuk...@gmail.com Thanks Renuka M On Wed, Aug 21, 2019 at 5:28 PM Matthias J. Sax wrote: > What is your wiki id? > > On 8/21/19 3:46 PM, Renuka M wrote: > > Hi Admin, > > > > Could you please provide me Permissions to create a KIP. > > > > Thanks > > Renuka M > > > >

Re: KIP Creation permission

What is your wiki id? On 8/21/19 3:46 PM, Renuka M wrote: > Hi Admin, > > Could you please provide me Permissions to create a KIP. > > Thanks > Renuka M > signature.asc Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk11 #767

See Changes: [github] KAFKA-8594: Add version 2.3 to Streams system tests (#7131) -- [...truncated 2.60 MB...] kafka.server.KafkaConfigTest > testListenerNamesWithAdverti

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

Hi all, I've made some tweaks to the KIP that I believe are improvements. More detail can be found on the KIP page itself, but as a brief summary, the three changes are: - The removal of the internal.request.verification property in favor of modifying the default value for the connect.protocol pr

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

+1 (non-binding) This will be great to have, thanks Jason! On Wed, Aug 21, 2019 at 4:29 AM Manikumar wrote: > +1 (binding). > > Thanks for the KIP. LGTM. > > > On Wed, Aug 21, 2019 at 3:12 PM Satish Duggana > wrote: > > > Hi Jason, > > +1 (non binding) Thanks for the KIP! > > > > Do we need to

Re: [VOTE] KIP-360: Improve handling of unknown producer when using EOS

Thanks Matthias. With that, I can close this vote. The total is +4 with 3 binding votes. -Jason On Wed, Aug 21, 2019 at 3:52 PM Matthias J. Sax wrote: > +1 (binding) > > On 4/10/19 12:17 AM, Magnus Edenhill wrote: > > +1 (non-binding) > > > > > > Den ons 10 apr. 2019 kl 02:38 skrev Guozhang Wan

KIP Creation permission

Hi Admin, Could you please provide me Permissions to create a KIP. Thanks Renuka M

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Thanks Jason! LGTM. On 8/21/19 3:07 PM, Jason Gustafson wrote: > Hi Matthias, > > Thanks, I appreciate the thorough review. I've revised the section to make > the logic clearer. I think you have it right except for the 1). We only > generate a new PID if the epoch cannot be incremented without o

Re: [VOTE] KIP-360: Improve handling of unknown producer when using EOS

+1 (binding) On 4/10/19 12:17 AM, Magnus Edenhill wrote: > +1 (non-binding) > > > Den ons 10 apr. 2019 kl 02:38 skrev Guozhang Wang : > >> +1 (binding). Thanks for the written KIP! The approach lgtm. >> >> One minor thing: the name of "last epoch" maybe a bit misleading (although >> it is for i

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Hi Matthias, Thanks, I appreciate the thorough review. I've revised the section to make the logic clearer. I think you have it right except for the 1). We only generate a new PID if the epoch cannot be incremented without overflow. -Jason On Tue, Aug 20, 2019 at 5:45 PM Matthias J. Sax wrote:

Re: ACL for group creation?

I think it's worth considering separating out the permissions needed to create a consumer group from the permissions needed to join one. We distinguish these permissions for topics, and people generally find it useful. We could start checking CREATE on GROUP, perhaps? It might be hard to do

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Hi Ryanne, Apache Ratis looks like a very interesting project, but I don't think this is the right use-case for it. At its heart, Apache Kafka is a system for managing logs. We should avoid adding a dependency on an external system to manage the logs of Kafka itself, since that is one of Kafk

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hi Guozhang, > My impression from your previous email is that inside the algorithm when we are "filling" them to instances some deterministic logic would be used to avoid the above case, is that correct? Yes, that was my plan, but I didn't formalize it. There was a requirement that the assignment

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Colin, have you considered leveraging Apache Ratis (incubating)? Ryanne On Wed, Aug 21, 2019, 1:28 PM Colin McCabe wrote: > On Wed, Aug 21, 2019, at 06:38, Eno Thereska wrote: > > Hi Colin, > > > > Nice KIP! For such a big change it would be good to add a pointer or > > two to related work that

Re: ACL for group creation?

+users mailing list David, I don't think I really understand your email. Are you saying that this can already be achieved only using the READ ACL? Thanks Adam On Wed, Aug 21, 2019 at 3:58 AM David Jacot wrote: > Hello, > > It would be better to ask such question on the user mailing list. >

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hello John, That sounds reasonable. Just double checked the code that with logging disabled the corresponding checkpoint file would not contain any values, just like a stateless task. So I think treating them logically the same is fine. Guozhang On Wed, Aug 21, 2019 at 11:41 AM John Roesler wr

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

On Tue, Aug 20, 2019, at 07:19, Ron Dagostino wrote: > Hi Colin. The diagram up at the top confused me -- specifically, the lines > connecting the controller/active-controller to the brokers. I had assumed > the arrows on those lines represented the direction of data flow, but that > is not the c

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Yes that makes sense to me. I was mainly curious to see how we would avoid threshing stateless tasks back-and-forth but can guarantee "convergence" since we do not require any stickiness. My impression from your previous email is that inside the algorithm when we are "filling" them to instances so

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hi again, Guozhang, While writing up the section on stateless tasks ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statelesstasks), I reconsidered whether stateful, but non-logged, tasks should actually re

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

On Wed, Aug 21, 2019, at 04:22, Ron Dagostino wrote: > Hi Colin. I like the concept of a "bridge release" for migrating off of > Zookeeper, but I worry that it may become a bottleneck if people hesitate > to replace Zookeeper -- they would be unable to adopt newer versions of > Kafka until taking

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

On Wed, Aug 21, 2019, at 06:38, Eno Thereska wrote: > Hi Colin, > > Nice KIP! For such a big change it would be good to add a pointer or > two to related work that provides some sort of soft proof that the > approach taken makes sense. Also such work often builds on other work > and it might be us

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Thanks, Guozhang. (Side note: I noticed on another pass over the discussion that I'd missed addressing your comment about the potential race condition between state cleanup and lag-based assignment. I've added a solution to the proposal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A

[jira] [Created] (KAFKA-8825) Add option to reset consumer offset by relative time

Jason Gustafson created KAFKA-8825: -- Summary: Add option to reset consumer offset by relative time Key: KAFKA-8825 URL: https://issues.apache.org/jira/browse/KAFKA-8825 Project: Kafka Issue

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

Hi David, Thanks for the KIP. I have a couple of questions. >> For the Java client, the idea is to define two constants in the code to >> store its name and its version. If possible, the version will be set >> automatically based on metadata coming from gradle (or the repo itself) to >> avoid h

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hello John, I've made another pass on the wiki page again, overall LGTM. One meta comment about the "stateless" tasks: how do we represent them in the metadata? Are they just treated as stateful tasks with logging disabled, or are specially handled? It is not very clear in the description. Guozh

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hi John, Thanks for the added section, I agree with your reasoning and I think we can still use the standby replicas now. Guozhang On Tue, Aug 20, 2019 at 3:13 PM John Roesler wrote: > Hi All, > > Thanks for the discussion. I've been considering the idea of giving the > "catching up" tasks a

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

I have also specifically called out that the assignment must achieve both "instance" and "task" balance: https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22 I've also addressed the problem o

[DISCUSS] KIP-510: Metrics library upgrade

Hi there, I've written KIP-510 proposing the upgrade of the metrics library which Kafka is using. Please, have a look and let me know what you think, Thanks, Mario

Re: [VOTE] KIP-440: Extend Connect Converter to support headers

Thanks for the KIP! This looks like a valuable addition. +1(binding) -Bill On Mon, Aug 5, 2019 at 6:15 PM Ryanne Dolan wrote: > +1, non-binding > > Ryanne > > On Mon, Aug 5, 2019 at 3:38 PM Randall Hauch wrote: > > > If my math is right, we have 3 non-binding +1 votes and 2 binding +1 > votes

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Hi Colin, Nice KIP! For such a big change it would be good to add a pointer or two to related work that provides some sort of soft proof that the approach taken makes sense. Also such work often builds on other work and it might be useful to trace its roots. May I recommend adding a pointer to "Ta

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

Congratulations, Mitchel! Thanks, Dongjin On Wed, Aug 21, 2019 at 3:38 AM Mitchell wrote: > Closing the Vote. > 4 Binding, 9 non-binding. No against. > > On Tue, Aug 20, 2019 at 1:35 PM Jason Gustafson > wrote: > > > > +1 Thanks for the KIP! > > > > -Jason > > > > On Thu, Aug 15, 2019 at 4:01

[DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

Hi all, I would like to start a discussion for KIP-511: https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers Let me know what you think. Best, David

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

Hi, +1 (binding). Thanks for the updated KIP. LGTM. Thanks, Manikumar On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass wrote: > Hi All, > > Bumping this, I'd be happy to get some additional feedback and/or votes. > > Thanks, > Viktor > > On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vas

Re: [DISCUSS] KIP-435: Internal Partition Reassignment Batching

Hey Folks, I think I'll open a vote early next week about this if there are no more feedback. Thanks, Viktor On Fri, Aug 9, 2019 at 5:25 PM Viktor Somogyi-Vass wrote: > Hey Stanislav, > > I reiterated on the current algorithm and indeed it would change the order > of replicas in ZK but wouldn'

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

+1 (binding). Thanks for the KIP. LGTM. On Wed, Aug 21, 2019 at 3:12 PM Satish Duggana wrote: > Hi Jason, > +1 (non binding) Thanks for the KIP! > > Do we need to have a separate JIRA to update the docs as it introduces new > metrics and a change in behavior for the existing metric? > > > > On

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Hi Colin. I like the concept of a "bridge release" for migrating off of Zookeeper, but I worry that it may become a bottleneck if people hesitate to replace Zookeeper -- they would be unable to adopt newer versions of Kafka until taking (what feels to them like) a giant leap. As an example, assum

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

Hi Jason, +1 (non binding) Thanks for the KIP! Do we need to have a separate JIRA to update the docs as it introduces new metrics and a change in behavior for the existing metric? On Wed, Aug 21, 2019 at 2:41 PM Mickael Maison wrote: > +1 (non binding) > Thanks Jason > > On Wed, Aug 21, 2019

[jira] [Created] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured

Ferran altimiras created KAFKA-8824: --- Summary: InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured Key: KAFKA-8824 URL: https://issues.apache.org/jira/browse/KAFKA-8824 Pr

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

+1 (non binding) Thanks Jason On Wed, Aug 21, 2019 at 8:15 AM David Jacot wrote: > > +1 (non-binding) > > Thanks for the KIP! > > Best, > David > > On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson wrote: > > > Hi All, > > > > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455 to

[jira] [Created] (KAFKA-8823) Retention is partially broken on recreated topic

Gregory Koshelev created KAFKA-8823: --- Summary: Retention is partially broken on recreated topic Key: KAFKA-8823 URL: https://issues.apache.org/jira/browse/KAFKA-8823 Project: Kafka Issue Ty

Re: ACL for group creation?

Hello, It would be better to ask such question on the user mailing list. The reason is that the group is created automatically when a consumer joins it. It is not created explicitly so it can be restricted. In your case, you could setup a ACL to authorize the application to only use the group yo

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

+1 (non-binding) Thanks for the KIP! Best, David On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson wrote: > Hi All, > > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455 to fix > a long-known shortcoming of URP reporting and to improve reassignment > monitoring: > > https://cw