Re: [VOTE] KIP-724: Drop support for message formats v0 and v1

[David Jacot]

+1 (binding), Thanks!

Le jeu. 10 juin 2021 à 05:52, Gwen Shapira  a
écrit :

> +1 (binding)
>
> Exciting to see this.
>
> On Wed, Jun 9, 2021, 11:29 AM Ismael Juma  wrote:
>
> > Hi all,
> >
> > Consensus was reached in the discussion thread and part of what is
> proposed
> > has to happen by 3.0, so starting the vote for KIP-724:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
> >
> > If there are concerns or objections, feel free to point them out in this
> > thread or the discuss thread.
> >
> > Ismael
> >
>

Re: [VOTE] KIP-618: Exactly-once support for source connectors

[Tom Bentley]

Hi Chris,

Thanks for the KIP and the discussion.

+1 (binding)

Kind regards,

Tom

On Thu, Jun 10, 2021 at 4:15 AM Randall Hauch  wrote:

> +1 (binding)
>
> I still have a few minor questions/suggestions about wording (e.g., in
> JavaDoc), but those can be handled via the discussion thread and won't
> change the intent of those API methods or the KIP.
>
> Best regards,
>
> Randall
>
> On Wed, Jun 9, 2021 at 7:32 AM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > Friendly reminder that the KIP freeze is today; please cast your votes if
> > you'd like to see this feature introduced in time for 3.0.
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Jun 7, 2021 at 1:49 AM Dongjin Lee  wrote:
> >
> > > +1 (non-binding).
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Fri, Jun 4, 2021 at 2:35 PM Ryanne Dolan 
> > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Thu, Jun 3, 2021, 10:23 AM Chris Egerton
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-618, which adds support for
> > > > exactly-once
> > > > > delivery guarantees for source connectors in the Kafka Connect
> > > framework.
> > > > >
> > > > > I suspect there might be a little more discussion to be had but
> with
> > > the
> > > > > KIP freeze deadline approaching, I wanted to give anyone following
> > > along
> > > > > the chance to cast a +1 as soon as they feel that we've gotten to a
> > > > > reasonable state.
> > > > >
> > > > > The KIP:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors
> > > > >
> > > > > The discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCAMdOrUX-VK5OSB3OOdJNXW_YKEJH9FjQZ4eyzr2GXUhSeDnF3Q%40mail.gmail.com%3E
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Ismael Juma]

KAFKA-12600 was a general change, not related to this tool specifically. I
am not convinced this tool is actually useful, I haven't seen anyone using
it in years.

Ismael

On Wed, Jun 9, 2021 at 9:51 PM Dongjin Lee  wrote:

> Hi Ismael,
>
> Before I submit this KIP, I reviewed some history. When KIP-499
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool
> >
> tried to resolve the inconsistencies between the command line tools, two
> tools were omitted, probably by mistake.
>
> - KAFKA-12878: Support --bootstrap-server kafka-streams-application-reset
> 
> - KAFKA-12899: Support --bootstrap-server in ReplicaVerificationTool
>  (this one)
>
> And it seems like this tool is still working. The last update was
> KAFKA-12600  by you,
> which will also be included in this 3.0.0 release. It is why I determined
> that this tool is worth updating.
>
> Thanks,
> Dongjin
>
> On Thu, Jun 10, 2021 at 1:26 PM Ismael Juma  wrote:
>
> > Hi Dongjin,
> >
> > Does this tool still work? I recall that there were some doubts about it
> > and that's why it wasn't updated previously.
> >
> > Ismael
> >
> > On Sat, Jun 5, 2021 at 2:38 PM Dongjin Lee  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > > ReplicaVerificationTool:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > >
> > > Best,
> > > Dongjin
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Dongjin Lee]

Hi Ismael,

Before I submit this KIP, I reviewed some history. When KIP-499

tried to resolve the inconsistencies between the command line tools, two
tools were omitted, probably by mistake.

- KAFKA-12878: Support --bootstrap-server kafka-streams-application-reset

- KAFKA-12899: Support --bootstrap-server in ReplicaVerificationTool
 (this one)

And it seems like this tool is still working. The last update was
KAFKA-12600  by you,
which will also be included in this 3.0.0 release. It is why I determined
that this tool is worth updating.

Thanks,
Dongjin

On Thu, Jun 10, 2021 at 1:26 PM Ismael Juma  wrote:

> Hi Dongjin,
>
> Does this tool still work? I recall that there were some doubts about it
> and that's why it wasn't updated previously.
>
> Ismael
>
> On Sat, Jun 5, 2021 at 2:38 PM Dongjin Lee  wrote:
>
> > Hi all,
> >
> > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > ReplicaVerificationTool:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> >
> > Best,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Ryan Leslie]

Thanks for the quick replies, Luke and Sophie.

I've not voted, but I agree with accepting the KIP since it's a superior 
feature. I was just reacting mostly to this comment since it didn't mention 
open issues:

> > > Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully
> > > enable cooperative rebalancing
> > > by default in 3.0 if we have KAFKA-12477 done in time, and if we don't
> > then
> > > there's no harm as it's
> > > not going to change the behavior.

But I see now, as Luke said, that the main issue is already considered a 
blocker so it was assumed. Though, I did also wonder if any bugs that may have 
existed since several version ago should actually hold up 3.0, which I know is 
especially about moving away from ZooKeeper.

My sentiment was just that during many release cycles of Kafka since 
cooperative was introduced, there have been issues discovered. And that makes 
sense given that the implementation was complex and quite a lot of code changed 
to make it happen. Hopefully the last of the kinks will have been worked out 
before 3.0. I just wondered if it should be a default in 3.0 if it hasn't yet 
been free of defects for a significant period of time. KIP-726 doesn't list any 
drawbacks for cooperative-sticky, but perhaps this is one. I also appreciate 
that it's already successfully adopted by many, particularly streams / connect 
users. But this may also be where the feature has the most benefit due to 
expensive setup/teardown during rebalance, and stop-the-world can be less of a 
concern for many "regular consumers".

This is probably irrelevant here, but another thing to mention is that the 
feature is also very new for C/C++ users in librdkafka, so not many in that 
community have tried incremental cooperative yet. And bugs are still recently 
being worked out there too.

Just playing devil's advocate here, sorry to come across as a negative nancy!

On 2021/06/09 00:05:41, Sophie Blee-Goldman  
wrote: 
> Hey Ryan,
> 
> Yes, I believe any open bugs regarding the cooperative-sticky assignor
> should be considered as blockers
> to it being made the default, if not blockers to the release in general. I
> don't think they need to block the
> acceptance of this KIP, though, just possibly the implementation of it.

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Ismael Juma]

Hi Dongjin,

Does this tool still work? I recall that there were some doubts about it
and that's why it wasn't updated previously.

Ismael

On Sat, Jun 5, 2021 at 2:38 PM Dongjin Lee  wrote:

> Hi all,
>
> I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> ReplicaVerificationTool:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Randall Hauch]

Thanks for the straightforward KIP, Dongjin.

+1 (binding)

Best regards,

Randall

On Wed, Jun 9, 2021 at 9:41 PM wenbing shen 
wrote:

> Hi Dongjin,
> Thanks for the KIP.
> +1 (non-binding)
>
> Thanks.
> Wenbing
>
> Dongjin Lee  于2021年6月10日周四 上午10:04写道：
>
> > Thanks, Gwen and Luke,
> >
> > As of present:
> > - binding: +1 (Gwen)
> > - non-binding: +1 (Luke)
> >
> > Anyone else?
> >
> > Regards,
> > Dongjin
> >
> > On Thu, Jun 10, 2021 at 1:24 AM Gwen Shapira 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On Wed, Jun 9, 2021 at 8:28 AM Dongjin Lee  wrote:
> > >
> > > > Bumping up the voting thread.
> > > >
> > > > Please note that today is the KIP freeze day. +1 non-binding until
> now.
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Sun, Jun 6, 2021 at 4:10 PM Luke Chen  wrote:
> > > >
> > > > > Hi Dongjin,
> > > > > Thanks for the KIP.
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > > On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee 
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to call for a vote on KIP-752: Support
> --bootstrap-server
> > in
> > > > > > ReplicaVerificationTool:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > > > > >
> > > > > > Best,
> > > > > > Dongjin
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > >
> > > > > >
> > > > > >
> > > > > > *github:  github.com/dongjinleekr
> > > > > > keybase:
> > > > > https://keybase.io/dongjinleekr
> > > > > > linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > speakerdeck:
> > > > > > speakerdeck.com/dongjin
> > > > > > *
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > >
> > > >
> > > > *github:  github.com/dongjinleekr
> > > > keybase:
> > > https://keybase.io/dongjinleekr
> > > > linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > speakerdeck:
> > > > speakerdeck.com/dongjin
> > > > *
> > > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Engineering Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>

Re: [VOTE] KIP-724: Drop support for message formats v0 and v1

[Gwen Shapira]

+1 (binding)

Exciting to see this.

On Wed, Jun 9, 2021, 11:29 AM Ismael Juma  wrote:

> Hi all,
>
> Consensus was reached in the discussion thread and part of what is proposed
> has to happen by 3.0, so starting the vote for KIP-724:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
>
> If there are concerns or objections, feel free to point them out in this
> thread or the discuss thread.
>
> Ismael
>

Re: [VOTE] KIP-618: Exactly-once support for source connectors

[Randall Hauch]

+1 (binding)

I still have a few minor questions/suggestions about wording (e.g., in
JavaDoc), but those can be handled via the discussion thread and won't
change the intent of those API methods or the KIP.

Best regards,

Randall

On Wed, Jun 9, 2021 at 7:32 AM Chris Egerton 
wrote:

> Hi all,
>
> Friendly reminder that the KIP freeze is today; please cast your votes if
> you'd like to see this feature introduced in time for 3.0.
>
> Cheers,
>
> Chris
>
> On Mon, Jun 7, 2021 at 1:49 AM Dongjin Lee  wrote:
>
> > +1 (non-binding).
> >
> > Thanks,
> > Dongjin
> >
> > On Fri, Jun 4, 2021 at 2:35 PM Ryanne Dolan 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Thu, Jun 3, 2021, 10:23 AM Chris Egerton
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-618, which adds support for
> > > exactly-once
> > > > delivery guarantees for source connectors in the Kafka Connect
> > framework.
> > > >
> > > > I suspect there might be a little more discussion to be had but with
> > the
> > > > KIP freeze deadline approaching, I wanted to give anyone following
> > along
> > > > the chance to cast a +1 as soon as they feel that we've gotten to a
> > > > reasonable state.
> > > >
> > > > The KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors
> > > >
> > > > The discussion thread:
> > > >
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCAMdOrUX-VK5OSB3OOdJNXW_YKEJH9FjQZ4eyzr2GXUhSeDnF3Q%40mail.gmail.com%3E
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>

Re: [VOTE] KIP-745: Connect API to restart connector and tasks

[Dongjin Lee]

+1 (non-binding).

As of present:

- binding: +2 (Randall, Konstantine)
- non-binding: +3 (Ryanne, Kalpesh, Dongjin)

We need one more +1 binding.

Thanks,
Dongjin

On Tue, Jun 8, 2021 at 6:31 AM Kalpesh Patel 
wrote:

> +1 (non-binding)
>
> Regards
> -Kalpesh
>
> On Mon, Jun 7, 2021 at 3:10 PM Ryanne Dolan  wrote:
>
> > +1 (non-binding)
> >
> > Ryanne
> >
> > On Mon, Jun 7, 2021, 3:03 PM Konstantine Karantasis
> >  wrote:
> >
> > > Thanks Randall.
> > >
> > > +1 (binding)
> > >
> > > Konstantine
> > >
> > > On Mon, Jun 7, 2021 at 12:47 PM Randall Hauch 
> wrote:
> > >
> > > > Hello all,
> > > >
> > > > I would like to start a vote on KIP-745:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> > > >
> > > > +1 (binding) from myself.
> > > >
> > > > Thanks, and best regards!
> > > >
> > > > Randall
> > > >
> > >
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[wenbing shen]

Hi Dongjin,
Thanks for the KIP.
+1 (non-binding)

Thanks.
Wenbing

Dongjin Lee  于2021年6月10日周四 上午10:04写道：

> Thanks, Gwen and Luke,
>
> As of present:
> - binding: +1 (Gwen)
> - non-binding: +1 (Luke)
>
> Anyone else?
>
> Regards,
> Dongjin
>
> On Thu, Jun 10, 2021 at 1:24 AM Gwen Shapira 
> wrote:
>
> > +1 (binding)
> >
> > On Wed, Jun 9, 2021 at 8:28 AM Dongjin Lee  wrote:
> >
> > > Bumping up the voting thread.
> > >
> > > Please note that today is the KIP freeze day. +1 non-binding until now.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Sun, Jun 6, 2021 at 4:10 PM Luke Chen  wrote:
> > >
> > > > Hi Dongjin,
> > > > Thanks for the KIP.
> > > > +1 (non-binding)
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee 
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-752: Support --bootstrap-server
> in
> > > > > ReplicaVerificationTool:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > > > >
> > > > > Best,
> > > > > Dongjin
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > >
> > > > >
> > > > >
> > > > > *github:  github.com/dongjinleekr
> > > > > keybase:
> > > > https://keybase.io/dongjinleekr
> > > > > linkedin:
> > > > kr.linkedin.com/in/dongjinleekr
> > > > > speakerdeck:
> > > > > speakerdeck.com/dongjin
> > > > > *
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Dongjin Lee]

Thanks, Gwen and Luke,

As of present:
- binding: +1 (Gwen)
- non-binding: +1 (Luke)

Anyone else?

Regards,
Dongjin

On Thu, Jun 10, 2021 at 1:24 AM Gwen Shapira 
wrote:

> +1 (binding)
>
> On Wed, Jun 9, 2021 at 8:28 AM Dongjin Lee  wrote:
>
> > Bumping up the voting thread.
> >
> > Please note that today is the KIP freeze day. +1 non-binding until now.
> >
> > Thanks,
> > Dongjin
> >
> > On Sun, Jun 6, 2021 at 4:10 PM Luke Chen  wrote:
> >
> > > Hi Dongjin,
> > > Thanks for the KIP.
> > > +1 (non-binding)
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > > > ReplicaVerificationTool:
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > >
> > > >
> > > > *github:  github.com/dongjinleekr
> > > > keybase:
> > > https://keybase.io/dongjinleekr
> > > > linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > speakerdeck:
> > > > speakerdeck.com/dongjin
> > > > *
> > > >
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

[Dongjin Lee]

Thanks Ismel, Tom and Ryanne,

I am now updating the KIP about the further works. Sure, You won't be
disappointed.

As of Present:

- binding: +2 (Ismael, Tom)
- non-binding: +1 (Ryanne)

Anyone else?

Best,
Dongjin

On Thu, Jun 10, 2021 at 2:03 AM Tom Bentley  wrote:

> Hi Dongjin,
>
> Thanks for the KIP, +1 (binding).
>
> Kind regards,
>
> Tom
>
> On Wed, Jun 9, 2021 at 5:16 PM Ismael Juma  wrote:
>
> > I'm +1 on the proposed change. As I stated in the discuss thread, I don't
> > think we should rule out the buffer size config, but we could list that
> as
> > future work vs rejected alternatives.
> >
> > Ismael
> >
> > On Sat, Jun 5, 2021 at 2:37 PM Dongjin Lee  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to open a voting thread for KIP-390: Support Compression Level
> > > (rebooted):
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
> > >
> > > Best,
> > > Dongjin
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #208

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 474334 lines...]
[2021-06-10T01:45:02.804Z] > Task :clients:jar UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :server-common:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :storage:api:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:api:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:api:classes UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :streams:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :streams:classes UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :streams:jar UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:json:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:json:classes UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:json:javadoc SKIPPED
[2021-06-10T01:45:03.735Z] > Task :raft:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :storage:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:json:javadocJar
[2021-06-10T01:45:03.735Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :clients:compileTestJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :clients:testClasses UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :metadata:compileJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :core:compileJava NO-SOURCE
[2021-06-10T01:45:03.735Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-06-10T01:45:03.735Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:json:testClasses UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :connect:json:testJar
[2021-06-10T01:45:03.735Z] > Task :connect:json:testSrcJar
[2021-06-10T01:45:03.735Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-06-10T01:45:03.735Z] > Task :core:compileScala UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :core:classes UP-TO-DATE
[2021-06-10T01:45:03.735Z] > Task :core:compileTestJava NO-SOURCE
[2021-06-10T01:45:04.719Z] > Task :core:compileTestScala UP-TO-DATE
[2021-06-10T01:45:04.719Z] > Task :core:testClasses UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task :connect:api:javadoc
[2021-06-10T01:45:09.313Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task :connect:api:jar UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-06-10T01:45:09.313Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task :connect:json:jar UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-06-10T01:45:09.313Z] > Task :connect:api:javadocJar
[2021-06-10T01:45:09.313Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-06-10T01:45:09.313Z] > Task :connect:json:publishToMavenLocal
[2021-06-10T01:45:09.313Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-06-10T01:45:09.313Z] > Task :connect:api:testJar
[2021-06-10T01:45:09.313Z] > Task :connect:api:testSrcJar
[2021-06-10T01:45:09.313Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-06-10T01:45:09.313Z] > Task :connect:api:publishToMavenLocal
[2021-06-10T01:45:11.695Z] > Task :streams:javadoc
[2021-06-10T01:45:11.695Z] > Task :streams:javadocJar
[2021-06-10T01:45:12.710Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-06-10T01:45:12.710Z] > Task :streams:testClasses UP-TO-DATE
[2021-06-10T01:45:12.710Z] > Task :streams:testJar
[2021-06-10T01:45:12.710Z] > Task :streams:testSrcJar
[2021-06-10T01:45:13.642Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-06-10T01:45:13.642Z] > Task :streams:publishToMavenLocal
[2021-06-10T01:45:14.572Z] > Task :clients:javadoc
[2021-06-10T01:45:15.501Z] > Task :clients:javadocJar
[2021-06-10T01:45:16.432Z] > Task :clients:testJar
[2021-06-10T01:45:17.362Z] > Task :clients:testSrcJar
[2021-06-10T01:45:17.362Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-06-10T01:45:17.362Z] > Task :clients:publishToMavenLocal
[2021-06-10T01:45:17.362Z] 
[2021-06-10T01:45:17.362Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-06-10T01:45:17.362Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2021-06-10T01:45:17.362Z] See 
https://docs.gradle.org/7.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-06-10T01:45:17.362Z] 
[2021-06-10T01:45:17.362Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-06-10T01:45:17.362Z] Please consult deprecation warnings for more details.
[2021-06-10T01:45:17.362Z] 
[2021-06-10T01:45:17.362Z] BUILD SUCCESSFUL in 37s
[2021-06-10T01:45:17.362Z] 71 actionable tasks

[jira] [Created] (KAFKA-12927) Kafka account no authentication failure anti-theft cracking mechanism

[SingleThread (Jira)]

SingleThread created KAFKA-12927:


 Summary: Kafka account no authentication failure anti-theft 
cracking mechanism
 Key: KAFKA-12927
 URL: https://issues.apache.org/jira/browse/KAFKA-12927
 Project: Kafka
  Issue Type: Improvement
Reporter: SingleThread
 Attachments: image-2021-06-10-09-43-32-739.png

After the Kafka account no authentication fails, there is no policy similar to 
account lockout, so there is a security problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12920) Consumer's cooperative sticky assignor need to clear generation / assignment data upon `onPartitionsLost`

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-12920.

Resolution: Not A Bug

> Consumer's cooperative sticky assignor need to clear generation / assignment 
> data upon `onPartitionsLost`
> -
>
> Key: KAFKA-12920
> URL: https://issues.apache.org/jira/browse/KAFKA-12920
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: bug, consumer
>
> Consumer's cooperative-sticky assignor does not track the owned partitions 
> inside the assignor --- i.e. when it reset its state in event of 
> ``onPartitionsLost``, the ``memberAssignment`` and ``generation`` inside the 
> assignor would not be cleared. This would cause a member to join with empty 
> generation on the protocol while with non-empty user-data encoding the old 
> assignment still (and hence pass the validation check on broker side during 
> JoinGroup), and eventually cause a single partition to be assigned to 
> multiple consumers within a generation.
> We should let the assignor to also clear its assignment/generation when 
> ``onPartitionsLost`` is triggered in order to avoid this scenario.
> Note that 1) for the regular sticky assignor the generation would still have 
> an older value, and this would cause the previously owned partitions to be 
> discarded during the assignment, and 2) for Streams' sticky assignor, it’s 
> encoding would indeed be cleared along with ``onPartitionsLost``. Hence only 
> Consumer's cooperative-sticky assignor have this issue to solve.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Colin McCabe]

+1 (binding).

Looking forward to 4.0 (heh)

best,
Colin

On Wed, Jun 9, 2021, at 02:45, Tom Bentley wrote:
> Hi Ismael,
> 
> Thanks for the KIP, +1 (binding).
> 
> Cheers,
> 
> Tom
> 
> On Wed, Jun 9, 2021 at 10:04 AM Mickael Maison 
> wrote:
> 
> > +1 (binding)
> > Thanks
> >
> > On Wed, Jun 9, 2021 at 1:25 AM Israel Ekpo  wrote:
> > >
> > > This is great!
> > >
> > > Thanks for the KIP
> > >
> > > +1 (non-binding)
> > >
> > >
> > > On Tue, Jun 8, 2021 at 3:13 AM Bruno Cadonna  wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Thank you for the KIP!
> > > >
> > > > +1 (binding)
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On 08.06.21 10:11, Matthew de Detrich wrote:
> > > > > +1 (non binding)
> > > > >
> > > > > On Tue, Jun 8, 2021 at 4:48 AM Luke Chen  wrote:
> > > > >
> > > > >> Hi Ismael,
> > > > >> Thanks for the KIP.
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> Thanks.
> > > > >> Luke
> > > > >>
> > > > >> On Tue, Jun 8, 2021 at 7:18 AM Konstantine Karantasis
> > > > >>  wrote:
> > > > >>
> > > > >>> Thanks for the KIP Ismael. I agree that giving an early enough
> > > > >> deprecation
> > > > >>> notice for an upcoming change like this one and doing so in a major
> > > > >> release
> > > > >>> is best in this case.
> > > > >>>
> > > > >>> +1 (binding)
> > > > >>>
> > > > >>> Konstantine
> > > > >>>
> > > > >>> On Mon, Jun 7, 2021 at 7:30 AM Ryanne Dolan  > >
> > > > >> wrote:
> > > > >>>
> > > >  +1 (non-binding)
> > > > 
> > > >  Ryanne
> > > > 
> > > >  On Mon, Jun 7, 2021, 9:26 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > > >
> > > >  wrote:
> > > > 
> > > > > +1 (non-binding)
> > > > >
> > > > > On Mon, 7 Jun 2021 at 19:30, Dongjin Lee 
> > wrote:
> > > > >
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> As a note: I think the exact removal schedule may be changed.
> > > > >>
> > > > >> Best,
> > > > >> Dongjin
> > > > >>
> > > > >> On Mon, Jun 7, 2021, 10:56 PM Josep Prat
> > > > >>  > > > 
> > > > >> wrote:
> > > > >>
> > > > >>> Thanks Ismael, it's a +1 (non-binding) from
> > > > <
> > https://www.google.com/maps/search/Ismael,+it's+a+%2B1+(non-binding)+from?entry=gmail&source=g
> > >
> > > > my side,
> > > > >>>
> > > > >>>
> > > > >>> Best,
> > > > >>>
> > > > >>> ———
> > > > >>> Josep Prat
> > > > >>>
> > > > >>> Aiven Deutschland GmbH
> > > > >>>
> > > > >>> Immanuelkirchstraße 26, 10405 Berlin
> > > > >>>
> > > > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>>
> > > > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >>>
> > > > >>> m: +491715557497
> > > > >>>
> > > > >>> w: aiven.io
> > > > >>>
> > > > >>> e: josep.p...@aiven.io
> > > > >>>
> > > > >>> On Mon, Jun 7, 2021, 15:51 Ismael Juma 
> > > > >> wrote:
> > > > >>>
> > > >  Since the goal is to provide ample warning regarding the
> > future
> > > > > removal
> > > > >>> of
> > > >  Java 8 support and the change we're proposing for 3.0 is
> > > > >> simply a
> > > >  documentation update, I am starting the vote:
> > > > 
> > > > 
> > > > >>>
> > > > >>
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > > > 
> > > >  If there are concerns or objections, feel free to point them
> > > > >> out
> > > > >>> in
> > > > >> this
> > > >  thread or the discuss thread.
> > > > 
> > > >  Ismael
> > > > 
> > > > >>>
> > > > >>
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > >
> >
> >
> 

Re: [VOTE] KIP-724: Drop support for message formats v0 and v1

[Colin McCabe]

+1 (binding)

Thanks, Ismael.

best,
Colin

On Wed, Jun 9, 2021, at 17:41, Dongjin Lee wrote:
> +1 (non-binding).
> 
> Best,
> Dongjin
> 
> On Thu, Jun 10, 2021, 7:20 AM Jason Gustafson 
> wrote:
> 
> > +1 Thanks Ismael!
> >
> > On Wed, Jun 9, 2021 at 11:29 AM Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > Consensus was reached in the discussion thread and part of what is
> > proposed
> > > has to happen by 3.0, so starting the vote for KIP-724:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
> > >
> > > If there are concerns or objections, feel free to point them out in this
> > > thread or the discuss thread.
> > >
> > > Ismael
> > >
> >
> 

Re: [VOTE] KIP-746: Revise KRaft Metadata Records

[David Arthur]

Thanks, Colin, looks good to me. +1

On Wed, Jun 9, 2021 at 8:32 PM Israel Ekpo  wrote:

> Makes sense to me
>
> +1 (non-binding)
>
>
> On Wed, Jun 9, 2021 at 7:05 PM Jun Rao  wrote:
>
> > Hi, Colin,
> >
> > Thanks for the KIP. +1 from me.
> >
> > Jun
> >
> > On Wed, Jun 9, 2021 at 9:36 AM Jason Gustafson
>  > >
> > wrote:
> >
> > > +1 Thanks Colin!
> > >
> > > On Thu, Jun 3, 2021 at 4:30 PM Colin McCabe 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call a vote for KIP-746: Revise KRaft Metadata Records.
> > This
> > > > is a minor KIP which revises the KRaft metadata records slightly for
> > the
> > > > upcoming 3.0 release.
> > > >
> > > > The KIP is at: https://cwiki.apache.org/confluence/x/34zOCg
> > > >
> > > > best,
> > > > Colin
> > > >
> > >
> >
>


-- 
David Arthur

Re: [VOTE] KIP-724: Drop support for message formats v0 and v1

[Dongjin Lee]

+1 (non-binding).

Best,
Dongjin

On Thu, Jun 10, 2021, 7:20 AM Jason Gustafson 
wrote:

> +1 Thanks Ismael!
>
> On Wed, Jun 9, 2021 at 11:29 AM Ismael Juma  wrote:
>
> > Hi all,
> >
> > Consensus was reached in the discussion thread and part of what is
> proposed
> > has to happen by 3.0, so starting the vote for KIP-724:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
> >
> > If there are concerns or objections, feel free to point them out in this
> > thread or the discuss thread.
> >
> > Ismael
> >
>

Re: [VOTE] KIP-746: Revise KRaft Metadata Records

[Israel Ekpo]

Makes sense to me

+1 (non-binding)


On Wed, Jun 9, 2021 at 7:05 PM Jun Rao  wrote:

> Hi, Colin,
>
> Thanks for the KIP. +1 from me.
>
> Jun
>
> On Wed, Jun 9, 2021 at 9:36 AM Jason Gustafson  >
> wrote:
>
> > +1 Thanks Colin!
> >
> > On Thu, Jun 3, 2021 at 4:30 PM Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call a vote for KIP-746: Revise KRaft Metadata Records.
> This
> > > is a minor KIP which revises the KRaft metadata records slightly for
> the
> > > upcoming 3.0 release.
> > >
> > > The KIP is at: https://cwiki.apache.org/confluence/x/34zOCg
> > >
> > > best,
> > > Colin
> > >
> >
>

[jira] [Created] (KAFKA-12926) ConsumerGroupCommand's java.lang.NullPointerException at negative offsets while running kafka-consumer-groups.sh

[Ignacio Acuna (Jira)]

Ignacio Acuna created KAFKA-12926:
-

 Summary: ConsumerGroupCommand's java.lang.NullPointerException at 
negative offsets while running kafka-consumer-groups.sh
 Key: KAFKA-12926
 URL: https://issues.apache.org/jira/browse/KAFKA-12926
 Project: Kafka
  Issue Type: Bug
  Components: admin, clients
Reporter: Ignacio Acuna
Assignee: Ignacio Acuna


Hi everybody, hope everyone is doing great.

*i) Introduction:*
I noticed the following exception (on a cluster with brokers running 2.3.1) 
when trying to describe a consumer group (using the Kafka 2.7.1):

 
{code:java}
./kafka-consumer-groups.sh --describe --group order-validations{code}
{code:java}
Error: Executing consumer group command failed due to null
java.lang.NullPointerException
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$collectGroupsOffsets$6(ConsumerGroupCommand.scala:579)
 at 
scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:99)
 at 
scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:86)
 at 
scala.collection.convert.JavaCollectionWrappers$JSetWrapper.map(JavaCollectionWrappers.scala:180)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$collectGroupsOffsets$5(ConsumerGroupCommand.scala:578)
 at scala.collection.immutable.List.flatMap(List.scala:293)
 at scala.collection.immutable.List.flatMap(List.scala:79)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$collectGroupsOffsets$2(ConsumerGroupCommand.scala:574)
 at scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
 at scala.collection.mutable.Growable.addAll(Growable.scala:62)
 at scala.collection.mutable.Growable.addAll$(Growable.scala:59)
 at scala.collection.mutable.HashMap.addAll(HashMap.scala:117)
 at scala.collection.mutable.HashMap$.from(HashMap.scala:570)
 at scala.collection.mutable.HashMap$.from(HashMap.scala:563)
 at scala.collection.MapOps$WithFilter.map(Map.scala:358)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsOffsets(ConsumerGroupCommand.scala:569)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeGroups(ConsumerGroupCommand.scala:369)
 at kafka.admin.ConsumerGroupCommand$.run(ConsumerGroupCommand.scala:76)
 at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
 at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala){code}
 

When trying on and older version of AdminClient (2.3.1):
{code:java}
Error: Executing consumer group command failed due to 
java.lang.IllegalArgumentException: Invalid negative offset
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
Invalid negative offset
 at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
 at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
 at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
 at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.getCommittedOffsets(ConsumerGroupCommand.scala:595)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$collectGroupsOffsets$2(ConsumerGroupCommand.scala:421)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService$$Lambda$131/4CB1EFD0.apply(Unknown
 Source)
 at 
scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:827)
 at 
scala.collection.TraversableLike$WithFilter$$Lambda$132/4CD49E20.apply(Unknown
 Source)
 at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
 at scala.collection.mutable.HashMap$$Lambda$133/4CD4A4F0.apply(Unknown 
Source)
 at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
 at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
 at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
 at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
 at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:826)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsOffsets(ConsumerGroupCommand.scala:419)
 at 
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeGroups(ConsumerGroupCommand.scala:312)
 at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
 at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: java.lang.IllegalArgumentException: Invalid negative offset
 at 
org.apache.kafka.clients.consumer.OffsetAndMetadata.(OffsetAndMetadata.java:50)
 at 
org.apache.kafka.clients.admin.KafkaAdminClient$24$1.handleResponse(KafkaAdminClient.java:2832)
 at 
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1032)
 at 
org.apache.kafka.

Re: [DISCUSS] KIP-756: Move StreamsResetter tool outside of core

[Matthias J. Sax]

Thanks for the KIP.

The KIP seems to post quite some implementation details, why? We should
focus on the public API only to make it easier to read.

The new class also does not contain a `main()` method that we need to
run the resetter as command line tool.

nit: can we use `applicationId` instead of `groupId` ?

Should `Properties` be optional for `StreamsResetter`?

Should the constructors of `StreamsResetterOptions` be private (similar
to `ResetOffsetsOptions` ?

Why is `DATETIME_FORMAT` public?

Should we follow a builder pattern for `StreamsResetterOptions` to force
consumer removal and for dry run?


-Matthias

On 6/8/21 12:30 PM, Jorge Esteban Quilcate Otoya wrote:
> Hi everyone,
> 
> Hope you are all well and safe.
> 
> I'd like to propose the following KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-756%3A+Move+StreamsResetter+tool+outside+of+core
> 
> The goal is to move `StreamsResetter` tool outside of `core` module, and
> allow Kafka Streams users to access it from their applications. More info
> here: https://issues.apache.org/jira/browse/KAFKA-4327
> 
> Looking forward to your feedback.
> 
> Cheers,
> Jorge.
> 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #207

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 473998 lines...]
[2021-06-09T23:21:55.759Z] [INFO] 

[2021-06-09T23:21:55.759Z] [INFO] Parameter: groupId, Value: streams.examples
[2021-06-09T23:21:55.759Z] [INFO] Parameter: artifactId, Value: streams.examples
[2021-06-09T23:21:55.759Z] [INFO] Parameter: version, Value: 0.1
[2021-06-09T23:21:55.759Z] [INFO] Parameter: package, Value: myapps
[2021-06-09T23:21:55.759Z] [INFO] Parameter: packageInPathFormat, Value: myapps
[2021-06-09T23:21:55.759Z] [INFO] Parameter: package, Value: myapps
[2021-06-09T23:21:55.759Z] [INFO] Parameter: version, Value: 0.1
[2021-06-09T23:21:55.759Z] [INFO] Parameter: groupId, Value: streams.examples
[2021-06-09T23:21:55.759Z] [INFO] Parameter: artifactId, Value: streams.examples
[2021-06-09T23:21:56.422Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[2021-06-09T23:21:56.422Z] [INFO] 

[2021-06-09T23:21:56.422Z] [INFO] BUILD SUCCESS
[2021-06-09T23:21:56.422Z] [INFO] 

[2021-06-09T23:21:56.422Z] [INFO] Total time:  1.680 s
[2021-06-09T23:21:56.422Z] [INFO] Finished at: 2021-06-09T23:21:55Z
[2021-06-09T23:21:56.422Z] [INFO] 

[Pipeline] dir
[2021-06-09T23:21:56.423Z] Running in 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2021-06-09T23:21:57.807Z] 
[2021-06-09T23:21:57.808Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() PASSED
[2021-06-09T23:21:57.808Z] 
[2021-06-09T23:21:57.808Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() STARTED
[2021-06-09T23:21:59.177Z] + mvn compile
[2021-06-09T23:22:00.361Z] [INFO] Scanning for projects...
[2021-06-09T23:22:00.361Z] [INFO] 
[2021-06-09T23:22:00.361Z] [INFO] -< 
streams.examples:streams.examples >--
[2021-06-09T23:22:00.361Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2021-06-09T23:22:00.361Z] [INFO] [ jar 
]-
[2021-06-09T23:22:00.361Z] [INFO] 
[2021-06-09T23:22:00.361Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2021-06-09T23:22:00.361Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-06-09T23:22:00.361Z] [INFO] Copying 1 resource
[2021-06-09T23:22:00.361Z] [INFO] 
[2021-06-09T23:22:00.361Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2021-06-09T23:22:01.483Z] [INFO] Changes detected - recompiling the module!
[2021-06-09T23:22:01.483Z] [INFO] Compiling 3 source files to 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2021-06-09T23:22:01.982Z] [INFO] 

[2021-06-09T23:22:01.982Z] [INFO] BUILD SUCCESS
[2021-06-09T23:22:01.982Z] [INFO] 

[2021-06-09T23:22:01.982Z] [INFO] Total time:  2.176 s
[2021-06-09T23:22:01.982Z] [INFO] Finished at: 2021-06-09T23:22:01Z
[2021-06-09T23:22:01.982Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2021-06-09T23:22:22.737Z] 
[2021-06-09T23:22:22.737Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() PASSED
[2021-06-09T23:22:22.737Z] 
[2021-06-09T23:22:22.737Z] SaslSslAdminIntegrationTest > testAclOperations() 
STARTED
[2021-06-09T23:22:52.457Z] 
[2021-06-09T23:22:52.457Z] SaslSslAdminIntegrationTest > testAclOperations() 
PASSED
[2021-06-09T23:22:52.457Z] 
[2021-06-09T23:22:52.457Z] SaslSslAdminIntegrationTest > testAclOperations2() 
STARTED
[2021-06-09T23:23:15.556Z] 
[2021-06-09T23:23:15.556Z] SaslSslAdminIntegrationTest > testAclOperations2() 
PASSED
[2021-06-09T23:23:15.556Z] 
[2021-06-09T23:23:15.556Z] SaslSslAdminIntegrationTest > testAclDelete() STARTED
[2021-06-09T23:23:41.259Z] 
[2021-06-09T23:23:41.259Z] SaslSslAdminIntegrationTest > testAclDelete() PASSED
[2021-06-09T23:23:41.259Z] 
[2021-06-09T23:23:41.259Z] TransactionsTest > testBumpTransactionalEpoch() 
STARTED
[2021-06-09T23:24:00.076Z] 
[2021-06-09T23:24:00.076Z] TransactionsTe

[jira] [Created] (KAFKA-12925) StateStore::prefixScan missing from intermediate interfaces

[Michael Viamari (Jira)]

Michael Viamari created KAFKA-12925:
---

 Summary: StateStore::prefixScan missing from intermediate 
interfaces
 Key: KAFKA-12925
 URL: https://issues.apache.org/jira/browse/KAFKA-12925
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.8.0
Reporter: Michael Viamari


[KIP-614|https://cwiki.apache.org/confluence/display/KAFKA/KIP-614%3A+Add+Prefix+Scan+support+for+State+Stores]
 and [KAFKA-10648|https://issues.apache.org/jira/browse/KAFKA-10648] introduced 
support for {{prefixScan}} to StateStores.

It seems that many of the intermediate {{StateStore}} interfaces are missing a 
definition for {{prefixScan}}, and as such is not accessible in all cases.

For example, when accessing the state stores through a the processor context, 
the {{KeyValueStoreReadWriteDecorator}} and associated interfaces do not define 
{{prefixScan}} and it falls back to the default implementation in 
{{KeyValueStore}}, which throws {{UnsupportedOperationException}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Konstantine Karantasis]

Thanks for the KIP Luke.

+1 (binding)

Konstantine


On Wed, Jun 9, 2021 at 6:36 AM Luke Chen  wrote:

> Hi all,
> Bump this thread to see if there's anyone wants to vote or have questions.
>
> Thank you.
> Luke
>
> Sophie Blee-Goldman  於 2021年6月8日 週二 上午4:40
> 寫道：
>
> > +1 (binding)
> >
> > Thanks Luke
> >
> > On Mon, Jun 7, 2021 at 3:26 AM Luke Chen  wrote:
> >
> > > Hi Ismael,
> > > Thanks for your comments. I updated the KIP for the "Compatibility,
> > Upgrade
> > > path" section.
> > > Simply put, no special upgrade path is necessary.
> > >
> > > Thank you.
> > > Luke
> > >
> > >
> > > On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma  wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > The KIP is a bit unclear with regards to whether two rolling bounces
> > are
> > > > required or not. What exactly is being proposed?
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
> > > >
> > > > > Hi all,
> > > > > I'd like to call for a vote on KIP-726: Make the
> "cooperative-sticky,
> > > > > range" as the default assignor.
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> > > > >
> > > > > This KIP is still waiting for the prerequisite stories to get
> > > completed,
> > > > > but it should be soon. Hopefully this can be put into V3.0 since
> > > > > cooperative rebalancing is a major
> > > > > improvement to the life of a consumer group (and its operator).
> > > > >
> > > > > The discussion thread can be found here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-746: Revise KRaft Metadata Records

[Jun Rao]

Hi, Colin,

Thanks for the KIP. +1 from me.

Jun

On Wed, Jun 9, 2021 at 9:36 AM Jason Gustafson 
wrote:

> +1 Thanks Colin!
>
> On Thu, Jun 3, 2021 at 4:30 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > I'd like to call a vote for KIP-746: Revise KRaft Metadata Records. This
> > is a minor KIP which revises the KRaft metadata records slightly for the
> > upcoming 3.0 release.
> >
> > The KIP is at: https://cwiki.apache.org/confluence/x/34zOCg
> >
> > best,
> > Colin
> >
>

Re: [VOTE] KIP-724: Drop support for message formats v0 and v1

[Jason Gustafson]

+1 Thanks Ismael!

On Wed, Jun 9, 2021 at 11:29 AM Ismael Juma  wrote:

> Hi all,
>
> Consensus was reached in the discussion thread and part of what is proposed
> has to happen by 3.0, so starting the vote for KIP-724:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
>
> If there are concerns or objections, feel free to point them out in this
> thread or the discuss thread.
>
> Ismael
>

[jira] [Resolved] (KAFKA-12874) Increase default consumer session timeout to 45s (KIP-735)

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12874.
-
Fix Version/s: 3.0.0
   Resolution: Fixed

> Increase default consumer session timeout to 45s (KIP-735)
> --
>
> Key: KAFKA-12874
> URL: https://issues.apache.org/jira/browse/KAFKA-12874
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.0.0
>
>
> As documented in KIP-735, we will increase the default session timeout to 
> 45s: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] KIP-724: Drop support for message formats v0 and v1

[Ismael Juma]

Hi all,

Consensus was reached in the discussion thread and part of what is proposed
has to happen by 3.0, so starting the vote for KIP-724:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1

If there are concerns or objections, feel free to point them out in this
thread or the discuss thread.

Ismael

Re: [DISCUSS] KIP-746: Revise KRaft Metadata Records

[Colin McCabe]

On Tue, Jun 8, 2021, at 09:53, Jun Rao wrote:
> Hi, Colin,
> 
> Thanks for the update KIP.
> 
> 2. BrokerRegistrationChangeRecord:
> 2.1 Should the listeners field be named EndPoints to be consistent with
> RegisterBrokerRecord?

Good point. I'll change this.

> 2.2 Does the listeners field indicate the addition/removal or the full
> current state of listeners?

Full state. I will clarify this in the KIP.

> 2.3. Does BrokerRegistrationChangeRecord need to also include IncarnationId?

Broker ID and broker epoch are sufficient to identify a broker.

Incarnation ID is used during broker registration to establish the broker 
epoch. The main advantage is it makes the broker registration RPC idempotent. 
But otherwise, we can continue identify a broker by ID and epoch like we always 
have.

> 2.4 Since BrokerRegistrationChangeRecord has the fence field, do we still
> need FenceBrokerRecord and UnfenceBrokerRecord? If we want to keep the
> latter two, should they also include IncarnationId?

The main reason I kept fence / unfence was for compatibility. If we want to 
break compatibility we could remove these.

best,
Colin

> 
> Jun
> 
> On Mon, Jun 7, 2021 at 7:59 AM Colin McCabe  wrote:
> 
> > I added a tagged "Listeners" field to BrokerRegistrationChangeRecord, as
> > we discussed earlier in this thread. This will be set only if the broker
> > listeners have changed.
> >
> > best,
> > Colin
> >
> >
> > On Sat, Jun 5, 2021, at 09:03, Colin McCabe wrote:
> > > Right, we are not supporting upgrade (as discussed earlier). However,
> > > since 2.8 was an official release that had the earlier versions of the
> > > records, I think it's good to bump the version when we make
> > > incompatible changes. For documentation purposes, it's good to be able
> > > to see the differences versus what we had in 2.8. It's also good to be
> > > consistent here. What do you think?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jun 3, 2021, at 19:45, Ismael Juma wrote:
> > > > Quick question: given that upgrades are not supported, do we actually
> > want
> > > > to introduce a new version for the RPCs?
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jun 2, 2021 at 11:02 AM Colin McCabe 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have posted a KIP about updating the KRaft metadata records for
> > 3.0.
> > > > >
> > > > > Check it out at : https://cwiki.apache.org/confluence/x/34zOCg
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Chris Egerton]

Hi Randall,

That's a fair assessment; if a user upgrades their cluster to 3.0 with no
changes to worker or connector configs, it's possible that their cluster
will break if their worker principal(s) lack the necessary ACLs on the
Kafka cluster that hosts the config topic.

If we wanted to take a more conservative approach, we could allow users to
opt in to the use of a transactional producer by their cluster's leader
through some worker configuration property. The rolling upgrade process
from some pre-3.0 cluster to a 3.0+ cluster with exactly-once source
support enabled would become:

1. Upgrade cluster to 3.0 (or a later version, if one is available)
2. Enable the use of a transactional producer by the cluster's leader
3. Enable exactly-once source support

Since steps 1 and 2 could take place within the same rolling upgrade, the
number of rolling upgrades for this new approach would be the same as the
current approach: two. The only downside would be additional configuration
complexity for the worker, and the upgrade process itself would be a little
trickier for users (and potentially more error-prone).

In order to reduce the added configuration complexity as much as possible,
we could expose this intermediate state (workers are on 3.0 and the leader
uses a transactional producer, but exactly-once source support is not
enabled) by renaming the "exactly.once.source.enabled" property to
"exactly.once.source.support", and permitting values of "disabled"
(default), "preparing", and "enabled". The "preparing" and "enabled" values
would provide the same behavior as the current proposal with
"exactly.once.source.enabled" set to "false" and "true", respectively, and
"disabled" would have the same behavior as the current proposal, except
without the use of a transactional producer by the leader.

I'll update the proposal with this new behavior shortly. Thanks for the
review!

Cheers,

Chris

On Wed, Jun 9, 2021 at 1:02 PM Randall Hauch  wrote:

> Chris,
>
> Sorry for the late question/comment. But the "Breaking Changes" concerns
> me. IIUC, when a user upgrades their 1.x or 2.x Connect cluster, then when
> they restart their 3.0 worker(s) the workers will fail due to this producer
> requirement even if they make no changes to their worker configs or
> connector configs. Is this correct?
>
> If so, I'm concerned about this. Even though the additional producer ACLs
> are seemingly minor and easy to change, it is likely that users will not
> read the docs before they upgrade, causing their simple upgrade to fail.
> And even though in 3.0 we could allow ourselves to cause breaking changes
> with a major release, I personally would prefer we not have any such
> breaking changes.
>
> Given that, what would be required for us to eliminate that breaking
> change, or change it from a breaking change to a prerequisite for enabling
> EOS support in their cluster?
>
> Thanks,
>
> Randall
>
> On Wed, Jun 2, 2021 at 8:42 AM Chris Egerton 
> wrote:
>
> > Hi Tom,
> >
> > I do agree that it'd be safer to default to "required", but since at the
> > time of the 3.0 release no existing connectors will have implemented the
> > "SourceConnector::exactlyOnceSupport" method, it'd require all Connect
> > users to downgrade to "requested" anyways in order to enable exactly-once
> > support on their workers. The friction there seems a little excessive; we
> > might consider changing the default from "requested" to "required" later
> on
> > down the line after connector developers have had enough time to put out
> > new connector versions that implement the new API. Thoughts?
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Jun 2, 2021 at 8:49 AM Tom Bentley  wrote:
> >
> > > Hi Chris,
> > >
> > > Just a minor question: I can see why the default for
> exactly.once.support
> > > is requested (you want a good first-run experience, I assume), but
> it's a
> > > little like engineering a safety catch and then not enabling it.
> Wouldn't
> > > it be safer to default to required, so that there's no way someone can
> > > mistakenly not get EoS without explicitly having configured it?
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> > > On Tue, Jun 1, 2021 at 4:48 PM Chris Egerton
>  > >
> > > wrote:
> > >
> > > > Hi Gunnar,
> > > >
> > > > Thanks for taking a look! I've addressed the low-hanging fruit in the
> > > KIP;
> > > > responses to other comments inline here:
> > > >
> > > > > * TransactionContext: What's the use case for the methods
> accepting a
> > > > source record (commitTransaction(SourceRecord
> > > > record), abortTransaction(SourceRecord record))?
> > > >
> > > > This allows developers to decouple transaction boundaries from record
> > > > batches. If a connector has a configuration that dictates how often
> it
> > > > returns from "SourceTask::poll", for example, it may be easier to
> > define
> > > > multiple transactions within a single batch or a single transaction
> > > across
> > > > several batches than to retrofit the connector

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

[Tom Bentley]

Hi Dongjin,

Thanks for the KIP, +1 (binding).

Kind regards,

Tom

On Wed, Jun 9, 2021 at 5:16 PM Ismael Juma  wrote:

> I'm +1 on the proposed change. As I stated in the discuss thread, I don't
> think we should rule out the buffer size config, but we could list that as
> future work vs rejected alternatives.
>
> Ismael
>
> On Sat, Jun 5, 2021 at 2:37 PM Dongjin Lee  wrote:
>
> > Hi all,
> >
> > I'd like to open a voting thread for KIP-390: Support Compression Level
> > (rebooted):
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
> >
> > Best,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

[Randall Hauch]

Chris,

Sorry for the late question/comment. But the "Breaking Changes" concerns
me. IIUC, when a user upgrades their 1.x or 2.x Connect cluster, then when
they restart their 3.0 worker(s) the workers will fail due to this producer
requirement even if they make no changes to their worker configs or
connector configs. Is this correct?

If so, I'm concerned about this. Even though the additional producer ACLs
are seemingly minor and easy to change, it is likely that users will not
read the docs before they upgrade, causing their simple upgrade to fail.
And even though in 3.0 we could allow ourselves to cause breaking changes
with a major release, I personally would prefer we not have any such
breaking changes.

Given that, what would be required for us to eliminate that breaking
change, or change it from a breaking change to a prerequisite for enabling
EOS support in their cluster?

Thanks,

Randall

On Wed, Jun 2, 2021 at 8:42 AM Chris Egerton 
wrote:

> Hi Tom,
>
> I do agree that it'd be safer to default to "required", but since at the
> time of the 3.0 release no existing connectors will have implemented the
> "SourceConnector::exactlyOnceSupport" method, it'd require all Connect
> users to downgrade to "requested" anyways in order to enable exactly-once
> support on their workers. The friction there seems a little excessive; we
> might consider changing the default from "requested" to "required" later on
> down the line after connector developers have had enough time to put out
> new connector versions that implement the new API. Thoughts?
>
> Cheers,
>
> Chris
>
> On Wed, Jun 2, 2021 at 8:49 AM Tom Bentley  wrote:
>
> > Hi Chris,
> >
> > Just a minor question: I can see why the default for exactly.once.support
> > is requested (you want a good first-run experience, I assume), but it's a
> > little like engineering a safety catch and then not enabling it. Wouldn't
> > it be safer to default to required, so that there's no way someone can
> > mistakenly not get EoS without explicitly having configured it?
> >
> > Thanks,
> >
> > Tom
> >
> > On Tue, Jun 1, 2021 at 4:48 PM Chris Egerton  >
> > wrote:
> >
> > > Hi Gunnar,
> > >
> > > Thanks for taking a look! I've addressed the low-hanging fruit in the
> > KIP;
> > > responses to other comments inline here:
> > >
> > > > * TransactionContext: What's the use case for the methods accepting a
> > > source record (commitTransaction(SourceRecord
> > > record), abortTransaction(SourceRecord record))?
> > >
> > > This allows developers to decouple transaction boundaries from record
> > > batches. If a connector has a configuration that dictates how often it
> > > returns from "SourceTask::poll", for example, it may be easier to
> define
> > > multiple transactions within a single batch or a single transaction
> > across
> > > several batches than to retrofit the connector's poll logic to work
> with
> > > transaction boundaries.
> > >
> > > > * SourceTaskContext: Instead of guarding against NSME, is there a way
> > for
> > > a
> > > connector to query the KC version and thus derive its capabilities?
> Going
> > > forward, a generic API for querying capabilities could be nice, so a
> > > connector can query for capabilities of the runtime in a safe and
> > > compatible way.
> > >
> > > This would be a great quality-of-life improvement for connector and
> > > framework developers alike, but I think it may be best left for a
> > separate
> > > KIP. The current approach, clunky though it may be, seems like a
> nuisance
> > > at worst. It's definitely worth addressing but I'm not sure we have the
> > > time to think through all the details thoroughly enough in time for the
> > > upcoming KIP freeze.
> > >
> > > > * SourceConnector: Would it make sense to merge the two methods
> perhaps
> > > and
> > > return one enum of { SUPPORTED, NOT_SUPPORTED,
> SUPPORTED_WITH_BOUNDARIES
> > }?
> > >
> > > Hmm... at first glance I like the idea of merging the two methods a
> lot.
> > > The one thing that gives me pause is that there may be connectors that
> > > would like to define their own transaction boundaries without providing
> > > exactly-once guarantees. We could add UNSUPPORTED_WITH_BOUNDARIES to
> > > accommodate that, but then, it might actually be simpler to keep the
> two
> > > methods separate in case we add some third variable to the mix that
> would
> > > also have to be reflected in the possible ExactlyOnceSupport enum
> values.
> > >
> > > > Or, alternatively return an enum from
> canDefineTransactionBoundaries(),
> > > too; even if it only has two values now, that'd allow for extension in
> > the
> > > future
> > >
> > > This is fine by me; we just have to figure out exactly which enum
> values
> > > would be suitable. It's a little clunky but right now I'm toying with
> > > something like "ConnectorDefinedTransactionBoundaries" with values of
> > > "SUPPORTED" and "NOT_SUPPORTED" and a default of "NOT_SUPPORTED". If we
> > > need more granularity in the future t

Re: [DISCUSS] KIP-724: Drop support for message formats v0 and v1

[Jun Rao]

Hi, Ismael,

Sounds good. The KIP looks good to me.

Thanks,

Jun

On Wed, Jun 9, 2021 at 9:16 AM Ismael Juma  wrote:

> Thanks Jun. After further thought and discussion with Jason, we decided to
> move `log.record.version.force.upgrade` to a "Future Work" section.
>
> There are more details in the KIP, but there are some complexities in
> making it possible to remove read support for v0/v1 formats. Given that, we
> decided to focus on the items that have to happen by 3.0 in this KIP:
> ensuring all new client writes use v2. The rest can happen via a subsequent
> KIP if we can devise a good solution.
>
> I will start the vote thread in parallel, but happy to discuss further.
>
> Ismael
>
> On Tue, Jun 8, 2021 at 9:19 AM Jun Rao  wrote:
>
> > Hi, Ismael,
> >
> > Thanks for the KIP.
> >
> > Regarding log.record.version.force.upgrade:
> > 1. Is that a topic level or a broker level config? The slight benefit of
> > topic level is that it allows people to upgrade incrementally. The
> > potential downside is the additional complexity associated with it.
> > 2. When log.record.version.force.upgrade is set to 2, do we need to scan
> > all batches in all existing segments during broker restart to verify the
> > record format? Since this is rarely needed, the performance impact may
> not
> > be a big concern. However, it will probably be useful to document this in
> > the KIP.
> > 3. Related to 2, we need a way to know when the upgrading to v2 message
> > format has completed. This could be as simple as some logging. It would
> be
> > useful to document this in the KIP.
> > 4. Related to 2, once the upgrade to v2 message format has completed, do
> we
> > expect the user to set log.record.version.force.upgrade to null?
> Otherwise,
> > we will pay the batch scanning overhead on every broker restart. It would
> > be useful to document the upgrade process related to this.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jun 8, 2021 at 8:21 AM Ismael Juma  wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for the thoughtful comments. Responses below.
> > >
> > > 1. Good point. I updated the KIP to up-convert during replication if
> the
> > > IBP is 3.0. I am still not 100% sure if this is truly required, but it
> > does
> > > seem like a good idea to avoid flip-flopping like this. I'll follow up
> > with
> > > you to understand if this is a correctness requirement or something
> that
> > > makes the system easier to reason about and then update the KIP and
> this
> > > thread.
> > >
> > > 2. Agreed. I updated the KIP to mention that we will up-convert to
> single
> > > record batches for the uncompressed case.
> > >
> > > 3. I added a note regarding produce request versions we will no longer
> > > support. I originally did not include this since the broker doesn't
> seem
> > to
> > > do the validation for produce requests while it does for fetch
> requests.
> > > But from a specification perspective, it makes sense to include it
> since
> > > the current behavior is confusing (it may be this way for compatibility
> > > with now very old clients). With regards to list offsets v0, I suggest
> we
> > > do that via a separate KIP. As you said, there might be other similar
> > > opportunities and since the target would be AK 4.0, we have a bit of
> time
> > > to do the analysis.
> > >
> > > Ismael
> > >
> > > On Fri, Jun 4, 2021 at 2:10 PM Jason Gustafson
> >  > > >
> > > wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > I agree it would be awesome to drop support for the old formats! A
> few
> > > > comments below:
> > > >
> > > > 1. Suppose that a partition with 3 replicas begins at v0. One broker
> is
> > > > upgraded to v2 with the new "force upgrade" config as part of a
> rolling
> > > > restart, which leaves two replicas on v0 and one on v2. The KIP
> > documents
> > > > that replicas will take the format from the leader without any
> > > conversion,
> > > > so I'm a little concerned that leader changes in this state could
> > result
> > > in
> > > > the version flipping from v2 to v0. Could that happen? Say, for
> > example,
> > > if
> > > > the third replica was catching up while the rolling restart was in
> > > > progress. Since it is probably an unlikely case in practice, would it
> > > make
> > > > sense to allow up-conversion during replication when "force upgrade"
> > has
> > > > been enabled? Not sure I see any great alternatives to this.
> > > >
> > > > 2. A related question concerns batching when upgrading from the
> > > non-batched
> > > > formats (i.e. v0 and v1 when no compression is in use). I suspect the
> > > only
> > > > reasonable way to do this is to treat each individual record as a
> batch
> > > of
> > > > 1 during the conversion process. Otherwise, I am not sure we can come
> > up
> > > > with a deterministic approach to re-batching which can be applied
> > > > consistently on all replicas (especially taking into account
> > compaction).
> > > > The reason this is important is that replication is aligned

Re: [VOTE] KIP-746: Revise KRaft Metadata Records

[Jason Gustafson]

+1 Thanks Colin!

On Thu, Jun 3, 2021 at 4:30 PM Colin McCabe  wrote:

> Hi all,
>
> I'd like to call a vote for KIP-746: Revise KRaft Metadata Records. This
> is a minor KIP which revises the KRaft metadata records slightly for the
> upcoming 3.0 release.
>
> The KIP is at: https://cwiki.apache.org/confluence/x/34zOCg
>
> best,
> Colin
>

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Gwen Shapira]

+1 (binding)

On Wed, Jun 9, 2021 at 8:28 AM Dongjin Lee  wrote:

> Bumping up the voting thread.
>
> Please note that today is the KIP freeze day. +1 non-binding until now.
>
> Thanks,
> Dongjin
>
> On Sun, Jun 6, 2021 at 4:10 PM Luke Chen  wrote:
>
> > Hi Dongjin,
> > Thanks for the KIP.
> > +1 (non-binding)
> >
> > Thanks.
> > Luke
> >
> > On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > > ReplicaVerificationTool:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > >
> > > Best,
> > > Dongjin
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-724: Drop support for message formats v0 and v1

[Ismael Juma]

Thanks Jun. After further thought and discussion with Jason, we decided to
move `log.record.version.force.upgrade` to a "Future Work" section.

There are more details in the KIP, but there are some complexities in
making it possible to remove read support for v0/v1 formats. Given that, we
decided to focus on the items that have to happen by 3.0 in this KIP:
ensuring all new client writes use v2. The rest can happen via a subsequent
KIP if we can devise a good solution.

I will start the vote thread in parallel, but happy to discuss further.

Ismael

On Tue, Jun 8, 2021 at 9:19 AM Jun Rao  wrote:

> Hi, Ismael,
>
> Thanks for the KIP.
>
> Regarding log.record.version.force.upgrade:
> 1. Is that a topic level or a broker level config? The slight benefit of
> topic level is that it allows people to upgrade incrementally. The
> potential downside is the additional complexity associated with it.
> 2. When log.record.version.force.upgrade is set to 2, do we need to scan
> all batches in all existing segments during broker restart to verify the
> record format? Since this is rarely needed, the performance impact may not
> be a big concern. However, it will probably be useful to document this in
> the KIP.
> 3. Related to 2, we need a way to know when the upgrading to v2 message
> format has completed. This could be as simple as some logging. It would be
> useful to document this in the KIP.
> 4. Related to 2, once the upgrade to v2 message format has completed, do we
> expect the user to set log.record.version.force.upgrade to null? Otherwise,
> we will pay the batch scanning overhead on every broker restart. It would
> be useful to document the upgrade process related to this.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 8, 2021 at 8:21 AM Ismael Juma  wrote:
>
> > Hi Jason,
> >
> > Thanks for the thoughtful comments. Responses below.
> >
> > 1. Good point. I updated the KIP to up-convert during replication if the
> > IBP is 3.0. I am still not 100% sure if this is truly required, but it
> does
> > seem like a good idea to avoid flip-flopping like this. I'll follow up
> with
> > you to understand if this is a correctness requirement or something that
> > makes the system easier to reason about and then update the KIP and this
> > thread.
> >
> > 2. Agreed. I updated the KIP to mention that we will up-convert to single
> > record batches for the uncompressed case.
> >
> > 3. I added a note regarding produce request versions we will no longer
> > support. I originally did not include this since the broker doesn't seem
> to
> > do the validation for produce requests while it does for fetch requests.
> > But from a specification perspective, it makes sense to include it since
> > the current behavior is confusing (it may be this way for compatibility
> > with now very old clients). With regards to list offsets v0, I suggest we
> > do that via a separate KIP. As you said, there might be other similar
> > opportunities and since the target would be AK 4.0, we have a bit of time
> > to do the analysis.
> >
> > Ismael
> >
> > On Fri, Jun 4, 2021 at 2:10 PM Jason Gustafson
>  > >
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > I agree it would be awesome to drop support for the old formats! A few
> > > comments below:
> > >
> > > 1. Suppose that a partition with 3 replicas begins at v0. One broker is
> > > upgraded to v2 with the new "force upgrade" config as part of a rolling
> > > restart, which leaves two replicas on v0 and one on v2. The KIP
> documents
> > > that replicas will take the format from the leader without any
> > conversion,
> > > so I'm a little concerned that leader changes in this state could
> result
> > in
> > > the version flipping from v2 to v0. Could that happen? Say, for
> example,
> > if
> > > the third replica was catching up while the rolling restart was in
> > > progress. Since it is probably an unlikely case in practice, would it
> > make
> > > sense to allow up-conversion during replication when "force upgrade"
> has
> > > been enabled? Not sure I see any great alternatives to this.
> > >
> > > 2. A related question concerns batching when upgrading from the
> > non-batched
> > > formats (i.e. v0 and v1 when no compression is in use). I suspect the
> > only
> > > reasonable way to do this is to treat each individual record as a batch
> > of
> > > 1 during the conversion process. Otherwise, I am not sure we can come
> up
> > > with a deterministic approach to re-batching which can be applied
> > > consistently on all replicas (especially taking into account
> compaction).
> > > The reason this is important is that replication is aligned by batches,
> > so
> > > if the batches are not consistent, the replication won't work. The only
> > > problem with using single-record batches is that the v2 message format
> > has
> > > some additional overhead for the batch itself when compared with the v0
> > and
> > > v1 individual record format. So the up-conversion process would likely
> > > increase storage. I do

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

[Ismael Juma]

I'm +1 on the proposed change. As I stated in the discuss thread, I don't
think we should rule out the buffer size config, but we could list that as
future work vs rejected alternatives.

Ismael

On Sat, Jun 5, 2021 at 2:37 PM Dongjin Lee  wrote:

> Hi all,
>
> I'd like to open a voting thread for KIP-390: Support Compression Level
> (rebooted):
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Bumping up the voting thread.

Please note that today is the KIP freeze day.

Thanks,
Dongjin

On Mon, Jun 7, 2021 at 9:28 PM Dongjin Lee  wrote:

> Bumping up the voting thread.
>
>
> As a reminder: Please note that without the Log4j2 appender, we can't
> complete KIP-653: Upgrade log4j to log4j2
> 
>  (accepted)
> since we can't entirely remove the log4j artifact from the classpath for
> the tools dependency.
>
>
> Best,
>
> Dongjin
>
> On Tue, May 25, 2021 at 10:45 PM Dongjin Lee  wrote:
>
>> Hi Kafka dev,
>>
>> I'd like to kick-off the voting for KIP-719: Add Log4J2 Appender.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>>
>> Best,
>> Dongjin
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>>
>>
>>
>> *github:  github.com/dongjinleekr
>> keybase: https://keybase.io/dongjinleekr
>> linkedin: kr.linkedin.com/in/dongjinleekr
>> speakerdeck: speakerdeck.com/dongjin
>> *
>>
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Dongjin Lee]

Bumping up the voting thread.

Please note that today is the KIP freeze day. +1 non-binding until now.

Thanks,
Dongjin

On Sun, Jun 6, 2021 at 4:10 PM Luke Chen  wrote:

> Hi Dongjin,
> Thanks for the KIP.
> +1 (non-binding)
>
> Thanks.
> Luke
>
> On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee  wrote:
>
> > Hi all,
> >
> > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > ReplicaVerificationTool:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> >
> > Best,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-390: Support Compression Level (rebooted)

[Dongjin Lee]

Bumping up the voting thread.

Please note that today is the KIP freeze day. +1 non-binding until now.

Thanks,
Dongjin

On Sun, Jun 6, 2021 at 11:47 PM Ryanne Dolan  wrote:

> +1 (non-binding), thanks!
>
> Ryanne
>
> On Sat, Jun 5, 2021, 4:37 PM Dongjin Lee  wrote:
>
> > Hi all,
> >
> > I'd like to open a voting thread for KIP-390: Support Compression Level
> > (rebooted):
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
> >
> > Best,
> > Dongjin
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

[Ismael Juma]

Thanks! We could move that to a "Future Work" section instead of "Rejected
Alternatives" if the results are inconclusive.

Ismael

On Wed, Jun 9, 2021 at 8:09 AM Dongjin Lee  wrote:

> > I am OK with doing compression level first, but I don't want to rule out
> the buffer size change without understanding better.
>
> I see. I am now retrying buffer size configuration & benchmark. As soon as
> I get a promising result, I will update the KIP.
>
> Thanks,
> Dongjin
>
> On Wed, Jun 9, 2021 at 12:36 AM Ismael Juma  wrote:
>
> > Btw, I am OK with doing compression level first, but I don't want to rule
> > out the buffer size change without understanding better.
> >
> > Ismael
> >
> > On Tue, Jun 8, 2021 at 8:33 AM Ismael Juma  wrote:
> >
> > > Hi Dongjin,
> > >
> > > I was thinking of a simple test: Snappy with 1 KB block size vs 32 KB
> > > block size. If the compression rate is similar for both, then it seems
> > very
> > > wasteful to use 32 KB. I suspect you will see a significant difference
> > > though.
> > >
> > > Ismael
> > >
> > > On Tue, Jun 8, 2021 at 8:27 AM Dongjin Lee  wrote:
> > >
> > >> Hi Ismael,
> > >>
> > >> I added the linear write benchmark result to the proposal. Like the
> > >> producer benchmark, the least compression level showed the best MB/sec
> > for
> > >> any case. I tested several configurations, but the result was almost
> the
> > >> same.
> > >>
> > >> If you have any proposals for the benchmark, don't hesitate to give
> me a
> > >> suggestion. I am a newbie to run the linear write benchmark.
> > >>
> > >> Best,
> > >> Dongjin
> > >>
> > >> On Sun, Jun 6, 2021 at 8:20 AM Dongjin Lee 
> wrote:
> > >>
> > >> > Hi Ismael,
> > >> >
> > >> > Thanks for the reply.
> > >> >
> > >> > > So you're saying that reducing the buffer size didn't reduce the
> > >> > compression rate for codecs like lz4?
> > >> >
> > >> > Of course, there were some improvements in compressed size when I
> > tried
> > >> > the 'buffer.size' option, but the gain was not significant. I tried
> > >> several
> > >> > datasets, but the result was the same. It made me so skeptical about
> > >> adding
> > >> > this option, which seemed to make the configuration option complex
> > only.
> > >> >
> > >> > In contrast, 'compression.level' showed its effectiveness
> immediately.
> > >> It
> > >> > is why I decided to focus on the 'compression.level' in this rework.
> > >> >
> > >> > As you can see in the update KIP with the benchmark, IMHO, the true
> > >> value
> > >> > of supporting the compression option may not be the compressed size
> or
> > >> > rate, but speed. By tweaking the compression level slightly, it
> showed
> > >> > great produce performance gain.
> > >> >
> > >> > Thanks,
> > >> > Dongjin
> > >> >
> > >> >
> > >> > On Sun, Jun 6, 2021 at 6:48 AM Ismael Juma 
> wrote:
> > >> >
> > >> >> Thanks Dongjin. So you're saying that reducing the buffer size
> didn't
> > >> >> reduce the compression rate for codecs like lz4? If so, that would
> > >> suggest
> > >> >> reducing the default value, but that seems odd.
> > >> >>
> > >> >> Ismael
> > >> >>
> > >> >> On Sat, Jun 5, 2021, 9:25 AM Dongjin Lee 
> wrote:
> > >> >>
> > >> >> > Hello Kafka dev,
> > >> >> >
> > >> >> > I hope to reboot the discussion of KIP-390: Support Compression
> > Level
> > >> >> > <
> > >> >> >
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
> > >> >> > >.
> > >> >> > It proposes to add a new option, 'compression.level', that
> controls
> > >> the
> > >> >> > compression level.
> > >> >> >
> > >> >> > This KIP has been submitted more than one year ago, but had been
> > >> >> neglected
> > >> >> > for a long time. Recently I reworked it from scratch with the
> > >> following
> > >> >> > differences:
> > >> >> >
> > >> >> > 1. Tested how it works with a real-world dataset. As you can see
> in
> > >> the
> > >> >> > updated KIP, *this feature can improve the producer's
> > message/second
> > >> >> rate
> > >> >> > by more than 50%*, such a significant enhancement.
> > >> >> > 2. Dropped 'compression.buffer.size' option that was in the
> initial
> > >> >> work.
> > >> >> > With the repeated benchmarks, I could not find any evidence this
> > >> option
> > >> >> > results in meaningful differences. So I removed it.
> > >> >> >
> > >> >> > All feedback will be highly appreciated.
> > >> >> >
> > >> >> > Best,
> > >> >> > Dongjin
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > *Dongjin Lee*
> > >> >> >
> > >> >> > *A hitchhiker in the mathematical world.*
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > *github:  github.com/dongjinleekr
> > >> >> > keybase:
> > >> >> https://keybase.io/dongjinleekr
> > >> >> > linkedin:
> > >> >> kr.linkedin.com/in/dongjinleekr
> > >> >> > speakerdeck:
> > >> >> > speakerdeck.com/dongjin
> > >> >> > 

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

[Dongjin Lee]

> I am OK with doing compression level first, but I don't want to rule out
the buffer size change without understanding better.

I see. I am now retrying buffer size configuration & benchmark. As soon as
I get a promising result, I will update the KIP.

Thanks,
Dongjin

On Wed, Jun 9, 2021 at 12:36 AM Ismael Juma  wrote:

> Btw, I am OK with doing compression level first, but I don't want to rule
> out the buffer size change without understanding better.
>
> Ismael
>
> On Tue, Jun 8, 2021 at 8:33 AM Ismael Juma  wrote:
>
> > Hi Dongjin,
> >
> > I was thinking of a simple test: Snappy with 1 KB block size vs 32 KB
> > block size. If the compression rate is similar for both, then it seems
> very
> > wasteful to use 32 KB. I suspect you will see a significant difference
> > though.
> >
> > Ismael
> >
> > On Tue, Jun 8, 2021 at 8:27 AM Dongjin Lee  wrote:
> >
> >> Hi Ismael,
> >>
> >> I added the linear write benchmark result to the proposal. Like the
> >> producer benchmark, the least compression level showed the best MB/sec
> for
> >> any case. I tested several configurations, but the result was almost the
> >> same.
> >>
> >> If you have any proposals for the benchmark, don't hesitate to give me a
> >> suggestion. I am a newbie to run the linear write benchmark.
> >>
> >> Best,
> >> Dongjin
> >>
> >> On Sun, Jun 6, 2021 at 8:20 AM Dongjin Lee  wrote:
> >>
> >> > Hi Ismael,
> >> >
> >> > Thanks for the reply.
> >> >
> >> > > So you're saying that reducing the buffer size didn't reduce the
> >> > compression rate for codecs like lz4?
> >> >
> >> > Of course, there were some improvements in compressed size when I
> tried
> >> > the 'buffer.size' option, but the gain was not significant. I tried
> >> several
> >> > datasets, but the result was the same. It made me so skeptical about
> >> adding
> >> > this option, which seemed to make the configuration option complex
> only.
> >> >
> >> > In contrast, 'compression.level' showed its effectiveness immediately.
> >> It
> >> > is why I decided to focus on the 'compression.level' in this rework.
> >> >
> >> > As you can see in the update KIP with the benchmark, IMHO, the true
> >> value
> >> > of supporting the compression option may not be the compressed size or
> >> > rate, but speed. By tweaking the compression level slightly, it showed
> >> > great produce performance gain.
> >> >
> >> > Thanks,
> >> > Dongjin
> >> >
> >> >
> >> > On Sun, Jun 6, 2021 at 6:48 AM Ismael Juma  wrote:
> >> >
> >> >> Thanks Dongjin. So you're saying that reducing the buffer size didn't
> >> >> reduce the compression rate for codecs like lz4? If so, that would
> >> suggest
> >> >> reducing the default value, but that seems odd.
> >> >>
> >> >> Ismael
> >> >>
> >> >> On Sat, Jun 5, 2021, 9:25 AM Dongjin Lee  wrote:
> >> >>
> >> >> > Hello Kafka dev,
> >> >> >
> >> >> > I hope to reboot the discussion of KIP-390: Support Compression
> Level
> >> >> > <
> >> >> >
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level
> >> >> > >.
> >> >> > It proposes to add a new option, 'compression.level', that controls
> >> the
> >> >> > compression level.
> >> >> >
> >> >> > This KIP has been submitted more than one year ago, but had been
> >> >> neglected
> >> >> > for a long time. Recently I reworked it from scratch with the
> >> following
> >> >> > differences:
> >> >> >
> >> >> > 1. Tested how it works with a real-world dataset. As you can see in
> >> the
> >> >> > updated KIP, *this feature can improve the producer's
> message/second
> >> >> rate
> >> >> > by more than 50%*, such a significant enhancement.
> >> >> > 2. Dropped 'compression.buffer.size' option that was in the initial
> >> >> work.
> >> >> > With the repeated benchmarks, I could not find any evidence this
> >> option
> >> >> > results in meaningful differences. So I removed it.
> >> >> >
> >> >> > All feedback will be highly appreciated.
> >> >> >
> >> >> > Best,
> >> >> > Dongjin
> >> >> >
> >> >> >
> >> >> > --
> >> >> > *Dongjin Lee*
> >> >> >
> >> >> > *A hitchhiker in the mathematical world.*
> >> >> >
> >> >> >
> >> >> >
> >> >> > *github:  github.com/dongjinleekr
> >> >> > keybase:
> >> >> https://keybase.io/dongjinleekr
> >> >> > linkedin:
> >> >> kr.linkedin.com/in/dongjinleekr
> >> >> > speakerdeck:
> >> >> > speakerdeck.com/dongjin
> >> >> > *
> >> >> >
> >> >>
> >> >
> >> >
> >> > --
> >> > *Dongjin Lee*
> >> >
> >> > *A hitchhiker in the mathematical world.*
> >> >
> >> >
> >> >
> >> > *github:  github.com/dongjinleekr
> >> > keybase:
> >> https://keybase.io/dongjinleekr
> >> > linkedin:
> >> kr.linkedin.com/in/dongjinleekr
> >> > speakerdeck:
> >> speakerdeck.com/dongjin
> >> > 

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Luke Chen]

Hi all,
Bump this thread to see if there's anyone wants to vote or have questions.

Thank you.
Luke

Sophie Blee-Goldman  於 2021年6月8日 週二 上午4:40 寫道：

> +1 (binding)
>
> Thanks Luke
>
> On Mon, Jun 7, 2021 at 3:26 AM Luke Chen  wrote:
>
> > Hi Ismael,
> > Thanks for your comments. I updated the KIP for the "Compatibility,
> Upgrade
> > path" section.
> > Simply put, no special upgrade path is necessary.
> >
> > Thank you.
> > Luke
> >
> >
> > On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma  wrote:
> >
> > > Hi Luke,
> > >
> > > The KIP is a bit unclear with regards to whether two rolling bounces
> are
> > > required or not. What exactly is being proposed?
> > >
> > > Ismael
> > >
> > > On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > > I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> > > > range" as the default assignor.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> > > >
> > > > This KIP is still waiting for the prerequisite stories to get
> > completed,
> > > > but it should be soon. Hopefully this can be put into V3.0 since
> > > > cooperative rebalancing is a major
> > > > improvement to the life of a consumer group (and its operator).
> > > >
> > > > The discussion thread can be found here:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> >
>

Re: [VOTE] KIP-738: Removal of Connect's internal converter properties

[Chris Egerton]

Hi all,

The KIP passes with three binding +1s from Randall, Tom, and Konstantine,
and one non-binding +1 from Ryanne. I'll have a PR out shortly. Thanks to
all who participated!

Cheers,

Chris


On Fri, Jun 4, 2021 at 9:28 PM Konstantine Karantasis
 wrote:

> +1 (binding)  but also with the note that we don't tend to require a KIP in
> order to remove configurations that have been deprecated in previous KIPs.
>
> A jira ticket with the right release as a target should suffice. In future
> KIPs adding a note that says that a feature is deprecated and may be
> removed in future major releases can help make this clear in the future.
>
> Konstantine.
>
>
> On Thu, Jun 3, 2021 at 10:47 AM Tom Bentley  wrote:
>
> > +1 (binding).
> >
> > Thanks Chris!
> >
> > On Wed, Jun 2, 2021 at 11:00 PM Randall Hauch  wrote:
> >
> > > Thanks for putting this together, Chris.
> > >
> > > Technically, we don't need a new KIP to explicitly remove an API,
> config,
> > > etc. that was previously deprecated under an earlier approved KIP. But
> > > KIP-174 could have been a bit more explicit (note taken for future
> KIPs)
> > > that deprecation means "deprecation and future removal", and KIP-738
> does
> > > add a nice migration path for anyone stuck using a non-default internal
> > > converter.
> > >
> > > So I'm +1 (binding) to just wrap this up and remove these configs.
> > >
> > > I'd also point out there is a wiki page [1] and dev discussion thread
> [2]
> > > highlighting all Connect-related KIPs, including all deprecations. I'll
> > > update that page to include KIP-736, but it'd be great to have more
> > > discussion there.
> > >
> > > Randall
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047362
> > > [2]
> > >
> > >
> >
> https://lists.apache.org/thread.html/r01e9fa52998a337e435e5d0effca02e74b0552bdec271c1eeca39cd2%40%3Cdev.kafka.apache.org%3E
> > >
> > > On Tue, May 18, 2021 at 11:26 AM Ryanne Dolan 
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks!
> > > >
> > > > Ryanne
> > > >
> > > > On Tue, May 18, 2021, 6:38 AM Chris Egerton
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-738:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-738%3A+Removal+of+Connect%27s+internal+converter+properties
> > > > >
> > > > > The discussion thread (which was originally titled with "KIP-736")
> > can
> > > be
> > > > > found here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202105.mbox/%3CCAMdOrUV0bqqs-ry7Q6FkNNn21ZhODTrg2d61zE5WZJw1MpQvSQ%40mail.gmail.com%3E
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-618: Exactly-once support for source connectors

[Chris Egerton]

Hi all,

Friendly reminder that the KIP freeze is today; please cast your votes if
you'd like to see this feature introduced in time for 3.0.

Cheers,

Chris

On Mon, Jun 7, 2021 at 1:49 AM Dongjin Lee  wrote:

> +1 (non-binding).
>
> Thanks,
> Dongjin
>
> On Fri, Jun 4, 2021 at 2:35 PM Ryanne Dolan  wrote:
>
> > +1 (non-binding)
> >
> > On Thu, Jun 3, 2021, 10:23 AM Chris Egerton  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-618, which adds support for
> > exactly-once
> > > delivery guarantees for source connectors in the Kafka Connect
> framework.
> > >
> > > I suspect there might be a little more discussion to be had but with
> the
> > > KIP freeze deadline approaching, I wanted to give anyone following
> along
> > > the chance to cast a +1 as soon as they feel that we've gotten to a
> > > reasonable state.
> > >
> > > The KIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors
> > >
> > > The discussion thread:
> > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202005.mbox/%3CCAMdOrUX-VK5OSB3OOdJNXW_YKEJH9FjQZ4eyzr2GXUhSeDnF3Q%40mail.gmail.com%3E
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Tom Bentley]

Hi Ismael,

Thanks for the KIP, +1 (binding).

Cheers,

Tom

On Wed, Jun 9, 2021 at 10:04 AM Mickael Maison 
wrote:

> +1 (binding)
> Thanks
>
> On Wed, Jun 9, 2021 at 1:25 AM Israel Ekpo  wrote:
> >
> > This is great!
> >
> > Thanks for the KIP
> >
> > +1 (non-binding)
> >
> >
> > On Tue, Jun 8, 2021 at 3:13 AM Bruno Cadonna  wrote:
> >
> > > Hi Ismael,
> > >
> > > Thank you for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > Bruno
> > >
> > > On 08.06.21 10:11, Matthew de Detrich wrote:
> > > > +1 (non binding)
> > > >
> > > > On Tue, Jun 8, 2021 at 4:48 AM Luke Chen  wrote:
> > > >
> > > >> Hi Ismael,
> > > >> Thanks for the KIP.
> > > >> +1 (non-binding)
> > > >>
> > > >> Thanks.
> > > >> Luke
> > > >>
> > > >> On Tue, Jun 8, 2021 at 7:18 AM Konstantine Karantasis
> > > >>  wrote:
> > > >>
> > > >>> Thanks for the KIP Ismael. I agree that giving an early enough
> > > >> deprecation
> > > >>> notice for an upcoming change like this one and doing so in a major
> > > >> release
> > > >>> is best in this case.
> > > >>>
> > > >>> +1 (binding)
> > > >>>
> > > >>> Konstantine
> > > >>>
> > > >>> On Mon, Jun 7, 2021 at 7:30 AM Ryanne Dolan  >
> > > >> wrote:
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  Ryanne
> > > 
> > >  On Mon, Jun 7, 2021, 9:26 AM Satish Duggana <
> satish.dugg...@gmail.com
> > > >
> > >  wrote:
> > > 
> > > > +1 (non-binding)
> > > >
> > > > On Mon, 7 Jun 2021 at 19:30, Dongjin Lee 
> wrote:
> > > >
> > > >> +1 (non-binding)
> > > >>
> > > >> As a note: I think the exact removal schedule may be changed.
> > > >>
> > > >> Best,
> > > >> Dongjin
> > > >>
> > > >> On Mon, Jun 7, 2021, 10:56 PM Josep Prat
> > > >>  > > 
> > > >> wrote:
> > > >>
> > > >>> Thanks Ismael, it's a +1 (non-binding) from
> > > <
> https://www.google.com/maps/search/Ismael,+it's+a+%2B1+(non-binding)+from?entry=gmail&source=g
> >
> > > my side,
> > > >>>
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> ———
> > > >>> Josep Prat
> > > >>>
> > > >>> Aiven Deutschland GmbH
> > > >>>
> > > >>> Immanuelkirchstraße 26, 10405 Berlin
> > > >>>
> > > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>>
> > > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>>
> > > >>> m: +491715557497
> > > >>>
> > > >>> w: aiven.io
> > > >>>
> > > >>> e: josep.p...@aiven.io
> > > >>>
> > > >>> On Mon, Jun 7, 2021, 15:51 Ismael Juma 
> > > >> wrote:
> > > >>>
> > >  Since the goal is to provide ample warning regarding the
> future
> > > > removal
> > > >>> of
> > >  Java 8 support and the change we're proposing for 3.0 is
> > > >> simply a
> > >  documentation update, I am starting the vote:
> > > 
> > > 
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > > 
> > >  If there are concerns or objections, feel free to point them
> > > >> out
> > > >>> in
> > > >> this
> > >  thread or the discuss thread.
> > > 
> > >  Ismael
> > > 
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > > >
> > >
>
>

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Tom Bentley]

Hi Ismael,

Thanks for the KIP, +1 (binding).

Cheers,

Tom



On Wed, Jun 9, 2021 at 10:29 AM David Jacot  wrote:

> +1 (binding)
>
> Thanks for the KIP, Ismael!
>
> Le mer. 9 juin 2021 à 11:06, Mickael Maison  a
> écrit :
>
> > +1 (binding)
> > Thanks for the KIP
> >
> > On Tue, Jun 8, 2021 at 9:15 AM Bruno Cadonna  wrote:
> > >
> > > Hi Ismael,
> > >
> > > Thank you for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > Bruno
> > >
> > > On 08.06.21 10:11, Matthew de Detrich wrote:
> > > > +1 (non binding)
> > > >
> > > > On Tue, Jun 8, 2021 at 4:34 AM Luke Chen  wrote:
> > > >
> > > >> Hi Ismael,
> > > >> Thanks for the KIP!
> > > >> +1 (non-binding)
> > > >>
> > > >> Thanks.
> > > >> Luke
> > > >>
> > > >> On Tue, Jun 8, 2021 at 8:16 AM Israel Ekpo 
> > wrote:
> > > >>
> > > >>> I support this as well. The motivation and proposed changes makes
> > sense
> > > >> to
> > > >>> me
> > > >>>
> > > >>> +1 (non-binding)
> > > >>>
> > > >>> On Mon, Jun 7, 2021 at 6:28 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > >>> wrote:
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
> > >   wrote:
> > > 
> > > > +1 (binding)
> > > >
> > > > Konstantine
> > > >
> > > > On Mon, Jun 7, 2021 at 6:51 AM Josep Prat
> > > >>  > > 
> > > > wrote:
> > > >
> > > >> Hi Ismael,
> > > >>
> > > >> Good KIP, it's a +1 (non binding) from my side.
> > > >>
> > > >> Best,
> > > >>
> > > >> ———
> > > >> Josep Prat
> > > >>
> > > >> Aiven Deutschland GmbH
> > > >>
> > > >> Immanuelkirchstraße 26, 10405 Berlin
> > > >>
> > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>
> > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>
> > > >> m: +491715557497
> > > >>
> > > >> w: aiven.io
> > > >>
> > > >> e: josep.p...@aiven.io
> > > >>
> > > >> On Mon, Jun 7, 2021, 15:49 Ismael Juma 
> wrote:
> > > >>
> > > >>> Since the goal is to provide ample warning regarding the future
> > >  removal
> > > >> of
> > > >>> Scala 2.12 and the change we're proposing for 3.0 is simply a
> > > >> documentation
> > > >>> update, I am starting the vote:
> > > >>>
> > > >>>
> > > >>
> > > > https://cwiki.apache.org/confluence/pages/view
> > >  page.action?pageId=181308218
> > >  <
> > > >>>
> > > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> > > 
> > > >>>
> > > >>> If there are concerns or objections, feel free to point them
> out
> > > >> in
> > > > this
> > > >>> thread or the discuss thread.
> > > >>>
> > > >>> Ismael
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > > >
> >
>

[jira] [Created] (KAFKA-12924) Replace EasyMock and PowerMock with Mockito in streams module(metrics)

[YI-CHEN WANG (Jira)]

YI-CHEN WANG created KAFKA-12924:


 Summary: Replace EasyMock and PowerMock with Mockito in streams 
module(metrics)
 Key: KAFKA-12924
 URL: https://issues.apache.org/jira/browse/KAFKA-12924
 Project: Kafka
  Issue Type: Sub-task
Reporter: YI-CHEN WANG
Assignee: YI-CHEN WANG






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[David Jacot]

+1 (binding)

Thanks for the KIP, Ismael!

Le mer. 9 juin 2021 à 11:06, Mickael Maison  a
écrit :

> +1 (binding)
> Thanks for the KIP
>
> On Tue, Jun 8, 2021 at 9:15 AM Bruno Cadonna  wrote:
> >
> > Hi Ismael,
> >
> > Thank you for the KIP!
> >
> > +1 (binding)
> >
> > Best,
> > Bruno
> >
> > On 08.06.21 10:11, Matthew de Detrich wrote:
> > > +1 (non binding)
> > >
> > > On Tue, Jun 8, 2021 at 4:34 AM Luke Chen  wrote:
> > >
> > >> Hi Ismael,
> > >> Thanks for the KIP!
> > >> +1 (non-binding)
> > >>
> > >> Thanks.
> > >> Luke
> > >>
> > >> On Tue, Jun 8, 2021 at 8:16 AM Israel Ekpo 
> wrote:
> > >>
> > >>> I support this as well. The motivation and proposed changes makes
> sense
> > >> to
> > >>> me
> > >>>
> > >>> +1 (non-binding)
> > >>>
> > >>> On Mon, Jun 7, 2021 at 6:28 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > >>> wrote:
> > >>>
> >  +1 (non-binding)
> > 
> >  On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
> >   wrote:
> > 
> > > +1 (binding)
> > >
> > > Konstantine
> > >
> > > On Mon, Jun 7, 2021 at 6:51 AM Josep Prat
> > >>  > 
> > > wrote:
> > >
> > >> Hi Ismael,
> > >>
> > >> Good KIP, it's a +1 (non binding) from my side.
> > >>
> > >> Best,
> > >>
> > >> ———
> > >> Josep Prat
> > >>
> > >> Aiven Deutschland GmbH
> > >>
> > >> Immanuelkirchstraße 26, 10405 Berlin
> > >>
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>
> > >> m: +491715557497
> > >>
> > >> w: aiven.io
> > >>
> > >> e: josep.p...@aiven.io
> > >>
> > >> On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
> > >>
> > >>> Since the goal is to provide ample warning regarding the future
> >  removal
> > >> of
> > >>> Scala 2.12 and the change we're proposing for 3.0 is simply a
> > >> documentation
> > >>> update, I am starting the vote:
> > >>>
> > >>>
> > >>
> > > https://cwiki.apache.org/confluence/pages/view
> >  page.action?pageId=181308218
> >  <
> > >>>
> > >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> > 
> > >>>
> > >>> If there are concerns or objections, feel free to point them out
> > >> in
> > > this
> > >>> thread or the discuss thread.
> > >>>
> > >>> Ismael
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> > >
> > >
>

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Mickael Maison]

+1 (binding)
Thanks for the KIP

On Tue, Jun 8, 2021 at 9:15 AM Bruno Cadonna  wrote:
>
> Hi Ismael,
>
> Thank you for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 08.06.21 10:11, Matthew de Detrich wrote:
> > +1 (non binding)
> >
> > On Tue, Jun 8, 2021 at 4:34 AM Luke Chen  wrote:
> >
> >> Hi Ismael,
> >> Thanks for the KIP!
> >> +1 (non-binding)
> >>
> >> Thanks.
> >> Luke
> >>
> >> On Tue, Jun 8, 2021 at 8:16 AM Israel Ekpo  wrote:
> >>
> >>> I support this as well. The motivation and proposed changes makes sense
> >> to
> >>> me
> >>>
> >>> +1 (non-binding)
> >>>
> >>> On Mon, Jun 7, 2021 at 6:28 PM Satish Duggana 
> >>> wrote:
> >>>
>  +1 (non-binding)
> 
>  On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
>   wrote:
> 
> > +1 (binding)
> >
> > Konstantine
> >
> > On Mon, Jun 7, 2021 at 6:51 AM Josep Prat
> >>  
> > wrote:
> >
> >> Hi Ismael,
> >>
> >> Good KIP, it's a +1 (non binding) from my side.
> >>
> >> Best,
> >>
> >> ———
> >> Josep Prat
> >>
> >> Aiven Deutschland GmbH
> >>
> >> Immanuelkirchstraße 26, 10405 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> m: +491715557497
> >>
> >> w: aiven.io
> >>
> >> e: josep.p...@aiven.io
> >>
> >> On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
> >>
> >>> Since the goal is to provide ample warning regarding the future
>  removal
> >> of
> >>> Scala 2.12 and the change we're proposing for 3.0 is simply a
> >> documentation
> >>> update, I am starting the vote:
> >>>
> >>>
> >>
> > https://cwiki.apache.org/confluence/pages/view
>  page.action?pageId=181308218
>  <
> >>>
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> 
> >>>
> >>> If there are concerns or objections, feel free to point them out
> >> in
> > this
> >>> thread or the discuss thread.
> >>>
> >>> Ismael
> >>>
> >>
> >
> 
> >>>
> >>
> >
> >

[jira] [Created] (KAFKA-12923) Log.splitOverflowedSegment logic can skip producer state snapshot deletion

[Kowshik Prakasam (Jira)]

Kowshik Prakasam created KAFKA-12923:


 Summary: Log.splitOverflowedSegment logic can skip producer state 
snapshot deletion
 Key: KAFKA-12923
 URL: https://issues.apache.org/jira/browse/KAFKA-12923
 Project: Kafka
  Issue Type: Improvement
Reporter: Kowshik Prakasam


In Log.splitOverflowedSegment logic, we probably don't have to delete producer 
state snapshot 
[here|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/Log.scala#L2341]
 since the old segment is replaced with a new segment with the same base offset.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Mickael Maison]

+1 (binding)
Thanks

On Wed, Jun 9, 2021 at 1:25 AM Israel Ekpo  wrote:
>
> This is great!
>
> Thanks for the KIP
>
> +1 (non-binding)
>
>
> On Tue, Jun 8, 2021 at 3:13 AM Bruno Cadonna  wrote:
>
> > Hi Ismael,
> >
> > Thank you for the KIP!
> >
> > +1 (binding)
> >
> > Best,
> > Bruno
> >
> > On 08.06.21 10:11, Matthew de Detrich wrote:
> > > +1 (non binding)
> > >
> > > On Tue, Jun 8, 2021 at 4:48 AM Luke Chen  wrote:
> > >
> > >> Hi Ismael,
> > >> Thanks for the KIP.
> > >> +1 (non-binding)
> > >>
> > >> Thanks.
> > >> Luke
> > >>
> > >> On Tue, Jun 8, 2021 at 7:18 AM Konstantine Karantasis
> > >>  wrote:
> > >>
> > >>> Thanks for the KIP Ismael. I agree that giving an early enough
> > >> deprecation
> > >>> notice for an upcoming change like this one and doing so in a major
> > >> release
> > >>> is best in this case.
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> Konstantine
> > >>>
> > >>> On Mon, Jun 7, 2021 at 7:30 AM Ryanne Dolan 
> > >> wrote:
> > >>>
> >  +1 (non-binding)
> > 
> >  Ryanne
> > 
> >  On Mon, Jun 7, 2021, 9:26 AM Satish Duggana  > >
> >  wrote:
> > 
> > > +1 (non-binding)
> > >
> > > On Mon, 7 Jun 2021 at 19:30, Dongjin Lee  wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> As a note: I think the exact removal schedule may be changed.
> > >>
> > >> Best,
> > >> Dongjin
> > >>
> > >> On Mon, Jun 7, 2021, 10:56 PM Josep Prat
> > >>  > 
> > >> wrote:
> > >>
> > >>> Thanks Ismael, it's a +1 (non-binding) from
> > 
> > my side,
> > >>>
> > >>>
> > >>> Best,
> > >>>
> > >>> ———
> > >>> Josep Prat
> > >>>
> > >>> Aiven Deutschland GmbH
> > >>>
> > >>> Immanuelkirchstraße 26, 10405 Berlin
> > >>>
> > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>
> > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>>
> > >>> m: +491715557497
> > >>>
> > >>> w: aiven.io
> > >>>
> > >>> e: josep.p...@aiven.io
> > >>>
> > >>> On Mon, Jun 7, 2021, 15:51 Ismael Juma 
> > >> wrote:
> > >>>
> >  Since the goal is to provide ample warning regarding the future
> > > removal
> > >>> of
> >  Java 8 support and the change we're proposing for 3.0 is
> > >> simply a
> >  documentation update, I am starting the vote:
> > 
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > 
> >  If there are concerns or objections, feel free to point them
> > >> out
> > >>> in
> > >> this
> >  thread or the discuss thread.
> > 
> >  Ismael
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> > >
> > >
> >

[jira] [Created] (KAFKA-12922) MirrorCheckpointTask should close topic filter

[Viktor Somogyi-Vass (Jira)]

Viktor Somogyi-Vass created KAFKA-12922:
---

 Summary: MirrorCheckpointTask should close topic filter
 Key: KAFKA-12922
 URL: https://issues.apache.org/jira/browse/KAFKA-12922
 Project: Kafka
  Issue Type: Improvement
  Components: mirrormaker
Affects Versions: 2.8.0
Reporter: Viktor Somogyi-Vass
Assignee: Viktor Somogyi-Vass


When a lot of connectors are restarted it turned out that underlying 
ConfigConsumers are not closed property and from the logs we can see that the 
old ones are still running.

Turns out that MirrorCheckpointTask utilizes a TopicFilter, but never closes 
it, leaking resources.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)