[jira] [Created] (KAFKA-12912) fdgfdgfg

[mushfiqur rahoman (Jira)]

mushfiqur rahoman created KAFKA-12912:
-

 Summary: fdgfdgfg
 Key: KAFKA-12912
 URL: https://issues.apache.org/jira/browse/KAFKA-12912
 Project: Kafka
  Issue Type: Bug
Reporter: mushfiqur rahoman


fdgdfgdfgdfgdfgdfgfdgdfgdfgdfgdfgdgfgdfgdf



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Josep Prat]

Hi all,
Thanks for participating. The feedback was really useful.

I'm closing this vote with three binding +1 and zero -1 votes.

Binding +1:
— Bruno
— Guozhang
— Sophie
Non-Binding +1:
— Myself

I hope I closed the vote the right way. Let me know if something I did was
not correct.

I'll start with the implementation.

Best,
———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Mon, Jun 7, 2021, 22:38 Sophie Blee-Goldman 
wrote:

> Thanks for the KIP! +1 (binding)
>
> -Sophie
>
> On Mon, Jun 7, 2021 at 11:19 AM Guozhang Wang  wrote:
>
> > Thanks, that update LGTM. +1!
> >
> > Guozhang
> >
> > On Mon, Jun 7, 2021 at 9:45 AM Josep Prat 
> > wrote:
> >
> > > Hi Guozhang,
> > > Let me know if the updated KIP meets your requests. And thanks again
> for
> > > your feedback!
> > >
> > > Thanks,
> > >
> > > On Sat, Jun 5, 2021 at 4:38 PM Josep Prat  wrote:
> > >
> > > > Done, KIP is now updated to reflect this.
> > > >
> > > > On Sat, Jun 5, 2021 at 4:29 PM Josep Prat 
> wrote:
> > > >
> > > >> Hi Guozhang,
> > > >>
> > > >> Thanks for your review, let's exclude the "localThreadsMetadata
> > > returning
> > > >> StreamsMetadata" change then. This way, as well, this KIP is
> > completely
> > > >> consistent on making those Metadata classes interfaces with internal
> > > >> implementations.
> > > >> I'll update the KIP document right away.
> > > >>
> > > >> Thanks for replying on Saturday :)
> > > >>
> > > >> On Sat, Jun 5, 2021 at 4:22 PM Guozhang Wang 
> > > wrote:
> > > >>
> > > >>> Hi Josep,
> > > >>>
> > > >>> I think the most significant change would be,
> "localThreadsMetadata"
> > > >>> returns a StreamsMetadata instead of ThreadMetadata, and
> > > StreamsMetadata
> > > >>> would expose a new API to return a set of ThreadMetadata.
> > > >>>
> > > >>> All others (including the repackaging and splitting of interface /
> > > impl)
> > > >>> are minor indeed.
> > > >>>
> > > >>> After a second thought, I feel it is not fair to squeeze in this
> > > >>> significant change into your KIP, without the community having a
> > > >>> separating
> > > >>> discussion about it, so I think we can table it for now and only
> > align
> > > on
> > > >>> the other minor things: 1) have a o.a.k.streams.StreamsMetadata
> > > interface
> > > >>> (along with an internal implementation class), 2) deprecate the
> > > >>> o.a.k.streams.state.StreamsMetadata class and also the
> corresponding
> > > >>> caller
> > > >>> of Streams that returns this class.
> > > >>>
> > > >>> Guozhang
> > > >>>
> > > >>> On Fri, Jun 4, 2021 at 4:13 PM Josep Prat
> >  > > >
> > > >>> wrote:
> > > >>>
> > > >>> > Hi Guozhang,
> > > >>> > So if I understand correctly, it's only a couple of small changes
> > > that
> > > >>> need
> > > >>> > to be made to this KIP to be aligned with KAFKA-12370, right?
> > > >>> >
> > > >>> > I'm guessing that StreamsMetadata would not only moved to
> > > >>> o.a.k.streams but
> > > >>> > also be split with Interface + internal implementation, am I
> right?
> > > >>> >
> > > >>> >
> > > >>> > If that's the case, I could, most probably, update the KIP by
> > > Saturday
> > > >>> > afternoon CEST.
> > > >>> >
> > > >>> > Let me know if I understood you correctly.
> > > >>> >
> > > >>> > Thanks for the comments!
> > > >>> >
> > > >>> > ———
> > > >>> > Josep Prat
> > > >>> >
> > > >>> > Aiven Deutschland GmbH
> > > >>> >
> > > >>> > Immanuelkirchstraße 26, 10405 Berlin
> > > >>> >
> > > >>> > Amtsgericht Charlottenburg, HRB 209739 B
> > > >>> >
> > > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>> >
> > > >>> > m: +491715557497
> > > >>> >
> > > >>> > w: aiven.io
> > > >>> >
> > > >>> > e: josep.p...@aiven.io
> > > >>> >
> > > >>> > On Sat, Jun 5, 2021, 00:11 Guozhang Wang 
> > wrote:
> > > >>> >
> > > >>> > > Hello Josep,
> > > >>> > >
> > > >>> > > Thanks for the proposal! The write-up looks good to me in
> > general.
> > > >>> I'm
> > > >>> > just
> > > >>> > > wondering if you feel comfortable to align this with another
> > > JIRA/KIP
> > > >>> > > further down the road:
> > > >>> > >
> > > >>> > > https://issues.apache.org/jira/browse/KAFKA-12370
> > > >>> > >
> > > >>> > > Which tries to clean up the metadata hierarchy and the callers
> as
> > > >>> > > StreamsMetadata -> ThreadMetadata -> TaskMetadata, and most
> > Streams
> > > >>> APIs
> > > >>> > > return the top-level StreamsMetadata.
> > > >>> > >
> > > >>> > > It just have slight differences with the current proposal: 1)
> > > >>> instead of
> > > >>> > > returning a ThreadMetadata, "localThreadsMetadata" returns
> > > >>> > > a StreamsMetadata, and 2) the `StreamsMetadata` would also be
> > moved
> > > >>> from
> > > >>> > > o.a.k.streams.state to o.a.k.streams.
> > > >>> > >
> > > >>> > > What do you think about this? It's totally okay if you are not
> > > >>> > comforta

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Ryan Leslie (BLOOMBERG/ 919 3RD A)]

Hey guys,

Should open bugs concerning cooperative-sticky also be considered blockers to 
making it the default? For example, KAFKA-12896 is perhaps still being 
investigated:

https://issues.apache.org/jira/browse/KAFKA-12896

Thanks,

Ryan

From: dev@kafka.apache.org At: 06/07/21 19:37:45 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the 
default assignor

Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully
enable cooperative rebalancing
by default in 3.0 if we have KAFKA-12477 done in time, and if we don't then
there's no harm as it's
not going to change the behavior.

On Wed, Jun 2, 2021 at 7:28 PM Luke Chen  wrote:

> Hi Sophie,
> Thanks for the reminder. Yes, I was thinking this KIP doesn't have to be
> put into a major release since it will be fully backward compatible, so no
> need to push it. Currently, if we want to work on this KIP, we need
> KAFKA-12477 and KAFKA-12487. But you're right, we can at least try our best
> to see if we can make it into V3.0 since cooperative rebalancing is a major
> improvement. I'll kick off a vote later.
>
> Thank you.
> Luke
>
> On Thu, Jun 3, 2021 at 7:08 AM Sophie Blee-Goldman
>  wrote:
>
> > Hey Luke,
> >
> > It's been a while since the last update on this, which is mostly my fault
> > for picking up
> > other things in the meantime. I'm planning to get back to work
> > on KAFKA-12477 next
> > week but there are minimal changes to the current implementation given
> the
> > proposal
> > I put forth earlier in this KIP discussion, so I think we're good to go.
> >
> > Although this KIP no longer requires a major release since it should be
> > fully compatible, I
> > still hope we can get it in to 3.0 since cooperative rebalancing is a
> major
> > improvement to
> > the life of a consumer group (and its operator). Can we make sure the KIP
> > reflects the latest
> > and then kick off a vote by next Monday at the latest so we can make KIP
> > freeze?
> >
> > Thanks!
> > Sophie
> >
> > On Fri, Apr 16, 2021 at 2:33 PM Guozhang Wang 
> wrote:
> >
> > > 1) From user's perspective, it is always possible that a commit within
> > > onPartitionsRevoked throw in practice (e.g. if the member missed the
> > > previous rebalance where its assigned partitions are already
> re-assigned)
> > > -- and the onPartitionsLost was introduced for that exact reason, i.e.
> it
> > > is primarily for optimizations, but not for correctness guarantees --
> on
> > > the other hand, it would be surprising to users to see the commit
> returns
> > > and then later found it not going through. Given that, I'd suggest we
> > still
> > > throw the exception right away. Regarding the flag itself though, I
> agree
> > > that keeping it set until the next succeeded join group makes sense to
> be
> > > safer.
> > >
> > > 2) That's crystal, thank you for the clarification.
> > >
> > > On Wed, Apr 14, 2021 at 6:46 PM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > 1) Once the short-circuit is triggered, the member will downgrade to
> > the
> > > > EAGER protocol, but
> > > > won't necessarily try to rejoin the group right away.
> > > >
> > > > In the "happy path", the user has implemented #onPartitionsLost
> > correctly
> > > > and will not attempt
> > > > to commit partitions that are lost. And since these partitions have
> > > indeed
> > > > been revoked, the user
> > > > application should not attempt to commit those partitions after this
> > > point.
> > > > In this case, there's no
> > > > reason for the consumer to immediately rejoin the group. Since a
> > > > non-cooperative assignor was
> > > > selected, we know that all partitions have been assigned. This member
> > can
> > > > continue on as usual,
> > > > processing the remaining un-revoked partitions and will follow the
> > EAGER
> > > > protocol in the next
> > > > rebalance. There's no user-facing impact or handling required; all
> that
> > > > happens is that the work
> > > > since the last commit on those revoked partitions has been lost.
> > > >
> > > > In the less-happy path, the user has implemented #onPartitionsLost
> > > > incorrectly or not implemented
> > > > it at all, falling back on the default which invokes
> > #onPartitionsRevoked
> > > > which in turn will attempt to
> > > > commit those partitions during the rebalance callback. In this case
> we
> > > rely
> > > > on the flag to prevent
> > > > this commit request from being sent to the broker.
> > > >
> > > > Originally I was thinking we should throw a CommitFailedException up
> > > > through the #onPartitionsLost
> > > > callback, and eventually up through poll(), then rejoin the group.
> But
> > > now
> > > > I'm wondering if this is really
> > > > necessary -- the important point in all cases is just to prevent the
> > > > commit, but there's no reason the
> > > > consumer should not be allowed to continue processing its other
> > > partitions,
> > > > and it hasn't dropped out
> > > > of the g

[jira] [Created] (KAFKA-12911) Configure automatic formatter for org.apache.kafka.streams.processor

[Dongjin Lee (Jira)]

Dongjin Lee created KAFKA-12911:
---

 Summary: Configure automatic formatter for 
org.apache.kafka.streams.processor
 Key: KAFKA-12911
 URL: https://issues.apache.org/jira/browse/KAFKA-12911
 Project: Kafka
  Issue Type: Sub-task
Reporter: Dongjin Lee
Assignee: Dongjin Lee


As an incremental approach to introduce automatic code formatter, we will 
configure automatic formatter for org.apache.kafka.streams.processor package 
with 127 (main) + 78 (test) = 205 files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12910) Configure automatic formatter for org.apache.kafka.streams.state

[Dongjin Lee (Jira)]

Dongjin Lee created KAFKA-12910:
---

 Summary: Configure automatic formatter for 
org.apache.kafka.streams.state
 Key: KAFKA-12910
 URL: https://issues.apache.org/jira/browse/KAFKA-12910
 Project: Kafka
  Issue Type: Sub-task
Reporter: Dongjin Lee
Assignee: Dongjin Lee


As of 48379bd6e5, there are 893 java files in streams module.

As an incremental approach to introduce automatic code formatter, we will 
configure automatic formatter for org.apache.kafka.streams.processor package 
with 147 (main) + 91 (test) = 238 files 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Luke Chen]

Hi Ismael,
Thanks for the KIP.
+1 (non-binding)

Thanks.
Luke

On Tue, Jun 8, 2021 at 7:18 AM Konstantine Karantasis
 wrote:

> Thanks for the KIP Ismael. I agree that giving an early enough deprecation
> notice for an upcoming change like this one and doing so in a major release
> is best in this case.
>
> +1 (binding)
>
> Konstantine
>
> On Mon, Jun 7, 2021 at 7:30 AM Ryanne Dolan  wrote:
>
> > +1 (non-binding)
> >
> > Ryanne
> >
> > On Mon, Jun 7, 2021, 9:26 AM Satish Duggana 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, 7 Jun 2021 at 19:30, Dongjin Lee  wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > As a note: I think the exact removal schedule may be changed.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Mon, Jun 7, 2021, 10:56 PM Josep Prat  >
> > > > wrote:
> > > >
> > > > > Thanks Ismael, it's a +1 (non-binding) from my side,
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:
> > > > >
> > > > > > Since the goal is to provide ample warning regarding the future
> > > removal
> > > > > of
> > > > > > Java 8 support and the change we're proposing for 3.0 is simply a
> > > > > > documentation update, I am starting the vote:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > > > > >
> > > > > > If there are concerns or objections, feel free to point them out
> in
> > > > this
> > > > > > thread or the discuss thread.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Luke Chen]

Hi Ismael,
Thanks for the KIP!
+1 (non-binding)

Thanks.
Luke

On Tue, Jun 8, 2021 at 8:16 AM Israel Ekpo  wrote:

> I support this as well. The motivation and proposed changes makes sense to
> me
>
> +1 (non-binding)
>
> On Mon, Jun 7, 2021 at 6:28 PM Satish Duggana 
> wrote:
>
> > +1 (non-binding)
> >
> > On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
> >  wrote:
> >
> > > +1 (binding)
> > >
> > > Konstantine
> > >
> > > On Mon, Jun 7, 2021 at 6:51 AM Josep Prat  >
> > > wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Good KIP, it's a +1 (non binding) from my side.
> > > >
> > > > Best,
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
> > > >
> > > > > Since the goal is to provide ample warning regarding the future
> > removal
> > > > of
> > > > > Scala 2.12 and the change we're proposing for 3.0 is simply a
> > > > documentation
> > > > > update, I am starting the vote:
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/view
> > page.action?pageId=181308218
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> >
> > > > >
> > > > > If there are concerns or objections, feel free to point them out in
> > > this
> > > > > thread or the discuss thread.
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #204

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 471433 lines...]
[2021-06-08T01:20:10.155Z] 
[2021-06-08T01:20:10.155Z] SaslSslAdminIntegrationTest > testAclDelete() STARTED
[2021-06-08T01:20:11.062Z] 
[2021-06-08T01:20:11.062Z] TopicCommandIntegrationTest > 
testCreateTopicDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-06-08T01:20:11.062Z] 
[2021-06-08T01:20:11.062Z] TopicCommandIntegrationTest > 
testCreateWithReplicaAssignment() STARTED
[2021-06-08T01:20:15.615Z] 
[2021-06-08T01:20:15.615Z] TopicCommandIntegrationTest > 
testCreateWithReplicaAssignment() PASSED
[2021-06-08T01:20:15.615Z] 
[2021-06-08T01:20:15.615Z] TopicCommandIntegrationTest > 
testAlterWithInvalidPartitionCount() STARTED
[2021-06-08T01:20:19.184Z] 
[2021-06-08T01:20:19.184Z] TopicCommandIntegrationTest > 
testAlterWithInvalidPartitionCount() PASSED
[2021-06-08T01:20:19.184Z] 
[2021-06-08T01:20:19.184Z] TopicCommandIntegrationTest > testCreate() STARTED
[2021-06-08T01:20:22.714Z] 
[2021-06-08T01:20:22.714Z] TopicCommandIntegrationTest > testCreate() PASSED
[2021-06-08T01:20:22.714Z] 
[2021-06-08T01:20:22.714Z] TopicCommandIntegrationTest > 
testDescribeUnderMinIsrPartitionsMixed() STARTED
[2021-06-08T01:20:29.645Z] 
[2021-06-08T01:20:29.645Z] TopicCommandIntegrationTest > 
testDescribeUnderMinIsrPartitionsMixed() PASSED
[2021-06-08T01:20:29.645Z] 
[2021-06-08T01:20:29.645Z] TopicCommandIntegrationTest > 
testCreateWhenAlreadyExists() STARTED
[2021-06-08T01:20:33.262Z] 
[2021-06-08T01:20:33.262Z] TopicCommandIntegrationTest > 
testCreateWhenAlreadyExists() PASSED
[2021-06-08T01:20:33.262Z] 
[2021-06-08T01:20:33.262Z] TopicCommandIntegrationTest > testDescribe() STARTED
[2021-06-08T01:20:33.406Z] 
[2021-06-08T01:20:33.406Z] SaslSslAdminIntegrationTest > testAclDelete() PASSED
[2021-06-08T01:20:33.406Z] 
[2021-06-08T01:20:33.406Z] TransactionsTest > testBumpTransactionalEpoch() 
STARTED
[2021-06-08T01:20:37.981Z] 
[2021-06-08T01:20:37.981Z] TopicCommandIntegrationTest > testDescribe() PASSED
[2021-06-08T01:20:37.981Z] 
[2021-06-08T01:20:37.981Z] TopicCommandIntegrationTest > testAlterAssignment() 
STARTED
[2021-06-08T01:20:41.673Z] 
[2021-06-08T01:20:41.673Z] TopicCommandIntegrationTest > testAlterAssignment() 
PASSED
[2021-06-08T01:20:41.673Z] 
[2021-06-08T01:20:41.673Z] DynamicConnectionQuotaTest > 
testDynamicConnectionQuota() STARTED
[2021-06-08T01:20:45.204Z] 
[2021-06-08T01:20:45.204Z] DynamicConnectionQuotaTest > 
testDynamicConnectionQuota() PASSED
[2021-06-08T01:20:45.204Z] 
[2021-06-08T01:20:45.204Z] DynamicConnectionQuotaTest > 
testDynamicListenerConnectionQuota() STARTED
[2021-06-08T01:20:49.915Z] 
[2021-06-08T01:20:49.915Z] TransactionsTest > testBumpTransactionalEpoch() 
PASSED
[2021-06-08T01:20:49.915Z] 
[2021-06-08T01:20:49.915Z] TransactionsTest > 
testSendOffsetsWithGroupMetadata() STARTED
[2021-06-08T01:20:52.133Z] 
[2021-06-08T01:20:52.133Z] DynamicConnectionQuotaTest > 
testDynamicListenerConnectionQuota() PASSED
[2021-06-08T01:20:52.133Z] 
[2021-06-08T01:20:52.133Z] DynamicConnectionQuotaTest > 
testDynamicListenerConnectionCreationRateQuota() STARTED
[2021-06-08T01:21:00.855Z] 
[2021-06-08T01:21:00.855Z] TransactionsTest > 
testSendOffsetsWithGroupMetadata() PASSED
[2021-06-08T01:21:00.855Z] 
[2021-06-08T01:21:00.855Z] TransactionsTest > testBasicTransactions() STARTED
[2021-06-08T01:21:08.565Z] 
[2021-06-08T01:21:08.565Z] TransactionsTest > testBasicTransactions() PASSED
[2021-06-08T01:21:08.565Z] 
[2021-06-08T01:21:08.565Z] TransactionsTest > testSendOffsetsWithGroupId() 
STARTED
[2021-06-08T01:21:20.844Z] 
[2021-06-08T01:21:20.844Z] TransactionsTest > testSendOffsetsWithGroupId() 
PASSED
[2021-06-08T01:21:20.844Z] 
[2021-06-08T01:21:20.844Z] TransactionsTest > testFencingOnSendOffsets() STARTED
[2021-06-08T01:21:28.433Z] 
[2021-06-08T01:21:28.433Z] TransactionsTest > testFencingOnSendOffsets() PASSED
[2021-06-08T01:21:28.433Z] 
[2021-06-08T01:21:28.433Z] TransactionsTest > testFencingOnAddPartitions() 
STARTED
[2021-06-08T01:21:35.975Z] 
[2021-06-08T01:21:35.975Z] TransactionsTest > testFencingOnAddPartitions() 
PASSED
[2021-06-08T01:21:35.975Z] 
[2021-06-08T01:21:35.975Z] TransactionsTest > 
testFencingOnTransactionExpiration() STARTED
[2021-06-08T01:21:45.332Z] 
[2021-06-08T01:21:45.332Z] TransactionsTest > 
testFencingOnTransactionExpiration() PASSED
[2021-06-08T01:21:45.332Z] 
[2021-06-08T01:21:45.332Z] TransactionsTest > 
testDelayedFetchIncludesAbortedTransaction() STARTED
[2021-06-08T01:21:54.147Z] 
[2021-06-08T01:21:54.147Z] TransactionsTest > 
testDelayedFetchIncludesAbortedTransaction() PASSED
[2021-06-08T01:21:54.147Z] 
[2021-06-08T01:21:54.147Z] TransactionsTest > 
testOffsetMetadataInSendOffsetsToTransaction() STARTED
[2021-06-08T01:21:59.340Z] 
[2021-06-08T01:21:59.340Z] DynamicConnectionQuotaTest > 
testDynamicListenerConnectionCreationRateQuota() PASSED
[2021-06-08T01:21:59.340

Re: [DISCUSS] KIP-753: ACL authentication, Host field support IP network segment

[lobo xu]

The KIP address is wrong in the last email. This is the correct Kip Wiki address

https://cwiki.apache.org/confluence/display/KAFKA/KIP-753%3A++ACL+authentication%2C+Host+field+support+IP+network+segment


On 2021/06/07 16:24:50, lobo xu  wrote: 
> Hi all
> 
> I'd like to discuss the following kip, any suggestions are welcome.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-753%3A++ACL+authentication%2C+Host+field+support+IP+network+segment。
> 
> Many thanks,
> 
> Lobo
> 

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Israel Ekpo]

I support this as well. The motivation and proposed changes makes sense to
me

+1 (non-binding)

On Mon, Jun 7, 2021 at 6:28 PM Satish Duggana 
wrote:

> +1 (non-binding)
>
> On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
>  wrote:
>
> > +1 (binding)
> >
> > Konstantine
> >
> > On Mon, Jun 7, 2021 at 6:51 AM Josep Prat 
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > Good KIP, it's a +1 (non binding) from my side.
> > >
> > > Best,
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
> > >
> > > > Since the goal is to provide ample warning regarding the future
> removal
> > > of
> > > > Scala 2.12 and the change we're proposing for 3.0 is simply a
> > > documentation
> > > > update, I am starting the vote:
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/view
> page.action?pageId=181308218
> 
> > > >
> > > > If there are concerns or objections, feel free to point them out in
> > this
> > > > thread or the discuss thread.
> > > >
> > > > Ismael
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Cyrus Vafadari]

Since it is just two endpoints, I agree it's fine to move them over.

However, the endpoint you suggest would imply that someone could attempt to
validate the configs of an SMT or Header Converter, since it doesn't
specify the plugin's type. So if you agree I'll update the proposal to move
the PUT endpoint to

PUT /plugins/connector/{connector-type}/config/validate

so we don't have to return 400s for all the other types for which
validation is not supported.

Does that sound good to you?


On Mon, Jun 7, 2021 at 4:49 PM Konstantine Karantasis
 wrote:

> Hi Cyrus,
>
> Given that `/connector-plugins` is a top level endpoint, I think the
> suggestion to deprecate `/connector-plugins` should affect all the nested
> endpoints.
>
> This should be straightforward here (in terms of KIP changes but also
> implementation), because the endpoints are only two:
>
> GET /connector-plugins
> and
> PUT /connector-plugins/{connector-type}/config/validate
>
> I find deprecating both of them in favor of:
> GET /plugins
> and
> PUT /plugins/{connector-type}/config/validate
>
> more clear and aligned with how we want the new endpoint to look like.
>
> Wdyt?
>
> Konstantine
>
>
> On Mon, Jun 7, 2021 at 4:18 PM Cyrus Vafadari  wrote:
>
> > I wasn't trying to suggest deprecating all verbs and endpoints prefixed
> > with `/connector-plugins`, just the `GET: /connector-plugins` as I
> detailed
> > in the KIP.   Maybe I'm not understanding your recommendation, or
> there's a
> > way I can make the KIP text clearer? Or do you disagree this is the right
> > move tactically? I thought my text was sufficient: "it will be redundant
> to
> > this new, more generally useful endpoint."
> >
> > On Mon, Jun 7, 2021 at 12:56 PM Konstantine Karantasis
> >  wrote:
> >
> > > Thanks for the response Cyrus.
> > >
> > > If we are suggesting deprecation of `/connector-plugins` in favor of
> > > `/plugins` I think we should mention explicitly that the new endpoint
> > > covers the existing functionality precisely, including the validation
> > > endpoint for connector plugins.
> > > I'd recommend extending the spec to refer to this, adding an example to
> > > make this clear and mentioning that the two endpoints will be
> equivalent
> > > with respect to existing functionality in the upgrade section.
> > >
> > > Konstantine
> > >
> > > On Mon, Jun 7, 2021 at 10:57 AM Cyrus Vafadari 
> > > wrote:
> > >
> > > > I will move to vote
> > > >
> > > > On Mon, Jun 7, 2021 at 10:44 AM Cyrus Vafadari 
> > > > wrote:
> > > >
> > > > > Thanks for the feedback,
> > > > >
> > > > > I added an example request/response for SMTs, and I thought about
> > your
> > > > > suggestion re:deprecation and am now explicitly proposing to mark
> the
> > > > > existing endpoint as deprecated, though I don't anticipate the need
> > to
> > > > > remove it will arise any time soon!
> > > > >
> > > > > Cyrus
> > > > >
> > > > > On Fri, Jun 4, 2021 at 7:35 PM Konstantine Karantasis
> > > > >  wrote:
> > > > >
> > > > >> Hi Cyrus.
> > > > >>
> > > > >> The proposal looks good and I like the API spec definition the way
> > > it's
> > > > >> presented.
> > > > >>
> > > > >> Having said that, a few examples that would list the request type
> > and
> > > > >> body,
> > > > >> the returned status and the json response would be nice too,
> > following
> > > > the
> > > > >> tradition of other KIPs.
> > > > >>
> > > > >> See
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> > > > >> for a recent example.
> > > > >>
> > > > >> I also see there's no mention regarding the future of the current
> > > > >> `/connector-plugins` endpoint or any deprecation plans.
> > > > >>
> > > > >> I think we should make our intentions clear in the KIP itself
> > > > >> which introduces the new endpoint as a superset of the old one,
> > beyond
> > > > the
> > > > >> discussion in this thread, for future readers.
> > > > >> Also, I think it's useful to keep in mind that deprecation doesn't
> > > > >> necessarily mean imminent removal. The next opportunity to remove
> > this
> > > > end
> > > > >> point would be the next major release at the earliest.
> > > > >>
> > > > >> Regards,
> > > > >> Konstantine
> > > > >>
> > > > >>
> > > > >> On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari <
> > cvafad...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Thanks! Anyone else from the community with final thoughts
> before
> > > > going
> > > > >> to
> > > > >> > vote?
> > > > >> >
> > > > >> > On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley <
> tbent...@redhat.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi Cyrus,
> > > > >> > >
> > > > >> > > That seems reasonable to me.
> > > > >> > >
> > > > >> > > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari <
> > > cvafad...@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > What do you think of "type" field remaining in, and
> returning
> > > > >> > > > "tr

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Konstantine Karantasis]

Hi Cyrus,

Given that `/connector-plugins` is a top level endpoint, I think the
suggestion to deprecate `/connector-plugins` should affect all the nested
endpoints.

This should be straightforward here (in terms of KIP changes but also
implementation), because the endpoints are only two:

GET /connector-plugins
and
PUT /connector-plugins/{connector-type}/config/validate

I find deprecating both of them in favor of:
GET /plugins
and
PUT /plugins/{connector-type}/config/validate

more clear and aligned with how we want the new endpoint to look like.

Wdyt?

Konstantine


On Mon, Jun 7, 2021 at 4:18 PM Cyrus Vafadari  wrote:

> I wasn't trying to suggest deprecating all verbs and endpoints prefixed
> with `/connector-plugins`, just the `GET: /connector-plugins` as I detailed
> in the KIP.   Maybe I'm not understanding your recommendation, or there's a
> way I can make the KIP text clearer? Or do you disagree this is the right
> move tactically? I thought my text was sufficient: "it will be redundant to
> this new, more generally useful endpoint."
>
> On Mon, Jun 7, 2021 at 12:56 PM Konstantine Karantasis
>  wrote:
>
> > Thanks for the response Cyrus.
> >
> > If we are suggesting deprecation of `/connector-plugins` in favor of
> > `/plugins` I think we should mention explicitly that the new endpoint
> > covers the existing functionality precisely, including the validation
> > endpoint for connector plugins.
> > I'd recommend extending the spec to refer to this, adding an example to
> > make this clear and mentioning that the two endpoints will be equivalent
> > with respect to existing functionality in the upgrade section.
> >
> > Konstantine
> >
> > On Mon, Jun 7, 2021 at 10:57 AM Cyrus Vafadari 
> > wrote:
> >
> > > I will move to vote
> > >
> > > On Mon, Jun 7, 2021 at 10:44 AM Cyrus Vafadari 
> > > wrote:
> > >
> > > > Thanks for the feedback,
> > > >
> > > > I added an example request/response for SMTs, and I thought about
> your
> > > > suggestion re:deprecation and am now explicitly proposing to mark the
> > > > existing endpoint as deprecated, though I don't anticipate the need
> to
> > > > remove it will arise any time soon!
> > > >
> > > > Cyrus
> > > >
> > > > On Fri, Jun 4, 2021 at 7:35 PM Konstantine Karantasis
> > > >  wrote:
> > > >
> > > >> Hi Cyrus.
> > > >>
> > > >> The proposal looks good and I like the API spec definition the way
> > it's
> > > >> presented.
> > > >>
> > > >> Having said that, a few examples that would list the request type
> and
> > > >> body,
> > > >> the returned status and the json response would be nice too,
> following
> > > the
> > > >> tradition of other KIPs.
> > > >>
> > > >> See
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> > > >> for a recent example.
> > > >>
> > > >> I also see there's no mention regarding the future of the current
> > > >> `/connector-plugins` endpoint or any deprecation plans.
> > > >>
> > > >> I think we should make our intentions clear in the KIP itself
> > > >> which introduces the new endpoint as a superset of the old one,
> beyond
> > > the
> > > >> discussion in this thread, for future readers.
> > > >> Also, I think it's useful to keep in mind that deprecation doesn't
> > > >> necessarily mean imminent removal. The next opportunity to remove
> this
> > > end
> > > >> point would be the next major release at the earliest.
> > > >>
> > > >> Regards,
> > > >> Konstantine
> > > >>
> > > >>
> > > >> On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari <
> cvafad...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Thanks! Anyone else from the community with final thoughts before
> > > going
> > > >> to
> > > >> > vote?
> > > >> >
> > > >> > On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley 
> > > >> wrote:
> > > >> >
> > > >> > > Hi Cyrus,
> > > >> > >
> > > >> > > That seems reasonable to me.
> > > >> > >
> > > >> > > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari <
> > cvafad...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > What do you think of "type" field remaining in, and returning
> > > >> > > > "transformation", "converter" or whatever type it is. This way
> > the
> > > >> > schema
> > > >> > > > remains consistent, and you can programmatically understand
> what
> > > >> > plugins
> > > >> > > > are returned on the holistic "GET /plugins" endpoint? It will
> be
> > > >> > slightly
> > > >> > > > redundant in the case where you specify the plugin-type as a
> > path
> > > >> > > > parameter.
> > > >> > > >
> > > >> > > > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley <
> > tbent...@redhat.com>
> > > >> > wrote:
> > > >> > > >
> > > >> > > > > Hi Cyrus,
> > > >> > > > >
> > > >> > > > > Re 2: A very minor thing but while type=source|sink for a
> > > >> connector,
> > > >> > it
> > > >> > > > > doesn't makes sense for the other plugin types, but so the
> > json
> > > >> for
> > > >> > > those
> > > >> > > > > plugins should omit that property ra

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Sophie Blee-Goldman]

Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully
enable cooperative rebalancing
by default in 3.0 if we have KAFKA-12477 done in time, and if we don't then
there's no harm as it's
not going to change the behavior.

On Wed, Jun 2, 2021 at 7:28 PM Luke Chen  wrote:

> Hi Sophie,
> Thanks for the reminder. Yes, I was thinking this KIP doesn't have to be
> put into a major release since it will be fully backward compatible, so no
> need to push it. Currently, if we want to work on this KIP, we need
> KAFKA-12477 and KAFKA-12487. But you're right, we can at least try our best
> to see if we can make it into V3.0 since cooperative rebalancing is a major
> improvement. I'll kick off a vote later.
>
> Thank you.
> Luke
>
> On Thu, Jun 3, 2021 at 7:08 AM Sophie Blee-Goldman
>  wrote:
>
> > Hey Luke,
> >
> > It's been a while since the last update on this, which is mostly my fault
> > for picking up
> > other things in the meantime. I'm planning to get back to work
> > on KAFKA-12477 next
> > week but there are minimal changes to the current implementation given
> the
> > proposal
> > I put forth earlier in this KIP discussion, so I think we're good to go.
> >
> > Although this KIP no longer requires a major release since it should be
> > fully compatible, I
> > still hope we can get it in to 3.0 since cooperative rebalancing is a
> major
> > improvement to
> > the life of a consumer group (and its operator). Can we make sure the KIP
> > reflects the latest
> > and then kick off a vote by next Monday at the latest so we can make KIP
> > freeze?
> >
> > Thanks!
> > Sophie
> >
> > On Fri, Apr 16, 2021 at 2:33 PM Guozhang Wang 
> wrote:
> >
> > > 1) From user's perspective, it is always possible that a commit within
> > > onPartitionsRevoked throw in practice (e.g. if the member missed the
> > > previous rebalance where its assigned partitions are already
> re-assigned)
> > > -- and the onPartitionsLost was introduced for that exact reason, i.e.
> it
> > > is primarily for optimizations, but not for correctness guarantees --
> on
> > > the other hand, it would be surprising to users to see the commit
> returns
> > > and then later found it not going through. Given that, I'd suggest we
> > still
> > > throw the exception right away. Regarding the flag itself though, I
> agree
> > > that keeping it set until the next succeeded join group makes sense to
> be
> > > safer.
> > >
> > > 2) That's crystal, thank you for the clarification.
> > >
> > > On Wed, Apr 14, 2021 at 6:46 PM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > 1) Once the short-circuit is triggered, the member will downgrade to
> > the
> > > > EAGER protocol, but
> > > > won't necessarily try to rejoin the group right away.
> > > >
> > > > In the "happy path", the user has implemented #onPartitionsLost
> > correctly
> > > > and will not attempt
> > > > to commit partitions that are lost. And since these partitions have
> > > indeed
> > > > been revoked, the user
> > > > application should not attempt to commit those partitions after this
> > > point.
> > > > In this case, there's no
> > > > reason for the consumer to immediately rejoin the group. Since a
> > > > non-cooperative assignor was
> > > > selected, we know that all partitions have been assigned. This member
> > can
> > > > continue on as usual,
> > > > processing the remaining un-revoked partitions and will follow the
> > EAGER
> > > > protocol in the next
> > > > rebalance. There's no user-facing impact or handling required; all
> that
> > > > happens is that the work
> > > > since the last commit on those revoked partitions has been lost.
> > > >
> > > > In the less-happy path, the user has implemented #onPartitionsLost
> > > > incorrectly or not implemented
> > > > it at all, falling back on the default which invokes
> > #onPartitionsRevoked
> > > > which in turn will attempt to
> > > > commit those partitions during the rebalance callback. In this case
> we
> > > rely
> > > > on the flag to prevent
> > > > this commit request from being sent to the broker.
> > > >
> > > > Originally I was thinking we should throw a CommitFailedException up
> > > > through the #onPartitionsLost
> > > > callback, and eventually up through poll(), then rejoin the group.
> But
> > > now
> > > > I'm wondering if this is really
> > > > necessary -- the important point in all cases is just to prevent the
> > > > commit, but there's no reason the
> > > > consumer should not be allowed to continue processing its other
> > > partitions,
> > > > and it hasn't dropped out
> > > > of the group. What do you think about this slight amendment to my
> > > original
> > > > proposal: if a user does end up
> > > > calling commit for whatever reason when invoking #onPartitionsLost,
> > we'll
> > > > just swallow the resulting
> > > > CommitFailedException. So the user application wouldn't see anything,
> > and
> > > > the only impact would be
> > > > that these partitions were not able to commit those last se

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Satish Duggana]

+1 (non-binding)

On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
 wrote:

> +1 (binding)
>
> Konstantine
>
> On Mon, Jun 7, 2021 at 6:51 AM Josep Prat 
> wrote:
>
> > Hi Ismael,
> >
> > Good KIP, it's a +1 (non binding) from my side.
> >
> > Best,
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
> >
> > > Since the goal is to provide ample warning regarding the future removal
> > of
> > > Scala 2.12 and the change we're proposing for 3.0 is simply a
> > documentation
> > > update, I am starting the vote:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> > >
> > > If there are concerns or objections, feel free to point them out in
> this
> > > thread or the discuss thread.
> > >
> > > Ismael
> > >
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #203

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 473581 lines...]
[2021-06-07T23:20:14.515Z] 
[2021-06-07T23:20:14.515Z] ZooKeeperClientTest > 
testGetChildrenExistingZNodeWithChildren() PASSED
[2021-06-07T23:20:14.515Z] 
[2021-06-07T23:20:14.515Z] ZooKeeperClientTest > testSetDataExistingZNode() 
STARTED
[2021-06-07T23:20:14.515Z] 
[2021-06-07T23:20:14.515Z] ZooKeeperClientTest > testSetDataExistingZNode() 
PASSED
[2021-06-07T23:20:14.515Z] 
[2021-06-07T23:20:14.515Z] ZooKeeperClientTest > 
testZNodeChildChangeHandlerForChildChangeNotTriggered() STARTED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > 
testZNodeChildChangeHandlerForChildChangeNotTriggered() PASSED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testMixedPipeline() STARTED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testMixedPipeline() PASSED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testGetDataExistingZNode() 
STARTED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testGetDataExistingZNode() 
PASSED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testDeleteExistingZNode() 
STARTED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testDeleteExistingZNode() 
PASSED
[2021-06-07T23:20:15.442Z] 
[2021-06-07T23:20:15.442Z] ZooKeeperClientTest > testSessionExpiry() STARTED
[2021-06-07T23:20:18.217Z] 
[2021-06-07T23:20:18.217Z] ZooKeeperClientTest > testSessionExpiry() PASSED
[2021-06-07T23:20:18.217Z] 
[2021-06-07T23:20:18.217Z] ZooKeeperClientTest > testSetDataNonExistentZNode() 
STARTED
[2021-06-07T23:20:18.217Z] 
[2021-06-07T23:20:18.217Z] ZooKeeperClientTest > testSetDataNonExistentZNode() 
PASSED
[2021-06-07T23:20:18.217Z] 
[2021-06-07T23:20:18.217Z] ZooKeeperClientTest > testConnectionViaNettyClient() 
STARTED
[2021-06-07T23:20:19.170Z] 
[2021-06-07T23:20:19.170Z] ZooKeeperClientTest > testConnectionViaNettyClient() 
PASSED
[2021-06-07T23:20:19.170Z] 
[2021-06-07T23:20:19.170Z] ZooKeeperClientTest > testDeleteNonExistentZNode() 
STARTED
[2021-06-07T23:20:19.170Z] 
[2021-06-07T23:20:19.170Z] ZooKeeperClientTest > testDeleteNonExistentZNode() 
PASSED
[2021-06-07T23:20:19.170Z] 
[2021-06-07T23:20:19.170Z] ZooKeeperClientTest > testExistsExistingZNode() 
STARTED
[2021-06-07T23:20:19.170Z] 
[2021-06-07T23:20:19.170Z] ZooKeeperClientTest > testExistsExistingZNode() 
PASSED
[2021-06-07T23:20:19.170Z] 
[2021-06-07T23:20:19.170Z] ZooKeeperClientTest > 
testZooKeeperStateChangeRateMetrics() STARTED
[2021-06-07T23:20:20.096Z] 
[2021-06-07T23:20:20.096Z] ZooKeeperClientTest > 
testZooKeeperStateChangeRateMetrics() PASSED
[2021-06-07T23:20:20.096Z] 
[2021-06-07T23:20:20.096Z] ZooKeeperClientTest > 
testZNodeChangeHandlerForDeletion() STARTED
[2021-06-07T23:20:20.096Z] 
[2021-06-07T23:20:20.096Z] ZooKeeperClientTest > 
testZNodeChangeHandlerForDeletion() PASSED
[2021-06-07T23:20:20.096Z] 
[2021-06-07T23:20:20.096Z] ZooKeeperClientTest > testGetAclNonExistentZNode() 
STARTED
[2021-06-07T23:20:20.096Z] 
[2021-06-07T23:20:20.096Z] ZooKeeperClientTest > testGetAclNonExistentZNode() 
PASSED
[2021-06-07T23:20:20.096Z] 
[2021-06-07T23:20:20.096Z] ZooKeeperClientTest > 
testStateChangeHandlerForAuthFailure() STARTED
[2021-06-07T23:20:20.964Z] 
[2021-06-07T23:20:20.964Z] ZooKeeperClientTest > 
testStateChangeHandlerForAuthFailure() PASSED
[2021-06-07T23:20:44.671Z] 
[2021-06-07T23:20:44.671Z] SaslSslAdminIntegrationTest > 
testCreateTopicsResponseMetadataAndConfig() PASSED
[2021-06-07T23:20:44.671Z] 
[2021-06-07T23:20:44.671Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() STARTED
[2021-06-07T23:21:07.766Z] 
[2021-06-07T23:21:07.766Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() PASSED
[2021-06-07T23:21:07.766Z] 
[2021-06-07T23:21:07.766Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() STARTED
[2021-06-07T23:21:30.926Z] 
[2021-06-07T23:21:30.926Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() PASSED
[2021-06-07T23:21:30.926Z] 
[2021-06-07T23:21:30.926Z] SaslSslAdminIntegrationTest > testAclOperations() 
STARTED
[2021-06-07T23:21:58.355Z] 
[2021-06-07T23:21:58.355Z] SaslSslAdminIntegrationTest > testAclOperations() 
PASSED
[2021-06-07T23:21:58.355Z] 
[2021-06-07T23:21:58.355Z] SaslSslAdminIntegrationTest > testAclOperations2() 
STARTED
[2021-06-07T23:22:21.876Z] 
[2021-06-07T23:22:21.876Z] SaslSslAdminIntegrationTest > testAclOperations2() 
PASSED
[2021-06-07T23:22:21.876Z] 
[2021-06-07T23:22:21.876Z] SaslSslAdminIntegrationTest > testAclDelete() STARTED
[2021-06-07T23:22:49.253Z] 
[2021-06-07T23:22:49.253Z] SaslSslAdminIntegrationTest > testAclDelete() PASSED
[2021-06-07T23:22:49.253Z] 
[2021-06-07T23:22:49.253Z] TransactionsTest > testBumpTransactionalEpoch() 
STARTED
[2021-06-07T23:23:05

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Konstantine Karantasis]

+1 (binding)

Konstantine

On Mon, Jun 7, 2021 at 6:51 AM Josep Prat 
wrote:

> Hi Ismael,
>
> Good KIP, it's a +1 (non binding) from my side.
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
>
> > Since the goal is to provide ample warning regarding the future removal
> of
> > Scala 2.12 and the change we're proposing for 3.0 is simply a
> documentation
> > update, I am starting the vote:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> >
> > If there are concerns or objections, feel free to point them out in this
> > thread or the discuss thread.
> >
> > Ismael
> >
>

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Cyrus Vafadari]

I wasn't trying to suggest deprecating all verbs and endpoints prefixed
with `/connector-plugins`, just the `GET: /connector-plugins` as I detailed
in the KIP.   Maybe I'm not understanding your recommendation, or there's a
way I can make the KIP text clearer? Or do you disagree this is the right
move tactically? I thought my text was sufficient: "it will be redundant to
this new, more generally useful endpoint."

On Mon, Jun 7, 2021 at 12:56 PM Konstantine Karantasis
 wrote:

> Thanks for the response Cyrus.
>
> If we are suggesting deprecation of `/connector-plugins` in favor of
> `/plugins` I think we should mention explicitly that the new endpoint
> covers the existing functionality precisely, including the validation
> endpoint for connector plugins.
> I'd recommend extending the spec to refer to this, adding an example to
> make this clear and mentioning that the two endpoints will be equivalent
> with respect to existing functionality in the upgrade section.
>
> Konstantine
>
> On Mon, Jun 7, 2021 at 10:57 AM Cyrus Vafadari 
> wrote:
>
> > I will move to vote
> >
> > On Mon, Jun 7, 2021 at 10:44 AM Cyrus Vafadari 
> > wrote:
> >
> > > Thanks for the feedback,
> > >
> > > I added an example request/response for SMTs, and I thought about your
> > > suggestion re:deprecation and am now explicitly proposing to mark the
> > > existing endpoint as deprecated, though I don't anticipate the need to
> > > remove it will arise any time soon!
> > >
> > > Cyrus
> > >
> > > On Fri, Jun 4, 2021 at 7:35 PM Konstantine Karantasis
> > >  wrote:
> > >
> > >> Hi Cyrus.
> > >>
> > >> The proposal looks good and I like the API spec definition the way
> it's
> > >> presented.
> > >>
> > >> Having said that, a few examples that would list the request type and
> > >> body,
> > >> the returned status and the json response would be nice too, following
> > the
> > >> tradition of other KIPs.
> > >>
> > >> See
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> > >> for a recent example.
> > >>
> > >> I also see there's no mention regarding the future of the current
> > >> `/connector-plugins` endpoint or any deprecation plans.
> > >>
> > >> I think we should make our intentions clear in the KIP itself
> > >> which introduces the new endpoint as a superset of the old one, beyond
> > the
> > >> discussion in this thread, for future readers.
> > >> Also, I think it's useful to keep in mind that deprecation doesn't
> > >> necessarily mean imminent removal. The next opportunity to remove this
> > end
> > >> point would be the next major release at the earliest.
> > >>
> > >> Regards,
> > >> Konstantine
> > >>
> > >>
> > >> On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari 
> > >> wrote:
> > >>
> > >> > Thanks! Anyone else from the community with final thoughts before
> > going
> > >> to
> > >> > vote?
> > >> >
> > >> > On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley 
> > >> wrote:
> > >> >
> > >> > > Hi Cyrus,
> > >> > >
> > >> > > That seems reasonable to me.
> > >> > >
> > >> > > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari <
> cvafad...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > What do you think of "type" field remaining in, and returning
> > >> > > > "transformation", "converter" or whatever type it is. This way
> the
> > >> > schema
> > >> > > > remains consistent, and you can programmatically understand what
> > >> > plugins
> > >> > > > are returned on the holistic "GET /plugins" endpoint? It will be
> > >> > slightly
> > >> > > > redundant in the case where you specify the plugin-type as a
> path
> > >> > > > parameter.
> > >> > > >
> > >> > > > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley <
> tbent...@redhat.com>
> > >> > wrote:
> > >> > > >
> > >> > > > > Hi Cyrus,
> > >> > > > >
> > >> > > > > Re 2: A very minor thing but while type=source|sink for a
> > >> connector,
> > >> > it
> > >> > > > > doesn't makes sense for the other plugin types, but so the
> json
> > >> for
> > >> > > those
> > >> > > > > plugins should omit that property rather than have type=null.
> > >> > > > >
> > >> > > > > Apart from that it seems reasonable to me. Thanks again,
> > >> > > > >
> > >> > > > > Tom
> > >> > > > >
> > >> > > > > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari <
> > >> cvafad...@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Tom,
> > >> > > > > >
> > >> > > > > > Thanks for taking the time to respond with thoughtful
> > questions:
> > >> > > > > >
> > >> > > > > > 1. I propose HTTP-400, will update the KIP to reflect this
> > >> proposal
> > >> > > > > > 2. I think the schema should be of the same format. It is
> > quite
> > >> > > > minimal,
> > >> > > > > so
> > >> > > > > > there is room to add fields in the future without breaking
> > >> > > > compatibility.
> > >> > > > > > I will update the KIP to specify.
> > >> > > > > > 3. Yes, that's right.
> > >> > > > > > 4. I personally don't see the value in deprecating si

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Konstantine Karantasis]

Thanks for the KIP Ismael. I agree that giving an early enough deprecation
notice for an upcoming change like this one and doing so in a major release
is best in this case.

+1 (binding)

Konstantine

On Mon, Jun 7, 2021 at 7:30 AM Ryanne Dolan  wrote:

> +1 (non-binding)
>
> Ryanne
>
> On Mon, Jun 7, 2021, 9:26 AM Satish Duggana 
> wrote:
>
> > +1 (non-binding)
> >
> > On Mon, 7 Jun 2021 at 19:30, Dongjin Lee  wrote:
> >
> > > +1 (non-binding)
> > >
> > > As a note: I think the exact removal schedule may be changed.
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Mon, Jun 7, 2021, 10:56 PM Josep Prat 
> > > wrote:
> > >
> > > > Thanks Ismael, it's a +1 (non-binding) from my side,
> > > >
> > > >
> > > > Best,
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:
> > > >
> > > > > Since the goal is to provide ample warning regarding the future
> > removal
> > > > of
> > > > > Java 8 support and the change we're proposing for 3.0 is simply a
> > > > > documentation update, I am starting the vote:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > > > >
> > > > > If there are concerns or objections, feel free to point them out in
> > > this
> > > > > thread or the discuss thread.
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-12909) Allow users to opt-into spurious left/outer stream-stream join improvement

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-12909:
---

 Summary: Allow users to opt-into spurious left/outer stream-stream 
join improvement
 Key: KAFKA-12909
 URL: https://issues.apache.org/jira/browse/KAFKA-12909
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax
 Fix For: 3.0.0


https://issues.apache.org/jira/browse/KAFKA-10847 improves left/outer 
stream-stream join, by not emitting left/outer results eagerly, but only after 
the grace period passed.

While this change is desired, there is an issue with regard to upgrades: if 
users don't specify a grace period, we fall back to a 24h default. Thus, 
left/outer join results would only be emitted 24h after the join window end. 
This change in behavior could break existing applications when upgrading to 
3.0.0 release. – And even if users do set a grace period explicitly, it's still 
unclear if the new delayed output behavior would work for them.

Thus, we propose to disable the fix of KAFAK-10847 by default, and let user 
opt-into the fix explicitly instead.

To allow users to enable the fix, we want to piggy-back on KIP-633 
(https://issues.apache.org/jira/browse/KAFKA-8613) that deprecated the existing 
`JoinWindows.of()` and `JoinWindows#grace()` methods in favor of 
`JoinWindows.ofSizeAndGrace()` – if users don't update their code, we would 
keep the fix disabled, and thus, if users upgrade their app nothing changes. 
Only if users switch to the new `ofSizeAndGrace()` API, we enable the fix and 
thus give users the opportunity to opt-in expliclity and pick an appropriate 
grace period for their application.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-745: Connect API to restart connector and tasks

[Kalpesh Patel]

+1 (non-binding)

Regards
-Kalpesh

On Mon, Jun 7, 2021 at 3:10 PM Ryanne Dolan  wrote:

> +1 (non-binding)
>
> Ryanne
>
> On Mon, Jun 7, 2021, 3:03 PM Konstantine Karantasis
>  wrote:
>
> > Thanks Randall.
> >
> > +1 (binding)
> >
> > Konstantine
> >
> > On Mon, Jun 7, 2021 at 12:47 PM Randall Hauch  wrote:
> >
> > > Hello all,
> > >
> > > I would like to start a vote on KIP-745:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> > >
> > > +1 (binding) from myself.
> > >
> > > Thanks, and best regards!
> > >
> > > Randall
> > >
> >
>

Re: [DISCUSS] KIP-748: Add Broker Count Metrics

[Ryan Dielhenn]

Hey Colin and David,

I added another table for the ZK version of RegisteredBrokerCount.

Best,
Ryan Dielhenn

On 2021/06/04 08:21:27, David Jacot  wrote: 
> Hi Ryan,
> 
> Thanks for the KIP.
> 
> +1 for adding RegisteredBrokerCount to the ZK controller as well. This
> would be really helpful.
> 
> Best,
> David
> 
> On Fri, Jun 4, 2021 at 12:44 AM Colin McCabe  wrote:
> 
> > Hi Ryan,
> >
> > Thanks for the KIP. I think it would be good to provide the
> > RegisteredBrokerCount metric for the ZK controller as well as for the
> > Quorum controller. Looks good aside from that!
> >
> > best,
> > Colin
> >
> > On Thu, Jun 3, 2021, at 14:09, Ryan Dielhenn wrote:
> > > Hey kafka-dev,
> > >
> > > I created KIP-748 as a proposal to add broker count metrics to the
> > > Quorum Controller.
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-748%3A+Add+Broker+Count+Metrics#KIP748:AddBrokerCountMetrics
> > >
> > > Best,
> > > Ryan Dielhenn
> > >
> >
> 

[jira] [Created] (KAFKA-12908) Load snapshot heuristic

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-12908:
--

 Summary: Load snapshot heuristic
 Key: KAFKA-12908
 URL: https://issues.apache.org/jira/browse/KAFKA-12908
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio


The {{KafkaRaftCient}} implementation only forces the {{RaftClient.Listener}} 
to load a snapshot only when the listener's next offset is less than the start 
offset.

This is technically correct but in some cases it may be more efficient to load 
a snapshot even when the next offset exists in the log. This is clearly true 
when the latest snapshot has less entries than the number of records from the 
next offset to the latest snapshot.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Sophie Blee-Goldman]

+1 (binding)

Thanks Luke

On Mon, Jun 7, 2021 at 3:26 AM Luke Chen  wrote:

> Hi Ismael,
> Thanks for your comments. I updated the KIP for the "Compatibility, Upgrade
> path" section.
> Simply put, no special upgrade path is necessary.
>
> Thank you.
> Luke
>
>
> On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma  wrote:
>
> > Hi Luke,
> >
> > The KIP is a bit unclear with regards to whether two rolling bounces are
> > required or not. What exactly is being proposed?
> >
> > Ismael
> >
> > On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
> >
> > > Hi all,
> > > I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> > > range" as the default assignor.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> > >
> > > This KIP is still waiting for the prerequisite stories to get
> completed,
> > > but it should be soon. Hopefully this can be put into V3.0 since
> > > cooperative rebalancing is a major
> > > improvement to the life of a consumer group (and its operator).
> > >
> > > The discussion thread can be found here:
> > >
> > >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> > >
> > > Thank you.
> > > Luke
> > >
> >
>

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Sophie Blee-Goldman]

Thanks for the KIP! +1 (binding)

-Sophie

On Mon, Jun 7, 2021 at 11:19 AM Guozhang Wang  wrote:

> Thanks, that update LGTM. +1!
>
> Guozhang
>
> On Mon, Jun 7, 2021 at 9:45 AM Josep Prat 
> wrote:
>
> > Hi Guozhang,
> > Let me know if the updated KIP meets your requests. And thanks again for
> > your feedback!
> >
> > Thanks,
> >
> > On Sat, Jun 5, 2021 at 4:38 PM Josep Prat  wrote:
> >
> > > Done, KIP is now updated to reflect this.
> > >
> > > On Sat, Jun 5, 2021 at 4:29 PM Josep Prat  wrote:
> > >
> > >> Hi Guozhang,
> > >>
> > >> Thanks for your review, let's exclude the "localThreadsMetadata
> > returning
> > >> StreamsMetadata" change then. This way, as well, this KIP is
> completely
> > >> consistent on making those Metadata classes interfaces with internal
> > >> implementations.
> > >> I'll update the KIP document right away.
> > >>
> > >> Thanks for replying on Saturday :)
> > >>
> > >> On Sat, Jun 5, 2021 at 4:22 PM Guozhang Wang 
> > wrote:
> > >>
> > >>> Hi Josep,
> > >>>
> > >>> I think the most significant change would be, "localThreadsMetadata"
> > >>> returns a StreamsMetadata instead of ThreadMetadata, and
> > StreamsMetadata
> > >>> would expose a new API to return a set of ThreadMetadata.
> > >>>
> > >>> All others (including the repackaging and splitting of interface /
> > impl)
> > >>> are minor indeed.
> > >>>
> > >>> After a second thought, I feel it is not fair to squeeze in this
> > >>> significant change into your KIP, without the community having a
> > >>> separating
> > >>> discussion about it, so I think we can table it for now and only
> align
> > on
> > >>> the other minor things: 1) have a o.a.k.streams.StreamsMetadata
> > interface
> > >>> (along with an internal implementation class), 2) deprecate the
> > >>> o.a.k.streams.state.StreamsMetadata class and also the corresponding
> > >>> caller
> > >>> of Streams that returns this class.
> > >>>
> > >>> Guozhang
> > >>>
> > >>> On Fri, Jun 4, 2021 at 4:13 PM Josep Prat
>  > >
> > >>> wrote:
> > >>>
> > >>> > Hi Guozhang,
> > >>> > So if I understand correctly, it's only a couple of small changes
> > that
> > >>> need
> > >>> > to be made to this KIP to be aligned with KAFKA-12370, right?
> > >>> >
> > >>> > I'm guessing that StreamsMetadata would not only moved to
> > >>> o.a.k.streams but
> > >>> > also be split with Interface + internal implementation, am I right?
> > >>> >
> > >>> >
> > >>> > If that's the case, I could, most probably, update the KIP by
> > Saturday
> > >>> > afternoon CEST.
> > >>> >
> > >>> > Let me know if I understood you correctly.
> > >>> >
> > >>> > Thanks for the comments!
> > >>> >
> > >>> > ———
> > >>> > Josep Prat
> > >>> >
> > >>> > Aiven Deutschland GmbH
> > >>> >
> > >>> > Immanuelkirchstraße 26, 10405 Berlin
> > >>> >
> > >>> > Amtsgericht Charlottenburg, HRB 209739 B
> > >>> >
> > >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>> >
> > >>> > m: +491715557497
> > >>> >
> > >>> > w: aiven.io
> > >>> >
> > >>> > e: josep.p...@aiven.io
> > >>> >
> > >>> > On Sat, Jun 5, 2021, 00:11 Guozhang Wang 
> wrote:
> > >>> >
> > >>> > > Hello Josep,
> > >>> > >
> > >>> > > Thanks for the proposal! The write-up looks good to me in
> general.
> > >>> I'm
> > >>> > just
> > >>> > > wondering if you feel comfortable to align this with another
> > JIRA/KIP
> > >>> > > further down the road:
> > >>> > >
> > >>> > > https://issues.apache.org/jira/browse/KAFKA-12370
> > >>> > >
> > >>> > > Which tries to clean up the metadata hierarchy and the callers as
> > >>> > > StreamsMetadata -> ThreadMetadata -> TaskMetadata, and most
> Streams
> > >>> APIs
> > >>> > > return the top-level StreamsMetadata.
> > >>> > >
> > >>> > > It just have slight differences with the current proposal: 1)
> > >>> instead of
> > >>> > > returning a ThreadMetadata, "localThreadsMetadata" returns
> > >>> > > a StreamsMetadata, and 2) the `StreamsMetadata` would also be
> moved
> > >>> from
> > >>> > > o.a.k.streams.state to o.a.k.streams.
> > >>> > >
> > >>> > > What do you think about this? It's totally okay if you are not
> > >>> > comfortable
> > >>> > > changing or expanding the scope of this KIP, that's totally fine
> > >>> with me
> > >>> > as
> > >>> > > well, and we can just change again in the future if necessary ---
> > >>> just
> > >>> > > trying to see if we can align the direction on the first shot
> here
> > :)
> > >>> > >
> > >>> > >
> > >>> > > Guozhang
> > >>> > >
> > >>> > > On Fri, Jun 4, 2021 at 1:51 AM Bruno Cadonna  >
> > >>> wrote:
> > >>> > >
> > >>> > > > Thanks, Josep!
> > >>> > > >
> > >>> > > > +1 (binding)
> > >>> > > >
> > >>> > > > Bruno
> > >>> > > >
> > >>> > > > On 04.06.21 10:27, Josep Prat wrote:
> > >>> > > > > Hi all,
> > >>> > > > > I'd like to call for a vote on KIP-744: Migrate TaskMetadata
> > and
> > >>> > > > > ThreadMetadata to an interface with internal implementation
> > >>> > > > > KIP page can be found here:
> > >>> > > https://cwiki.apache.org/confluence/x/XIrOCg
> > >>> >

Re: [VOTE] KIP-722: Enable connector client overrides by default

[Randall Hauch]

Thanks, all.

KIP-722 passes with four binding +1 votes (Konstantine, Tom, Mickael, and
me), one non-binding +1 (Ryanne), and no -1 votes.

Best regards,

Randall

On Mon, Jun 7, 2021 at 3:16 AM Mickael Maison 
wrote:

> +1 (binding)
>
> Thanks for the KIP!
>
> On Mon, Jun 7, 2021 at 9:08 AM Tom Bentley  wrote:
> >
> > Thanks Randall,
> >
> > +1 (binding).
> >
> > Tom
> >
> > On Sat, Jun 5, 2021 at 2:23 AM Konstantine Karantasis
> >  wrote:
> >
> > > Good to flip the switch on this useful feature with the opportunity of
> the
> > > major release here as well.
> > > I don't see any disadvantages in doing so.
> > >
> > > +1 (binding)
> > >
> > > Konstantine
> > >
> > > On Thu, May 6, 2021 at 6:07 AM Ryanne Dolan 
> wrote:
> > >
> > > > +1 (non-binding) Thanks!
> > > >
> > > > Ryanne
> > > >
> > > > On Wed, May 5, 2021, 4:04 PM Randall Hauch 
> wrote:
> > > >
> > > > > I'd like to start a vote on KIP-722:
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-722%3A+Enable+connector+client+overrides+by+default
> > > > >
> > > > > +1 (binding) from myself.
> > > > >
> > > > > Thanks, and best regards!
> > > > >
> > > > > Randall
> > > > >
> > > >
> > >
>

Re: [VOTE] KIP-721: Enable connector log contexts in Connect Log4j configuration

[Randall Hauch]

Thanks, all.

KIP-721 passes with three binding +1 votes (Konstantine, Tom, and me), one
non-binding +1, and no -1 votes.

Best regards,

Randall

On Mon, Jun 7, 2021 at 3:20 AM Tom Bentley  wrote:

> +1 (binding).
>
> Thanks Randall.
>
> Tom
>
> On Sat, Jun 5, 2021 at 2:29 AM Konstantine Karantasis
>  wrote:
>
> > I agree 3.0.0 is a good opportunity to enable this useful feature by
> > default.
> >
> > +1 (binding)
> >
> > Konstantine
> >
> > On Fri, May 7, 2021 at 6:33 PM Dongjin Lee  wrote:
> >
> > > +1 (Non-binding)
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Thu, May 6, 2021 at 3:19 AM Randall Hauch 
> wrote:
> > >
> > > > I'd like to start a vote on KIP-721:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-721%3A+Enable+connector+log+contexts+in+Connect+Log4j+configuration
> > > >
> > > > +1 (binding) from myself.
> > > >
> > > > Thanks, and best regards!
> > > >
> > > > Randall
> > > >
> > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > >
> > >
> > > *github:  github.com/dongjinleekr
> > > keybase:
> > https://keybase.io/dongjinleekr
> > > linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > speakerdeck:
> > > speakerdeck.com/dongjin
> > > *
> > >
> >
>

Re: [VOTE] KIP-745: Connect API to restart connector and tasks

[Ryanne Dolan]

+1 (non-binding)

Ryanne

On Mon, Jun 7, 2021, 3:03 PM Konstantine Karantasis
 wrote:

> Thanks Randall.
>
> +1 (binding)
>
> Konstantine
>
> On Mon, Jun 7, 2021 at 12:47 PM Randall Hauch  wrote:
>
> > Hello all,
> >
> > I would like to start a vote on KIP-745:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> >
> > +1 (binding) from myself.
> >
> > Thanks, and best regards!
> >
> > Randall
> >
>

Re: [VOTE] KIP-745: Connect API to restart connector and tasks

[Konstantine Karantasis]

Thanks Randall.

+1 (binding)

Konstantine

On Mon, Jun 7, 2021 at 12:47 PM Randall Hauch  wrote:

> Hello all,
>
> I would like to start a vote on KIP-745:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
>
> +1 (binding) from myself.
>
> Thanks, and best regards!
>
> Randall
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #202

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 473596 lines...]
[2021-06-07T19:51:20.816Z] 
[2021-06-07T19:51:20.816Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicPartition() STARTED
[2021-06-07T19:51:23.443Z] 
[2021-06-07T19:51:23.443Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicPartition() PASSED
[2021-06-07T19:51:23.443Z] 
[2021-06-07T19:51:23.443Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicOnly() STARTED
[2021-06-07T19:51:27.148Z] 
[2021-06-07T19:51:27.148Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfEmptyConsumerGroupWithTopicOnly() PASSED
[2021-06-07T19:51:27.148Z] 
[2021-06-07T19:51:27.148Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithTopicPartition() STARTED
[2021-06-07T19:51:30.723Z] 
[2021-06-07T19:51:30.723Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithTopicPartition() PASSED
[2021-06-07T19:51:30.723Z] 
[2021-06-07T19:51:30.723Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsNonExistingGroup() STARTED
[2021-06-07T19:51:33.517Z] 
[2021-06-07T19:51:33.517Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsNonExistingGroup() PASSED
[2021-06-07T19:51:33.517Z] 
[2021-06-07T19:51:33.517Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithUnknownTopicOnly() STARTED
[2021-06-07T19:51:37.104Z] 
[2021-06-07T19:51:37.104Z] DeleteOffsetsConsumerGroupCommandIntegrationTest > 
testDeleteOffsetsOfStableConsumerGroupWithUnknownTopicOnly() PASSED
[2021-06-07T19:51:37.104Z] 
[2021-06-07T19:51:37.104Z] TopicCommandIntegrationTest > 
testAlterPartitionCount() STARTED
[2021-06-07T19:51:40.844Z] 
[2021-06-07T19:51:40.844Z] TopicCommandIntegrationTest > 
testAlterPartitionCount() PASSED
[2021-06-07T19:51:40.844Z] 
[2021-06-07T19:51:40.844Z] TopicCommandIntegrationTest > 
testCreatePartitionsDoesNotRetryThrottlingQuotaExceededException() STARTED
[2021-06-07T19:51:46.753Z] 
[2021-06-07T19:51:46.753Z] TopicCommandIntegrationTest > 
testCreatePartitionsDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-06-07T19:51:46.753Z] 
[2021-06-07T19:51:46.753Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExistWithIfExists() STARTED
[2021-06-07T19:51:49.384Z] 
[2021-06-07T19:51:49.384Z] TopicCommandIntegrationTest > 
testAlterWhenTopicDoesntExistWithIfExists() PASSED
[2021-06-07T19:51:49.384Z] 
[2021-06-07T19:51:49.384Z] TopicCommandIntegrationTest > 
testCreateWithDefaultReplication() STARTED
[2021-06-07T19:51:53.145Z] 
[2021-06-07T19:51:53.145Z] TopicCommandIntegrationTest > 
testCreateWithDefaultReplication() PASSED
[2021-06-07T19:51:53.145Z] 
[2021-06-07T19:51:53.145Z] TopicCommandIntegrationTest > 
testDescribeAtMinIsrPartitions() STARTED
[2021-06-07T19:52:01.584Z] 
[2021-06-07T19:52:01.584Z] TopicCommandIntegrationTest > 
testDescribeAtMinIsrPartitions() PASSED
[2021-06-07T19:52:01.584Z] 
[2021-06-07T19:52:01.584Z] TopicCommandIntegrationTest > 
testCreateWithNegativeReplicationFactor() STARTED
[2021-06-07T19:52:04.215Z] 
[2021-06-07T19:52:04.215Z] TopicCommandIntegrationTest > 
testCreateWithNegativeReplicationFactor() PASSED
[2021-06-07T19:52:04.215Z] 
[2021-06-07T19:52:04.215Z] TopicCommandIntegrationTest > 
testCreateWithInvalidReplicationFactor() STARTED
[2021-06-07T19:52:08.965Z] 
[2021-06-07T19:52:08.965Z] TopicCommandIntegrationTest > 
testCreateWithInvalidReplicationFactor() PASSED
[2021-06-07T19:52:08.965Z] 
[2021-06-07T19:52:08.965Z] TopicCommandIntegrationTest > 
testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() STARTED
[2021-06-07T19:52:12.545Z] 
[2021-06-07T19:52:12.545Z] TopicCommandIntegrationTest > 
testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() PASSED
[2021-06-07T19:52:12.545Z] 
[2021-06-07T19:52:12.545Z] TopicCommandIntegrationTest > 
testListTopicsWithExcludeInternal() STARTED
[2021-06-07T19:52:16.127Z] 
[2021-06-07T19:52:16.127Z] TopicCommandIntegrationTest > 
testListTopicsWithExcludeInternal() PASSED
[2021-06-07T19:52:16.127Z] 
[2021-06-07T19:52:16.127Z] TopicCommandIntegrationTest > 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress() STARTED
[2021-06-07T19:52:19.708Z] 
[2021-06-07T19:52:19.708Z] TopicCommandIntegrationTest > 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress() PASSED
[2021-06-07T19:52:19.708Z] 
[2021-06-07T19:52:19.708Z] TopicCommandIntegrationTest > 
testCreateWithNegativePartitionCount() STARTED
[2021-06-07T19:52:23.326Z] 
[2021-06-07T19:52:23.326Z] TopicCommandIntegrationTest > 
testCreateWithNegativePartitionCount() PASSED
[2021-06-07T19:52:23.326Z] 
[2021-06-07T19:52:23.326Z] TopicCommandIntegrationTest > 
testAlterWhenTopic

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Konstantine Karantasis]

Thanks for the response Cyrus.

If we are suggesting deprecation of `/connector-plugins` in favor of
`/plugins` I think we should mention explicitly that the new endpoint
covers the existing functionality precisely, including the validation
endpoint for connector plugins.
I'd recommend extending the spec to refer to this, adding an example to
make this clear and mentioning that the two endpoints will be equivalent
with respect to existing functionality in the upgrade section.

Konstantine

On Mon, Jun 7, 2021 at 10:57 AM Cyrus Vafadari  wrote:

> I will move to vote
>
> On Mon, Jun 7, 2021 at 10:44 AM Cyrus Vafadari 
> wrote:
>
> > Thanks for the feedback,
> >
> > I added an example request/response for SMTs, and I thought about your
> > suggestion re:deprecation and am now explicitly proposing to mark the
> > existing endpoint as deprecated, though I don't anticipate the need to
> > remove it will arise any time soon!
> >
> > Cyrus
> >
> > On Fri, Jun 4, 2021 at 7:35 PM Konstantine Karantasis
> >  wrote:
> >
> >> Hi Cyrus.
> >>
> >> The proposal looks good and I like the API spec definition the way it's
> >> presented.
> >>
> >> Having said that, a few examples that would list the request type and
> >> body,
> >> the returned status and the json response would be nice too, following
> the
> >> tradition of other KIPs.
> >>
> >> See
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> >> for a recent example.
> >>
> >> I also see there's no mention regarding the future of the current
> >> `/connector-plugins` endpoint or any deprecation plans.
> >>
> >> I think we should make our intentions clear in the KIP itself
> >> which introduces the new endpoint as a superset of the old one, beyond
> the
> >> discussion in this thread, for future readers.
> >> Also, I think it's useful to keep in mind that deprecation doesn't
> >> necessarily mean imminent removal. The next opportunity to remove this
> end
> >> point would be the next major release at the earliest.
> >>
> >> Regards,
> >> Konstantine
> >>
> >>
> >> On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari 
> >> wrote:
> >>
> >> > Thanks! Anyone else from the community with final thoughts before
> going
> >> to
> >> > vote?
> >> >
> >> > On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley 
> >> wrote:
> >> >
> >> > > Hi Cyrus,
> >> > >
> >> > > That seems reasonable to me.
> >> > >
> >> > > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari  >
> >> > > wrote:
> >> > >
> >> > > > What do you think of "type" field remaining in, and returning
> >> > > > "transformation", "converter" or whatever type it is. This way the
> >> > schema
> >> > > > remains consistent, and you can programmatically understand what
> >> > plugins
> >> > > > are returned on the holistic "GET /plugins" endpoint? It will be
> >> > slightly
> >> > > > redundant in the case where you specify the plugin-type as a path
> >> > > > parameter.
> >> > > >
> >> > > > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley 
> >> > wrote:
> >> > > >
> >> > > > > Hi Cyrus,
> >> > > > >
> >> > > > > Re 2: A very minor thing but while type=source|sink for a
> >> connector,
> >> > it
> >> > > > > doesn't makes sense for the other plugin types, but so the json
> >> for
> >> > > those
> >> > > > > plugins should omit that property rather than have type=null.
> >> > > > >
> >> > > > > Apart from that it seems reasonable to me. Thanks again,
> >> > > > >
> >> > > > > Tom
> >> > > > >
> >> > > > > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari <
> >> cvafad...@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Tom,
> >> > > > > >
> >> > > > > > Thanks for taking the time to respond with thoughtful
> questions:
> >> > > > > >
> >> > > > > > 1. I propose HTTP-400, will update the KIP to reflect this
> >> proposal
> >> > > > > > 2. I think the schema should be of the same format. It is
> quite
> >> > > > minimal,
> >> > > > > so
> >> > > > > > there is room to add fields in the future without breaking
> >> > > > compatibility.
> >> > > > > > I will update the KIP to specify.
> >> > > > > > 3. Yes, that's right.
> >> > > > > > 4. I personally don't see the value in deprecating since it's
> a
> >> > > simple
> >> > > > > > alias (interface and impl will both be simple changes). I
> would
> >> be
> >> > > > > > comfortable kicking that can down the road if/when there is a
> >> need
> >> > > for
> >> > > > an
> >> > > > > > actual breaking change. That way we can keep the scope and
> diff
> >> > here
> >> > > > > tight.
> >> > > > > > But iff the community feels strongly that deprecating is the
> >> right
> >> > > > thing
> >> > > > > to
> >> > > > > > do, I'm happy to update the KIP to propose deprecating.
> >> > > > > >
> >> > > > > > Thanks again!
> >> > > > > >
> >> > > > > > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley <
> >> tbent...@redhat.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Cyrus,
> >> > > > > > >
> >> > > > > > > Thanks for the KIP. A f

[VOTE] KIP-745: Connect API to restart connector and tasks

[Randall Hauch]

Hello all,

I would like to start a vote on KIP-745:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks

+1 (binding) from myself.

Thanks, and best regards!

Randall

Re: [DISCUSS] KIP-745: Connect API to restart connector and tasks

[Randall Hauch]

Thanks, Konstantine. I've incorporated your suggestion about the `tick()`
mention, and it sounds like my recent changes satisfy all of your other
concerns (which seem relatively minor). So I'll go ahead and start a vote
and we can continue to fine tune the wording.

Best regards,

Randall

On Mon, Jun 7, 2021 at 2:31 PM Konstantine Karantasis
 wrote:

> Thanks Randall. I reply inline as well.
>
>
> On Sat, Jun 5, 2021 at 1:57 PM Randall Hauch  wrote:
>
> > Thanks for the review and feedback, Konstantine. I think I've addressed
> all
> > of your concerns below.
> >
> > On Fri, Jun 4, 2021 at 7:55 PM Konstantine Karantasis
> >  wrote:
> >
> > > Thanks for the KIP Randall!
> > > Really happy to see this feature coming along finally. Will indeed
> > resolve
> > > an unintuitive aspect of Connect's REST API.
> > >
> > > I'm in favor of the current approach overall. Just a few comments below
> > > that could use a brief discussion:
> > >
> > > 1) Is there a specific reason why you choose the config topic to
> persist
> > > the restart requests?
> > > I know it doesn't really matter which internal topic is used for
> > backwards
> > > compatibility, because the worker ignores records that does not
> > understand.
> > >
> >
> > I think there's value in the restart request record being ordered along
> > with the other config topic records, including configuration records,
> > target state change records, and session key updates. This also aligns
> with
> > how the distributed herder will process the restart requests within its
> > `tick()` method, again making sure there are no ongoing starts and stops
> > from rebalances or new connectors while the restarts are taking place.
> Plus
> > there is already a built-in config topic listener mechanism in the
> > distributed herder.
> >
> > I think those are good reasons to use the config topic. But there also is
> > not a viable alternative: the status topic has no existing listener
> > mechanism in the StatusBackingStore, and the offset topic is clearly not
> > applicable.
> >
>
> These are good points that in practice make the config topic preferrable.
> Even though some sound more like implementation details, I think saying
> explicitly why the config topic gives us a good tradeoff (you mostly say
> it) and why the status topic is not preferred (some info is missing here,
> especially regarding the listener) is valuable.
>

I've tried to mention why the config topic is applicable here in the final
paragraph of the "Distributed Runtime" section. And I previously added a
third Rejected Alternative that mentions the cons of the status topic and
explicitly mentions "the StatusBackingStore interface does not define a
listener mechanism". Let me know if you still are looking for other
clarification.


> Also a bit of a nit but mentioning the "tick()" method might not be clear
> for the uninitiated with the internals of the Connect framework.
> Calling this the main thread loop of the workers might be simpler.
>

Good idea. I've updated the KIP.

>
> > > But are there any implications upon worker restart? I read you
> > specifically
> > > mention that a worker will read restart requests upon its own restart.
> > But
> > > is this really desirable?
> > >
> >
> > It will read them, just like all of the config records, target state
> change
> > records, and session key updates. However, upon worker restart the herder
> > will ignore any restart requests, just like it ignores session key update
> > and task state change records.
> >
> >
> > > The config topic offers a way to persist configurations of the latest
> > > running connectors and upon cluster restart to bring these connectors
> > back
> > > up.
> > > However, I'm not sure we want to persist recent restart attempts, the
> > same
> > > way we persist configurations. Additionally configurations might get
> > > additional backups in order to allow for the config topic to be
> recovered
> > > for disaster recovery reasons. To that respect, I'm not sure that the
> > > restart requests require the same guarantees.
> > >
> >
> > This is not the first or only non-config record written to this topic, so
> > there's already precedent.
> >
>
> I acknowledge the precedent but we might need to draw a line at some point
> and make sure we maintain more permanent information in the config topic
> and more transient in the status topic, modulo any other concerns or
> trade-offs.
> As discussed above, I agree that the config topic is a better option here.
>
>
> >
> > > Given the above, would the status topic offer a better alternative in
> > order
> > > to broadcast the restart requests across the workers? Haven't looked
> > > closely what would be the implications of writing these new records to
> > the
> > > status topic, but this topic tends to be more transient.
> > > Or is it that you've preferred the config topic for the workers to be
> > aware
> > > of the interleaving between restart requests and requests for
> > > reconfigurati

[jira] [Created] (KAFKA-12907) Cannot lockdown access to topic creation using Kafka ACLs

[Krishna Kinnal (Jira)]

Krishna Kinnal created KAFKA-12907:
--

 Summary: Cannot lockdown access to topic creation using Kafka ACLs
 Key: KAFKA-12907
 URL: https://issues.apache.org/jira/browse/KAFKA-12907
 Project: Kafka
  Issue Type: Bug
  Components: admin, security
Affects Versions: 2.6.1
Reporter: Krishna Kinnal


We're using Apache Kafka 2.6.1 clusters.

 

We set a certain identity to have all privileges to a cluster by running this 
command:-
{noformat}
➜  bin/kafka-acls.sh --bootstrap-server kafka-cluster:9094 --command-config 
../kafka-admin.properties --add --allow-principal User:CN=kafka-admin 
--operation All --cluster {noformat}
 

According to this page - 
[https://docs.confluent.io/platform/current/kafka/authorization.html#operations]
 - we figured that this would give only the "kafka-admin" identity the 
privilege to create/delete topics, and to create/delete ACLs, among other 
privileges. We noticed that most privileges like creating/deleting/describing 
ACLs, listing groups, etc., are locked down to this identity as expected, 
however topic creation is not. Meaning, any identity (that is not 
"kafka-admin") with a valid cert can still create topics.

 

This means that if a rogue client were to get hold of a cert with an identity 
that is not the admin identity, that client can create a topic and send 
terabytes worth of data to the topic to severely affect the cluster's 
performance, and with that, the other co-tenants' performance too. This is not 
an ideal scenario for us as Kafka admins since we would like to lockdown topic 
creation access only to identities with certificates that we possess. 

 

We're wondering if there's a workaround to this solution currently (like 
whether we're probably missing a config somewhere), or if this is an open issue 
that needs a fix.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-745: Connect API to restart connector and tasks

[Konstantine Karantasis]

Thanks Randall. I reply inline as well.


On Sat, Jun 5, 2021 at 1:57 PM Randall Hauch  wrote:

> Thanks for the review and feedback, Konstantine. I think I've addressed all
> of your concerns below.
>
> On Fri, Jun 4, 2021 at 7:55 PM Konstantine Karantasis
>  wrote:
>
> > Thanks for the KIP Randall!
> > Really happy to see this feature coming along finally. Will indeed
> resolve
> > an unintuitive aspect of Connect's REST API.
> >
> > I'm in favor of the current approach overall. Just a few comments below
> > that could use a brief discussion:
> >
> > 1) Is there a specific reason why you choose the config topic to persist
> > the restart requests?
> > I know it doesn't really matter which internal topic is used for
> backwards
> > compatibility, because the worker ignores records that does not
> understand.
> >
>
> I think there's value in the restart request record being ordered along
> with the other config topic records, including configuration records,
> target state change records, and session key updates. This also aligns with
> how the distributed herder will process the restart requests within its
> `tick()` method, again making sure there are no ongoing starts and stops
> from rebalances or new connectors while the restarts are taking place. Plus
> there is already a built-in config topic listener mechanism in the
> distributed herder.
>
> I think those are good reasons to use the config topic. But there also is
> not a viable alternative: the status topic has no existing listener
> mechanism in the StatusBackingStore, and the offset topic is clearly not
> applicable.
>

These are good points that in practice make the config topic preferrable.
Even though some sound more like implementation details, I think saying
explicitly why the config topic gives us a good tradeoff (you mostly say
it) and why the status topic is not preferred (some info is missing here,
especially regarding the listener) is valuable.
Also a bit of a nit but mentioning the "tick()" method might not be clear
for the uninitiated with the internals of the Connect framework.
Calling this the main thread loop of the workers might be simpler.


>
> > But are there any implications upon worker restart? I read you
> specifically
> > mention that a worker will read restart requests upon its own restart.
> But
> > is this really desirable?
> >
>
> It will read them, just like all of the config records, target state change
> records, and session key updates. However, upon worker restart the herder
> will ignore any restart requests, just like it ignores session key update
> and task state change records.
>
>
> > The config topic offers a way to persist configurations of the latest
> > running connectors and upon cluster restart to bring these connectors
> back
> > up.
> > However, I'm not sure we want to persist recent restart attempts, the
> same
> > way we persist configurations. Additionally configurations might get
> > additional backups in order to allow for the config topic to be recovered
> > for disaster recovery reasons. To that respect, I'm not sure that the
> > restart requests require the same guarantees.
> >
>
> This is not the first or only non-config record written to this topic, so
> there's already precedent.
>

I acknowledge the precedent but we might need to draw a line at some point
and make sure we maintain more permanent information in the config topic
and more transient in the status topic, modulo any other concerns or
trade-offs.
As discussed above, I agree that the config topic is a better option here.


>
> > Given the above, would the status topic offer a better alternative in
> order
> > to broadcast the restart requests across the workers? Haven't looked
> > closely what would be the implications of writing these new records to
> the
> > status topic, but this topic tends to be more transient.
> > Or is it that you've preferred the config topic for the workers to be
> aware
> > of the interleaving between restart requests and requests for
> > reconfigurations? If that's the case an explicit call out might be
> > worthwhile in the KIP itself.
> >
>
> I guess I thought I already tried to do that. But I can add a bit more
> detail to the KIP to call this out even more.
>

+1. Indeed there's some explanation on that already. Maybe a subsection
header and a juxtaposition with the status topic could help to highlight
the reason behind this selection in a more clear way.


>
>
> >
> > 2) You mention in the KIP the status STOPPED. But this is not currently
> > part of the task or connector states that get persisted to the status
> > topic, as these are defined in AbstractStatus#State enum. Should we
> remove
> > any reference to a STOPPED state to avoid confusion? The only new state
> > seems to be the RESTARTING state.
> >
>
> Good catch. Removed references of the STOPPED state.
>
> And a few minor comments.
> > a) The sentence "We need to keep the existing behavior of the method to
> > just restart the Connector 

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Guozhang Wang]

Thanks, that update LGTM. +1!

Guozhang

On Mon, Jun 7, 2021 at 9:45 AM Josep Prat 
wrote:

> Hi Guozhang,
> Let me know if the updated KIP meets your requests. And thanks again for
> your feedback!
>
> Thanks,
>
> On Sat, Jun 5, 2021 at 4:38 PM Josep Prat  wrote:
>
> > Done, KIP is now updated to reflect this.
> >
> > On Sat, Jun 5, 2021 at 4:29 PM Josep Prat  wrote:
> >
> >> Hi Guozhang,
> >>
> >> Thanks for your review, let's exclude the "localThreadsMetadata
> returning
> >> StreamsMetadata" change then. This way, as well, this KIP is completely
> >> consistent on making those Metadata classes interfaces with internal
> >> implementations.
> >> I'll update the KIP document right away.
> >>
> >> Thanks for replying on Saturday :)
> >>
> >> On Sat, Jun 5, 2021 at 4:22 PM Guozhang Wang 
> wrote:
> >>
> >>> Hi Josep,
> >>>
> >>> I think the most significant change would be, "localThreadsMetadata"
> >>> returns a StreamsMetadata instead of ThreadMetadata, and
> StreamsMetadata
> >>> would expose a new API to return a set of ThreadMetadata.
> >>>
> >>> All others (including the repackaging and splitting of interface /
> impl)
> >>> are minor indeed.
> >>>
> >>> After a second thought, I feel it is not fair to squeeze in this
> >>> significant change into your KIP, without the community having a
> >>> separating
> >>> discussion about it, so I think we can table it for now and only align
> on
> >>> the other minor things: 1) have a o.a.k.streams.StreamsMetadata
> interface
> >>> (along with an internal implementation class), 2) deprecate the
> >>> o.a.k.streams.state.StreamsMetadata class and also the corresponding
> >>> caller
> >>> of Streams that returns this class.
> >>>
> >>> Guozhang
> >>>
> >>> On Fri, Jun 4, 2021 at 4:13 PM Josep Prat  >
> >>> wrote:
> >>>
> >>> > Hi Guozhang,
> >>> > So if I understand correctly, it's only a couple of small changes
> that
> >>> need
> >>> > to be made to this KIP to be aligned with KAFKA-12370, right?
> >>> >
> >>> > I'm guessing that StreamsMetadata would not only moved to
> >>> o.a.k.streams but
> >>> > also be split with Interface + internal implementation, am I right?
> >>> >
> >>> >
> >>> > If that's the case, I could, most probably, update the KIP by
> Saturday
> >>> > afternoon CEST.
> >>> >
> >>> > Let me know if I understood you correctly.
> >>> >
> >>> > Thanks for the comments!
> >>> >
> >>> > ———
> >>> > Josep Prat
> >>> >
> >>> > Aiven Deutschland GmbH
> >>> >
> >>> > Immanuelkirchstraße 26, 10405 Berlin
> >>> >
> >>> > Amtsgericht Charlottenburg, HRB 209739 B
> >>> >
> >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> >
> >>> > m: +491715557497
> >>> >
> >>> > w: aiven.io
> >>> >
> >>> > e: josep.p...@aiven.io
> >>> >
> >>> > On Sat, Jun 5, 2021, 00:11 Guozhang Wang  wrote:
> >>> >
> >>> > > Hello Josep,
> >>> > >
> >>> > > Thanks for the proposal! The write-up looks good to me in general.
> >>> I'm
> >>> > just
> >>> > > wondering if you feel comfortable to align this with another
> JIRA/KIP
> >>> > > further down the road:
> >>> > >
> >>> > > https://issues.apache.org/jira/browse/KAFKA-12370
> >>> > >
> >>> > > Which tries to clean up the metadata hierarchy and the callers as
> >>> > > StreamsMetadata -> ThreadMetadata -> TaskMetadata, and most Streams
> >>> APIs
> >>> > > return the top-level StreamsMetadata.
> >>> > >
> >>> > > It just have slight differences with the current proposal: 1)
> >>> instead of
> >>> > > returning a ThreadMetadata, "localThreadsMetadata" returns
> >>> > > a StreamsMetadata, and 2) the `StreamsMetadata` would also be moved
> >>> from
> >>> > > o.a.k.streams.state to o.a.k.streams.
> >>> > >
> >>> > > What do you think about this? It's totally okay if you are not
> >>> > comfortable
> >>> > > changing or expanding the scope of this KIP, that's totally fine
> >>> with me
> >>> > as
> >>> > > well, and we can just change again in the future if necessary ---
> >>> just
> >>> > > trying to see if we can align the direction on the first shot here
> :)
> >>> > >
> >>> > >
> >>> > > Guozhang
> >>> > >
> >>> > > On Fri, Jun 4, 2021 at 1:51 AM Bruno Cadonna 
> >>> wrote:
> >>> > >
> >>> > > > Thanks, Josep!
> >>> > > >
> >>> > > > +1 (binding)
> >>> > > >
> >>> > > > Bruno
> >>> > > >
> >>> > > > On 04.06.21 10:27, Josep Prat wrote:
> >>> > > > > Hi all,
> >>> > > > > I'd like to call for a vote on KIP-744: Migrate TaskMetadata
> and
> >>> > > > > ThreadMetadata to an interface with internal implementation
> >>> > > > > KIP page can be found here:
> >>> > > https://cwiki.apache.org/confluence/x/XIrOCg
> >>> > > > > Discussion thread can be found here:
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://lists.apache.org/x/thread.html/r1d20fb6dbd6b01bb84cbb17e992f4d08308980dfc5f2e0a68d674413@%3Cdev.kafka.apache.org%3E
> >>> > > > >
> >>> > > > > As it was pointed out, hopefully this KIP can be approved
> before
> >>> the
> >>> > > 3.0
> >>> > > > > deadline, as we can clean up some non naming compliant methods
> >

Re: [VOTE] KIP-334 - Include partitions in exceptions raised during consumer record deserialization/validation

[Sarwar Bhuiyan]

Thank you very much for all those that voted.

With four binding +1s and zero -1s, this vote passes.

Binding +1:
* Colin McCabe
* Matthias J. Sax
*  Sophie Blee-Goldman
* Jason Gustafson

Thanks everyone!

Sarwar

On Thu, 3 Jun 2021 at 00:16, Jason Gustafson 
wrote:

> +1 overall. There is one complication which I think we may have to address
> in the implementation. The KIP documents an override to `fillInStackTrace`.
> Is that necessary? The trace would indeed be useful in this case because
> the new exception will wrap the exception raised from the Deserializer.
> Unfortunately, `SerializationException` already has the same override, so
> we may have some difficulty getting around that. I think the solution is
> probably to remove the override from `SerializationException` since it also
> seems undesirable, but we can discuss when the patch is submitted.
>
> -Jason
>
> On Wed, Jun 2, 2021 at 2:34 PM Colin McCabe  wrote:
>
> > +1 (binding)
> >
> > Thanks, Sarwar.
> >
> > best,
> > Colin
> >
> > On Wed, Jun 2, 2021, at 13:29, Sarwar Bhuiyan wrote:
> > > Thanks Colin, Matthias, and Jason on the discussion on this really old
> > KIP.
> > >
> > > As discussed, I'd like to start to vote on KIP-334:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87297793
> > >
> > > Thank you.
> > >
> > > Sarwar
> > >
> > > --
> > >
> > >
> > > [image: Confluent] 
> > > Sarwar Bhuiyan
> > > Staff Customer Success Technical Architect
> > > +447949684437
> > > Follow us: [image: Blog]
> > > <
> >
> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog
> > >[image:
> > > Twitter] [image: LinkedIn]
> > > [image: Slack]
> > > [image: YouTube]
> > > 
> > > [image: Kafka Summit] 
> > >
> >
>
-- 


[image: Confluent] 
Sarwar Bhuiyan
Staff Customer Success Technical Architect
+447949684437
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]

[image: Kafka Summit] 

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Cyrus Vafadari]

I will move to vote

On Mon, Jun 7, 2021 at 10:44 AM Cyrus Vafadari  wrote:

> Thanks for the feedback,
>
> I added an example request/response for SMTs, and I thought about your
> suggestion re:deprecation and am now explicitly proposing to mark the
> existing endpoint as deprecated, though I don't anticipate the need to
> remove it will arise any time soon!
>
> Cyrus
>
> On Fri, Jun 4, 2021 at 7:35 PM Konstantine Karantasis
>  wrote:
>
>> Hi Cyrus.
>>
>> The proposal looks good and I like the API spec definition the way it's
>> presented.
>>
>> Having said that, a few examples that would list the request type and
>> body,
>> the returned status and the json response would be nice too, following the
>> tradition of other KIPs.
>>
>> See
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
>> for a recent example.
>>
>> I also see there's no mention regarding the future of the current
>> `/connector-plugins` endpoint or any deprecation plans.
>>
>> I think we should make our intentions clear in the KIP itself
>> which introduces the new endpoint as a superset of the old one, beyond the
>> discussion in this thread, for future readers.
>> Also, I think it's useful to keep in mind that deprecation doesn't
>> necessarily mean imminent removal. The next opportunity to remove this end
>> point would be the next major release at the earliest.
>>
>> Regards,
>> Konstantine
>>
>>
>> On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari 
>> wrote:
>>
>> > Thanks! Anyone else from the community with final thoughts before going
>> to
>> > vote?
>> >
>> > On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley 
>> wrote:
>> >
>> > > Hi Cyrus,
>> > >
>> > > That seems reasonable to me.
>> > >
>> > > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari 
>> > > wrote:
>> > >
>> > > > What do you think of "type" field remaining in, and returning
>> > > > "transformation", "converter" or whatever type it is. This way the
>> > schema
>> > > > remains consistent, and you can programmatically understand what
>> > plugins
>> > > > are returned on the holistic "GET /plugins" endpoint? It will be
>> > slightly
>> > > > redundant in the case where you specify the plugin-type as a path
>> > > > parameter.
>> > > >
>> > > > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley 
>> > wrote:
>> > > >
>> > > > > Hi Cyrus,
>> > > > >
>> > > > > Re 2: A very minor thing but while type=source|sink for a
>> connector,
>> > it
>> > > > > doesn't makes sense for the other plugin types, but so the json
>> for
>> > > those
>> > > > > plugins should omit that property rather than have type=null.
>> > > > >
>> > > > > Apart from that it seems reasonable to me. Thanks again,
>> > > > >
>> > > > > Tom
>> > > > >
>> > > > > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari <
>> cvafad...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Tom,
>> > > > > >
>> > > > > > Thanks for taking the time to respond with thoughtful questions:
>> > > > > >
>> > > > > > 1. I propose HTTP-400, will update the KIP to reflect this
>> proposal
>> > > > > > 2. I think the schema should be of the same format. It is quite
>> > > > minimal,
>> > > > > so
>> > > > > > there is room to add fields in the future without breaking
>> > > > compatibility.
>> > > > > > I will update the KIP to specify.
>> > > > > > 3. Yes, that's right.
>> > > > > > 4. I personally don't see the value in deprecating since it's a
>> > > simple
>> > > > > > alias (interface and impl will both be simple changes). I would
>> be
>> > > > > > comfortable kicking that can down the road if/when there is a
>> need
>> > > for
>> > > > an
>> > > > > > actual breaking change. That way we can keep the scope and diff
>> > here
>> > > > > tight.
>> > > > > > But iff the community feels strongly that deprecating is the
>> right
>> > > > thing
>> > > > > to
>> > > > > > do, I'm happy to update the KIP to propose deprecating.
>> > > > > >
>> > > > > > Thanks again!
>> > > > > >
>> > > > > > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley <
>> tbent...@redhat.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > Hi Cyrus,
>> > > > > > >
>> > > > > > > Thanks for the KIP. A few questions:
>> > > > > > >
>> > > > > > > 1. What status code does the /plugins/{plugin-type} endpoint
>> > return
>> > > > > when
>> > > > > > an
>> > > > > > > unknown type is given?
>> > > > > > > 2. The result of /connector-plugins is a list of objects with
>> > > > 'class',
>> > > > > > > 'type' and 'version' properties. Presumably
>> /plugins/connector is
>> > > the
>> > > > > > same,
>> > > > > > > but what is the schema for the other plugin types?
>> > > > > > > 3. You're not proposing an equivalent of the
>> > > > > > > /connector-plugins/{connectorType}/config/validate endpoint
>> for
>> > > > > > > non-connector types, which I think makes sense, because you
>> would
>> > > > > > validate
>> > > > > > > an SMT's config via its used by a connector, so the existing
>> > > endpoint
>> > > > > > > suffices, right?
>> 

[VOTE] KIP-494 KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Cyrus Vafadari]

Hi all!

I'd like to kick off voting for KIP-494 --
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120740150

Discussion Thread --
https://lists.apache.org/x/thread.html/aa8c6eb080cae48046bd6c5af0f5c98dc4866c3f073f535c517a4921@%3Cdev.kafka.apache.org%3E

Cyrus

[jira] [Resolved] (KAFKA-12338) The code of MetadataRecordSerde duplicate with MetadataParser

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-12338.
-
Resolution: Fixed

> The code of MetadataRecordSerde duplicate with MetadataParser
> -
>
> Key: KAFKA-12338
> URL: https://issues.apache.org/jira/browse/KAFKA-12338
> Project: Kafka
>  Issue Type: Improvement
>Reporter: dengziming
>Assignee: dengziming
>Priority: Major
>
> For example: 
> MetadataRecordSerde.recordSize
> ```
> size += ByteUtils.sizeOfUnsignedVarint(data.message().apiKey());
>  size += ByteUtils.sizeOfUnsignedVarint(data.version());
>  size += data.message().size(serializationCache, data.version());
> ```
>  
> MetadataParser.size
> ```
> long messageSize = message.size(cache, version);
> long totalSize = messageSize +
>  ByteUtils.sizeOfUnsignedVarint(message.apiKey()) +
>  ByteUtils.sizeOfUnsignedVarint(version);
> ```
> we can see that the logic is duplicated except that `MetadataRecordSerde` has 
> an extra `DEFAULT_FRAME_VERSION`, if we want to change the serde format of 
> metadata, we should modify 2 classes, this is unreasonable.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Cyrus Vafadari]

Thanks for the feedback,

I added an example request/response for SMTs, and I thought about your
suggestion re:deprecation and am now explicitly proposing to mark the
existing endpoint as deprecated, though I don't anticipate the need to
remove it will arise any time soon!

Cyrus

On Fri, Jun 4, 2021 at 7:35 PM Konstantine Karantasis
 wrote:

> Hi Cyrus.
>
> The proposal looks good and I like the API spec definition the way it's
> presented.
>
> Having said that, a few examples that would list the request type and body,
> the returned status and the json response would be nice too, following the
> tradition of other KIPs.
>
> See
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> for a recent example.
>
> I also see there's no mention regarding the future of the current
> `/connector-plugins` endpoint or any deprecation plans.
>
> I think we should make our intentions clear in the KIP itself
> which introduces the new endpoint as a superset of the old one, beyond the
> discussion in this thread, for future readers.
> Also, I think it's useful to keep in mind that deprecation doesn't
> necessarily mean imminent removal. The next opportunity to remove this end
> point would be the next major release at the earliest.
>
> Regards,
> Konstantine
>
>
> On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari 
> wrote:
>
> > Thanks! Anyone else from the community with final thoughts before going
> to
> > vote?
> >
> > On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley  wrote:
> >
> > > Hi Cyrus,
> > >
> > > That seems reasonable to me.
> > >
> > > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari 
> > > wrote:
> > >
> > > > What do you think of "type" field remaining in, and returning
> > > > "transformation", "converter" or whatever type it is. This way the
> > schema
> > > > remains consistent, and you can programmatically understand what
> > plugins
> > > > are returned on the holistic "GET /plugins" endpoint? It will be
> > slightly
> > > > redundant in the case where you specify the plugin-type as a path
> > > > parameter.
> > > >
> > > > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley 
> > wrote:
> > > >
> > > > > Hi Cyrus,
> > > > >
> > > > > Re 2: A very minor thing but while type=source|sink for a
> connector,
> > it
> > > > > doesn't makes sense for the other plugin types, but so the json for
> > > those
> > > > > plugins should omit that property rather than have type=null.
> > > > >
> > > > > Apart from that it seems reasonable to me. Thanks again,
> > > > >
> > > > > Tom
> > > > >
> > > > > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari <
> cvafad...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > Thanks for taking the time to respond with thoughtful questions:
> > > > > >
> > > > > > 1. I propose HTTP-400, will update the KIP to reflect this
> proposal
> > > > > > 2. I think the schema should be of the same format. It is quite
> > > > minimal,
> > > > > so
> > > > > > there is room to add fields in the future without breaking
> > > > compatibility.
> > > > > > I will update the KIP to specify.
> > > > > > 3. Yes, that's right.
> > > > > > 4. I personally don't see the value in deprecating since it's a
> > > simple
> > > > > > alias (interface and impl will both be simple changes). I would
> be
> > > > > > comfortable kicking that can down the road if/when there is a
> need
> > > for
> > > > an
> > > > > > actual breaking change. That way we can keep the scope and diff
> > here
> > > > > tight.
> > > > > > But iff the community feels strongly that deprecating is the
> right
> > > > thing
> > > > > to
> > > > > > do, I'm happy to update the KIP to propose deprecating.
> > > > > >
> > > > > > Thanks again!
> > > > > >
> > > > > > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley  >
> > > > wrote:
> > > > > >
> > > > > > > Hi Cyrus,
> > > > > > >
> > > > > > > Thanks for the KIP. A few questions:
> > > > > > >
> > > > > > > 1. What status code does the /plugins/{plugin-type} endpoint
> > return
> > > > > when
> > > > > > an
> > > > > > > unknown type is given?
> > > > > > > 2. The result of /connector-plugins is a list of objects with
> > > > 'class',
> > > > > > > 'type' and 'version' properties. Presumably /plugins/connector
> is
> > > the
> > > > > > same,
> > > > > > > but what is the schema for the other plugin types?
> > > > > > > 3. You're not proposing an equivalent of the
> > > > > > > /connector-plugins/{connectorType}/config/validate endpoint for
> > > > > > > non-connector types, which I think makes sense, because you
> would
> > > > > > validate
> > > > > > > an SMT's config via its used by a connector, so the existing
> > > endpoint
> > > > > > > suffices, right?
> > > > > > > 4. Would /connector-plugins eventually be deprecated in favour
> of
> > > > > > > /plugins/connector, or do we expect it to remain in the API
> > > > > indefinitely
> > > > > > > and accept that /connector-plugins and /plugins/connector
> provide
> > > >

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Josep Prat]

Hi Guozhang,
Let me know if the updated KIP meets your requests. And thanks again for
your feedback!

Thanks,

On Sat, Jun 5, 2021 at 4:38 PM Josep Prat  wrote:

> Done, KIP is now updated to reflect this.
>
> On Sat, Jun 5, 2021 at 4:29 PM Josep Prat  wrote:
>
>> Hi Guozhang,
>>
>> Thanks for your review, let's exclude the "localThreadsMetadata returning
>> StreamsMetadata" change then. This way, as well, this KIP is completely
>> consistent on making those Metadata classes interfaces with internal
>> implementations.
>> I'll update the KIP document right away.
>>
>> Thanks for replying on Saturday :)
>>
>> On Sat, Jun 5, 2021 at 4:22 PM Guozhang Wang  wrote:
>>
>>> Hi Josep,
>>>
>>> I think the most significant change would be, "localThreadsMetadata"
>>> returns a StreamsMetadata instead of ThreadMetadata, and StreamsMetadata
>>> would expose a new API to return a set of ThreadMetadata.
>>>
>>> All others (including the repackaging and splitting of interface / impl)
>>> are minor indeed.
>>>
>>> After a second thought, I feel it is not fair to squeeze in this
>>> significant change into your KIP, without the community having a
>>> separating
>>> discussion about it, so I think we can table it for now and only align on
>>> the other minor things: 1) have a o.a.k.streams.StreamsMetadata interface
>>> (along with an internal implementation class), 2) deprecate the
>>> o.a.k.streams.state.StreamsMetadata class and also the corresponding
>>> caller
>>> of Streams that returns this class.
>>>
>>> Guozhang
>>>
>>> On Fri, Jun 4, 2021 at 4:13 PM Josep Prat 
>>> wrote:
>>>
>>> > Hi Guozhang,
>>> > So if I understand correctly, it's only a couple of small changes that
>>> need
>>> > to be made to this KIP to be aligned with KAFKA-12370, right?
>>> >
>>> > I'm guessing that StreamsMetadata would not only moved to
>>> o.a.k.streams but
>>> > also be split with Interface + internal implementation, am I right?
>>> >
>>> >
>>> > If that's the case, I could, most probably, update the KIP by Saturday
>>> > afternoon CEST.
>>> >
>>> > Let me know if I understood you correctly.
>>> >
>>> > Thanks for the comments!
>>> >
>>> > ———
>>> > Josep Prat
>>> >
>>> > Aiven Deutschland GmbH
>>> >
>>> > Immanuelkirchstraße 26, 10405 Berlin
>>> >
>>> > Amtsgericht Charlottenburg, HRB 209739 B
>>> >
>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> >
>>> > m: +491715557497
>>> >
>>> > w: aiven.io
>>> >
>>> > e: josep.p...@aiven.io
>>> >
>>> > On Sat, Jun 5, 2021, 00:11 Guozhang Wang  wrote:
>>> >
>>> > > Hello Josep,
>>> > >
>>> > > Thanks for the proposal! The write-up looks good to me in general.
>>> I'm
>>> > just
>>> > > wondering if you feel comfortable to align this with another JIRA/KIP
>>> > > further down the road:
>>> > >
>>> > > https://issues.apache.org/jira/browse/KAFKA-12370
>>> > >
>>> > > Which tries to clean up the metadata hierarchy and the callers as
>>> > > StreamsMetadata -> ThreadMetadata -> TaskMetadata, and most Streams
>>> APIs
>>> > > return the top-level StreamsMetadata.
>>> > >
>>> > > It just have slight differences with the current proposal: 1)
>>> instead of
>>> > > returning a ThreadMetadata, "localThreadsMetadata" returns
>>> > > a StreamsMetadata, and 2) the `StreamsMetadata` would also be moved
>>> from
>>> > > o.a.k.streams.state to o.a.k.streams.
>>> > >
>>> > > What do you think about this? It's totally okay if you are not
>>> > comfortable
>>> > > changing or expanding the scope of this KIP, that's totally fine
>>> with me
>>> > as
>>> > > well, and we can just change again in the future if necessary ---
>>> just
>>> > > trying to see if we can align the direction on the first shot here :)
>>> > >
>>> > >
>>> > > Guozhang
>>> > >
>>> > > On Fri, Jun 4, 2021 at 1:51 AM Bruno Cadonna 
>>> wrote:
>>> > >
>>> > > > Thanks, Josep!
>>> > > >
>>> > > > +1 (binding)
>>> > > >
>>> > > > Bruno
>>> > > >
>>> > > > On 04.06.21 10:27, Josep Prat wrote:
>>> > > > > Hi all,
>>> > > > > I'd like to call for a vote on KIP-744: Migrate TaskMetadata and
>>> > > > > ThreadMetadata to an interface with internal implementation
>>> > > > > KIP page can be found here:
>>> > > https://cwiki.apache.org/confluence/x/XIrOCg
>>> > > > > Discussion thread can be found here:
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://lists.apache.org/x/thread.html/r1d20fb6dbd6b01bb84cbb17e992f4d08308980dfc5f2e0a68d674413@%3Cdev.kafka.apache.org%3E
>>> > > > >
>>> > > > > As it was pointed out, hopefully this KIP can be approved before
>>> the
>>> > > 3.0
>>> > > > > deadline, as we can clean up some non naming compliant methods
>>> > recently
>>> > > > > introduced.
>>> > > > >
>>> > > > >
>>> > > > > Please note that the scope of the KIP increased during the
>>> discussion
>>> > > to
>>> > > > > also include ThreadMetadata.
>>> > > > >
>>> > > > > Thank you,
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > -- Guozhang
>>> > >
>>> >
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>>
>> --
>>
>> Josep Prat
>>
>> *Aiven Deutschland 

[jira] [Created] (KAFKA-12906) Consumer should include partition and offset number in deserialization exception

[Sarwar Bhuiyan (Jira)]

Sarwar Bhuiyan created KAFKA-12906:
--

 Summary: Consumer should include partition and offset number in 
deserialization exception
 Key: KAFKA-12906
 URL: https://issues.apache.org/jira/browse/KAFKA-12906
 Project: Kafka
  Issue Type: Bug
Reporter: Sarwar Bhuiyan
Assignee: Sarwar Bhuiyan


When we encounter an exception when deserializing it, we raise it to the user 
and keep the consumer's current position at the offset of the failed record. 
The expectation is that the user will either propagate the exception and 
shutdown or seek past the failed record.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-753: ACL authentication, Host field support IP network segment

[lobo xu]

Hi all

I'd like to discuss the following kip, any suggestions are welcome.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-753%3A++ACL+authentication%2C+Host+field+support+IP+network+segment。

Many thanks,

Lobo

[jira] [Resolved] (KAFKA-10614) Group coordinator onElection/onResignation should guard against leader epoch

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-10614.
---
Fix Version/s: 3.0.0
   Resolution: Fixed

> Group coordinator onElection/onResignation should guard against leader epoch
> 
>
> Key: KAFKA-10614
> URL: https://issues.apache.org/jira/browse/KAFKA-10614
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Guozhang Wang
>Assignee: Tom Bentley
>Priority: Major
> Fix For: 3.0.0
>
>
> When there are a sequence of LeaderAndISR or StopReplica requests sent from 
> different controllers causing the group coordinator to elect / resign, we may 
> re-order the events due to race condition. For example:
> 1) First LeaderAndISR request received from old controller to resign as the 
> group coordinator.
> 2) Second LeaderAndISR request received from new controller to elect as the 
> group coordinator.
> 3) Although threads handling the 1/2) requests are synchronized on the 
> replica manager, their callback {{onLeadershipChange}} would trigger 
> {{onElection/onResignation}} which would schedule the loading / unloading on 
> background threads, and are not synchronized.
> 4) As a result, the {{onElection}} maybe triggered by the thread first, and 
> then {{onResignation}}. As a result, the coordinator would not recognize it 
> self as the coordinator and hence would respond any coordinator request with 
> {{NOT_COORDINATOR}}.
> Here are two proposals on top of my head:
> 1) Let the scheduled load / unload function to keep the passed in leader 
> epoch, and also materialize the epoch in memory. Then when execute the 
> unloading check against the leader epoch.
> 2) This may be a bit simpler: using a single background thread working on a 
> FIFO queue of loading / unloading jobs, since the caller are actually 
> synchronized on replica manager and order preserved, the enqueued loading / 
> unloading job would be correctly ordered as well. In that case we would avoid 
> the reordering. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12905) Replace EasyMock and PowerMock with Mockito for NamedCacheMetricsTest

[YI-CHEN WANG (Jira)]

YI-CHEN WANG created KAFKA-12905:


 Summary: Replace EasyMock and PowerMock with Mockito for 
NamedCacheMetricsTest
 Key: KAFKA-12905
 URL: https://issues.apache.org/jira/browse/KAFKA-12905
 Project: Kafka
  Issue Type: Improvement
Reporter: YI-CHEN WANG
Assignee: YI-CHEN WANG


For 
[Kafka-7438|https://issues.apache.org/jira/browse/KAFKA-7438?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20Reopened)%20AND%20assignee%20in%20(EMPTY)%20AND%20text%20~%20%22mockito%22]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #201

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 473617 lines...]
[2021-06-07T15:22:16.138Z] > Task :server-common:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :storage:api:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :connect:api:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :connect:api:classes UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :streams:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :streams:classes UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :streams:jar UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :storage:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :raft:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :connect:json:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :connect:json:classes UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :connect:json:javadoc SKIPPED
[2021-06-07T15:22:16.138Z] > Task :connect:json:javadocJar
[2021-06-07T15:22:16.138Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-06-07T15:22:16.138Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-06-07T15:22:16.138Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :metadata:compileJava UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :core:compileJava NO-SOURCE
[2021-06-07T15:22:16.138Z] > Task :core:compileScala UP-TO-DATE
[2021-06-07T15:22:16.138Z] > Task :core:classes UP-TO-DATE
[2021-06-07T15:22:22.862Z] > Task :connect:api:javadoc
[2021-06-07T15:22:22.862Z] > Task :connect:api:javadocJar
[2021-06-07T15:22:24.996Z] > Task :streams:javadoc
[2021-06-07T15:22:25.941Z] > Task :streams:javadocJar
[2021-06-07T15:22:27.804Z] > Task :clients:javadoc
[2021-06-07T15:22:28.841Z] > Task :clients:javadocJar
[2021-06-07T15:22:28.841Z] > Task :clients:processTestMessages UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :clients:compileTestJava UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :clients:testClasses UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :core:compileTestJava NO-SOURCE
[2021-06-07T15:22:28.841Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :connect:api:jar UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-06-07T15:22:28.841Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :connect:json:jar UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-06-07T15:22:28.841Z] > Task :connect:api:testJar
[2021-06-07T15:22:28.841Z] > Task :connect:json:testClasses UP-TO-DATE
[2021-06-07T15:22:28.841Z] > Task :connect:json:testJar
[2021-06-07T15:22:28.841Z] > Task :connect:api:testSrcJar
[2021-06-07T15:22:28.841Z] > Task :connect:json:testSrcJar
[2021-06-07T15:22:29.785Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-06-07T15:22:29.785Z] > Task :connect:json:publishToMavenLocal
[2021-06-07T15:22:29.785Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-06-07T15:22:29.785Z] > Task :connect:api:publishToMavenLocal
[2021-06-07T15:22:29.785Z] > Task :core:compileTestScala UP-TO-DATE
[2021-06-07T15:22:29.785Z] > Task :core:testClasses UP-TO-DATE
[2021-06-07T15:22:29.785Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-06-07T15:22:29.785Z] > Task :streams:testClasses UP-TO-DATE
[2021-06-07T15:22:29.785Z] > Task :clients:testJar
[2021-06-07T15:22:30.823Z] > Task :streams:testJar
[2021-06-07T15:22:30.823Z] > Task :clients:testSrcJar
[2021-06-07T15:22:30.823Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-06-07T15:22:30.823Z] > Task :clients:publishToMavenLocal
[2021-06-07T15:22:30.823Z] > Task :streams:testSrcJar
[2021-06-07T15:22:30.823Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-06-07T15:22:30.823Z] > Task :streams:publishToMavenLocal
[2021-06-07T15:22:30.823Z] 
[2021-06-07T15:22:30.823Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-06-07T15:22:30.823Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2021-06-07T15:22:30.823Z] See 
https://docs.gradle.org/7.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-06-07T15:22:30.823Z] 
[2021-06-07T15:22:30.823Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-06-07T15:22:30.823Z] Please consult deprecation warnings for more details.
[2021-06-07T15:22:30.823Z] 
[2021-06-07T15:22:30.823Z] BUILD SUCCESSFUL in 41s
[2021-06-07T15:22:30.823Z] 71 

Re: [DISCUSS] KIP-746: Revise KRaft Metadata Records

[Colin McCabe]

I added a tagged "Listeners" field to BrokerRegistrationChangeRecord, as we 
discussed earlier in this thread. This will be set only if the broker listeners 
have changed.

best,
Colin


On Sat, Jun 5, 2021, at 09:03, Colin McCabe wrote:
> Right, we are not supporting upgrade (as discussed earlier). However, 
> since 2.8 was an official release that had the earlier versions of the 
> records, I think it's good to bump the version when we make 
> incompatible changes. For documentation purposes, it's good to be able 
> to see the differences versus what we had in 2.8. It's also good to be 
> consistent here. What do you think?
> 
> best,
> Colin
> 
> 
> On Thu, Jun 3, 2021, at 19:45, Ismael Juma wrote:
> > Quick question: given that upgrades are not supported, do we actually want
> > to introduce a new version for the RPCs?
> > 
> > Ismael
> > 
> > On Wed, Jun 2, 2021 at 11:02 AM Colin McCabe  wrote:
> > 
> > > Hi all,
> > >
> > > I have posted a KIP about updating the KRaft metadata records for 3.0.
> > >
> > > Check it out at : https://cwiki.apache.org/confluence/x/34zOCg
> > >
> > > best,
> > > Colin
> > >
> > 
> 

[jira] [Resolved] (KAFKA-12620) Producer IDs generated by the controller

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-12620.
--
  Reviewer: Colin McCabe
Resolution: Done

> Producer IDs generated by the controller
> 
>
> Key: KAFKA-12620
> URL: https://issues.apache.org/jira/browse/KAFKA-12620
> Project: Kafka
>  Issue Type: Improvement
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
>
> This is to track the implementation of 
> [KIP-730|https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12904) Connect's validate REST endpoint uses incorrect timeout

[Randall Hauch (Jira)]

Randall Hauch created KAFKA-12904:
-

 Summary: Connect's validate REST endpoint uses incorrect timeout
 Key: KAFKA-12904
 URL: https://issues.apache.org/jira/browse/KAFKA-12904
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.6.0
Reporter: Randall Hauch


The fix for KAFKA-9374 changed how the `ConnectorPluginsResource` and its 
method to validate connector configurations used the 
`ConnectorsResource.REQUEST_TIMEOUT_MS` constant (90 seconds). However, in 
doing so it introduced a bug where the timeout was actually 1000x longer than 
desired/specified.

In particular, the following line is currently:
```
return 
validationCallback.get(ConnectorsResource.REQUEST_TIMEOUT_MS, TimeUnit.SECONDS);
```
but should be:
```
return 
validationCallback.get(ConnectorsResource.REQUEST_TIMEOUT_MS, 
TimeUnit.MILLISECONDS);
```




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Ryanne Dolan]

+1 (non-binding)

Ryanne

On Mon, Jun 7, 2021, 9:26 AM Satish Duggana 
wrote:

> +1 (non-binding)
>
> On Mon, 7 Jun 2021 at 19:30, Dongjin Lee  wrote:
>
> > +1 (non-binding)
> >
> > As a note: I think the exact removal schedule may be changed.
> >
> > Best,
> > Dongjin
> >
> > On Mon, Jun 7, 2021, 10:56 PM Josep Prat 
> > wrote:
> >
> > > Thanks Ismael, it's a +1 (non-binding) from my side,
> > >
> > >
> > > Best,
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:
> > >
> > > > Since the goal is to provide ample warning regarding the future
> removal
> > > of
> > > > Java 8 support and the change we're proposing for 3.0 is simply a
> > > > documentation update, I am starting the vote:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > > >
> > > > If there are concerns or objections, feel free to point them out in
> > this
> > > > thread or the discuss thread.
> > > >
> > > > Ismael
> > > >
> > >
> >
>

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Satish Duggana]

+1 (non-binding)

On Mon, 7 Jun 2021 at 19:30, Dongjin Lee  wrote:

> +1 (non-binding)
>
> As a note: I think the exact removal schedule may be changed.
>
> Best,
> Dongjin
>
> On Mon, Jun 7, 2021, 10:56 PM Josep Prat 
> wrote:
>
> > Thanks Ismael, it's a +1 (non-binding) from my side,
> >
> >
> > Best,
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:
> >
> > > Since the goal is to provide ample warning regarding the future removal
> > of
> > > Java 8 support and the change we're proposing for 3.0 is simply a
> > > documentation update, I am starting the vote:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > >
> > > If there are concerns or objections, feel free to point them out in
> this
> > > thread or the discuss thread.
> > >
> > > Ismael
> > >
> >
>

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Dongjin Lee]

+1 (non-binding)

As a note: I think the exact removal schedule may be changed.

Best,
Dongjin

On Mon, Jun 7, 2021, 10:56 PM Josep Prat 
wrote:

> Thanks Ismael, it's a +1 (non-binding) from my side,
>
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:
>
> > Since the goal is to provide ample warning regarding the future removal
> of
> > Java 8 support and the change we're proposing for 3.0 is simply a
> > documentation update, I am starting the vote:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> >
> > If there are concerns or objections, feel free to point them out in this
> > thread or the discuss thread.
> >
> > Ismael
> >
>

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Josep Prat]

Thanks Ismael, it's a +1 (non-binding) from my side,


Best,

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:

> Since the goal is to provide ample warning regarding the future removal of
> Java 8 support and the change we're proposing for 3.0 is simply a
> documentation update, I am starting the vote:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
>
> If there are concerns or objections, feel free to point them out in this
> thread or the discuss thread.
>
> Ismael
>

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Josep Prat]

Hi Ismael,

Good KIP, it's a +1 (non binding) from my side.

Best,

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:

> Since the goal is to provide ample warning regarding the future removal of
> Scala 2.12 and the change we're proposing for 3.0 is simply a documentation
> update, I am starting the vote:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
>
> If there are concerns or objections, feel free to point them out in this
> thread or the discuss thread.
>
> Ismael
>

[VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma]

Since the goal is to provide ample warning regarding the future removal of
Java 8 support and the change we're proposing for 3.0 is simply a
documentation update, I am starting the vote:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223

If there are concerns or objections, feel free to point them out in this
thread or the discuss thread.

Ismael

[VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma]

Since the goal is to provide ample warning regarding the future removal of
Scala 2.12 and the change we're proposing for 3.0 is simply a documentation
update, I am starting the vote:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218

If there are concerns or objections, feel free to point them out in this
thread or the discuss thread.

Ismael

Re: [VOTE] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Bumping up the voting thread.


As a reminder: Please note that without the Log4j2 appender, we can't
complete KIP-653: Upgrade log4j to log4j2

(accepted)
since we can't entirely remove the log4j artifact from the classpath for
the tools dependency.


Best,

Dongjin

On Tue, May 25, 2021 at 10:45 PM Dongjin Lee  wrote:

> Hi Kafka dev,
>
> I'd like to kick-off the voting for KIP-719: Add Log4J2 Appender.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[jira] [Created] (KAFKA-12903) Replace producer state entry with auto-generated protocol

[dengziming (Jira)]

dengziming created KAFKA-12903:
--

 Summary: Replace producer state entry with auto-generated protocol
 Key: KAFKA-12903
 URL: https://issues.apache.org/jira/browse/KAFKA-12903
 Project: Kafka
  Issue Type: Improvement
Reporter: dengziming
Assignee: dengziming






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Luke Chen]

Hi Ismael,
Thanks for your comments. I updated the KIP for the "Compatibility, Upgrade
path" section.
Simply put, no special upgrade path is necessary.

Thank you.
Luke


On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma  wrote:

> Hi Luke,
>
> The KIP is a bit unclear with regards to whether two rolling bounces are
> required or not. What exactly is being proposed?
>
> Ismael
>
> On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
>
> > Hi all,
> > I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> > range" as the default assignor.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> >
> > This KIP is still waiting for the prerequisite stories to get completed,
> > but it should be soon. Hopefully this can be put into V3.0 since
> > cooperative rebalancing is a major
> > improvement to the life of a consumer group (and its operator).
> >
> > The discussion thread can be found here:
> >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> >
> > Thank you.
> > Luke
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #200

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 473764 lines...]
[2021-06-07T08:38:44.707Z] MetricsTest > 
testMetricsReporterAfterDeletingTopic() PASSED
[2021-06-07T08:38:44.707Z] 
[2021-06-07T08:38:44.707Z] MetricsTest > testSessionExpireListenerMetrics() 
STARTED
[2021-06-07T08:38:47.370Z] 
[2021-06-07T08:38:47.370Z] MetricsTest > testSessionExpireListenerMetrics() 
PASSED
[2021-06-07T08:38:47.370Z] 
[2021-06-07T08:38:47.370Z] MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic() STARTED
[2021-06-07T08:38:47.720Z] 
[2021-06-07T08:38:47.720Z] TopicCommandIntegrationTest > 
testCreateAlterTopicWithRackAware() PASSED
[2021-06-07T08:38:47.720Z] 
[2021-06-07T08:38:47.720Z] TopicCommandIntegrationTest > 
testListTopicsWithIncludeList() STARTED
[2021-06-07T08:38:50.007Z] 
[2021-06-07T08:38:50.007Z] MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic() PASSED
[2021-06-07T08:38:50.007Z] 
[2021-06-07T08:38:50.007Z] MetricsTest > testYammerMetricsCountMetric() STARTED
[2021-06-07T08:38:51.304Z] 
[2021-06-07T08:38:51.304Z] TopicCommandIntegrationTest > 
testListTopicsWithIncludeList() PASSED
[2021-06-07T08:38:51.304Z] 
[2021-06-07T08:38:51.304Z] TopicCommandIntegrationTest > testTopicDeletion() 
STARTED
[2021-06-07T08:38:52.551Z] 
[2021-06-07T08:38:52.551Z] MetricsTest > testYammerMetricsCountMetric() PASSED
[2021-06-07T08:38:52.551Z] 
[2021-06-07T08:38:52.551Z] MetricsTest > testClusterIdMetric() STARTED
[2021-06-07T08:38:55.188Z] 
[2021-06-07T08:38:55.188Z] MetricsTest > testClusterIdMetric() PASSED
[2021-06-07T08:38:55.188Z] 
[2021-06-07T08:38:55.188Z] MetricsTest > testControllerMetrics() STARTED
[2021-06-07T08:38:55.978Z] 
[2021-06-07T08:38:55.978Z] TopicCommandIntegrationTest > testTopicDeletion() 
PASSED
[2021-06-07T08:38:55.978Z] 
[2021-06-07T08:38:55.978Z] TopicCommandIntegrationTest > 
testCreateWithDefaults() STARTED
[2021-06-07T08:38:56.947Z] 
[2021-06-07T08:38:56.947Z] MetricsTest > testControllerMetrics() PASSED
[2021-06-07T08:38:56.947Z] 
[2021-06-07T08:38:56.947Z] MetricsTest > testWindowsStyleTagNames() STARTED
[2021-06-07T08:38:58.610Z] 
[2021-06-07T08:38:58.610Z] TopicCommandIntegrationTest > 
testCreateWithDefaults() PASSED
[2021-06-07T08:38:58.610Z] 
[2021-06-07T08:38:58.610Z] TopicCommandIntegrationTest > 
testDescribeReportOverriddenConfigs() STARTED
[2021-06-07T08:38:59.735Z] 
[2021-06-07T08:38:59.735Z] MetricsTest > testWindowsStyleTagNames() PASSED
[2021-06-07T08:38:59.735Z] 
[2021-06-07T08:38:59.735Z] MetricsTest > testBrokerStateMetric() STARTED
[2021-06-07T08:39:02.400Z] 
[2021-06-07T08:39:02.400Z] MetricsTest > testBrokerStateMetric() PASSED
[2021-06-07T08:39:02.400Z] 
[2021-06-07T08:39:02.400Z] MetricsTest > testBrokerTopicMetricsBytesInOut() 
STARTED
[2021-06-07T08:39:03.281Z] 
[2021-06-07T08:39:03.281Z] TopicCommandIntegrationTest > 
testDescribeReportOverriddenConfigs() PASSED
[2021-06-07T08:39:03.281Z] 
[2021-06-07T08:39:03.281Z] TopicCommandIntegrationTest > 
testDescribeWhenTopicDoesntExist() STARTED
[2021-06-07T08:39:07.564Z] 
[2021-06-07T08:39:07.564Z] MetricsTest > testBrokerTopicMetricsBytesInOut() 
PASSED
[2021-06-07T08:39:07.564Z] 
[2021-06-07T08:39:07.564Z] MetricsTest > testJMXFilter() STARTED
[2021-06-07T08:39:07.900Z] 
[2021-06-07T08:39:07.900Z] TopicCommandIntegrationTest > 
testDescribeWhenTopicDoesntExist() PASSED
[2021-06-07T08:39:07.900Z] 
[2021-06-07T08:39:07.900Z] TopicCommandIntegrationTest > 
testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized() STARTED
[2021-06-07T08:39:09.347Z] 
[2021-06-07T08:39:09.347Z] MetricsTest > testJMXFilter() PASSED
[2021-06-07T08:39:11.105Z] 
[2021-06-07T08:39:11.105Z] 1261 tests completed, 1 failed, 7 skipped
[2021-06-07T08:39:11.105Z] There were failing tests. See the report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/build/reports/tests/integrationTest/index.html
[2021-06-07T08:39:11.483Z] 
[2021-06-07T08:39:11.483Z] TopicCommandIntegrationTest > 
testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized() PASSED
[2021-06-07T08:39:11.483Z] 
[2021-06-07T08:39:11.483Z] TopicCommandIntegrationTest > 
testAlterAssignmentWithMoreAssignmentThanPartitions() STARTED
[2021-06-07T08:39:12.044Z] 
[2021-06-07T08:39:12.044Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-06-07T08:39:12.044Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2021-06-07T08:39:12.044Z] See 
https://docs.gradle.org/7.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-06-07T08:39:12.044Z] 
[2021-06-07T08:39:12.044Z] BUILD SUCCESSFUL in 1h 54m 21s
[2021-06-07T08:39:12.044Z] 196 actionable tasks: 105 executed, 91 up-to-date
[2021-06-07T08:39:12.044Z] 
[2021-06-07T08:39:12.044Z] See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/p

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Ismael Juma]

Hi Luke,

The KIP is a bit unclear with regards to whether two rolling bounces are
required or not. What exactly is being proposed?

Ismael

On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:

> Hi all,
> I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> range" as the default assignor.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
>
> This KIP is still waiting for the prerequisite stories to get completed,
> but it should be soon. Hopefully this can be put into V3.0 since
> cooperative rebalancing is a major
> improvement to the life of a consumer group (and its operator).
>
> The discussion thread can be found here:
>
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
>
> Thank you.
> Luke
>

Re: [VOTE] KIP-722: Enable connector client overrides by default

[Mickael Maison]

+1 (binding)

Thanks for the KIP!

On Mon, Jun 7, 2021 at 9:08 AM Tom Bentley  wrote:
>
> Thanks Randall,
>
> +1 (binding).
>
> Tom
>
> On Sat, Jun 5, 2021 at 2:23 AM Konstantine Karantasis
>  wrote:
>
> > Good to flip the switch on this useful feature with the opportunity of the
> > major release here as well.
> > I don't see any disadvantages in doing so.
> >
> > +1 (binding)
> >
> > Konstantine
> >
> > On Thu, May 6, 2021 at 6:07 AM Ryanne Dolan  wrote:
> >
> > > +1 (non-binding) Thanks!
> > >
> > > Ryanne
> > >
> > > On Wed, May 5, 2021, 4:04 PM Randall Hauch  wrote:
> > >
> > > > I'd like to start a vote on KIP-722:
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-722%3A+Enable+connector+client+overrides+by+default
> > > >
> > > > +1 (binding) from myself.
> > > >
> > > > Thanks, and best regards!
> > > >
> > > > Randall
> > > >
> > >
> >

Re: [VOTE] KIP-721: Enable connector log contexts in Connect Log4j configuration

[Tom Bentley]

+1 (binding).

Thanks Randall.

Tom

On Sat, Jun 5, 2021 at 2:29 AM Konstantine Karantasis
 wrote:

> I agree 3.0.0 is a good opportunity to enable this useful feature by
> default.
>
> +1 (binding)
>
> Konstantine
>
> On Fri, May 7, 2021 at 6:33 PM Dongjin Lee  wrote:
>
> > +1 (Non-binding)
> >
> > Thanks,
> > Dongjin
> >
> > On Thu, May 6, 2021 at 3:19 AM Randall Hauch  wrote:
> >
> > > I'd like to start a vote on KIP-721:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-721%3A+Enable+connector+log+contexts+in+Connect+Log4j+configuration
> > >
> > > +1 (binding) from myself.
> > >
> > > Thanks, and best regards!
> > >
> > > Randall
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase:
> https://keybase.io/dongjinleekr
> > linkedin:
> kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
>

Re: [VOTE] KIP-722: Enable connector client overrides by default

[Tom Bentley]

Thanks Randall,

+1 (binding).

Tom

On Sat, Jun 5, 2021 at 2:23 AM Konstantine Karantasis
 wrote:

> Good to flip the switch on this useful feature with the opportunity of the
> major release here as well.
> I don't see any disadvantages in doing so.
>
> +1 (binding)
>
> Konstantine
>
> On Thu, May 6, 2021 at 6:07 AM Ryanne Dolan  wrote:
>
> > +1 (non-binding) Thanks!
> >
> > Ryanne
> >
> > On Wed, May 5, 2021, 4:04 PM Randall Hauch  wrote:
> >
> > > I'd like to start a vote on KIP-722:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-722%3A+Enable+connector+client+overrides+by+default
> > >
> > > +1 (binding) from myself.
> > >
> > > Thanks, and best regards!
> > >
> > > Randall
> > >
> >
>

[jira] [Created] (KAFKA-12902) Add UINT32 type in generator

[dengziming (Jira)]

dengziming created KAFKA-12902:
--

 Summary: Add UINT32 type in generator
 Key: KAFKA-12902
 URL: https://issues.apache.org/jira/browse/KAFKA-12902
 Project: Kafka
  Issue Type: Improvement
Reporter: dengziming
Assignee: dengziming


We support unit32 in Struct but don't support unit32 in generator protocol.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)