Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #781

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

[Ryanne Dolan]

Thanks Mickael, this makes sense to me! I've been wanting something like
this in order to decommission a broker without new partitions getting
accidentally assigned to it.

Ryanne

On Thu, Mar 17, 2022, 5:56 AM Mickael Maison 
wrote:

> Hi,
>
> I'd like to start a new discussion on KIP-660. I originally wrote this
> KIP in 2020 and the initial discussion
> (https://lists.apache.org/thread/xn7xyb74nyt281brto4x28r9rzxm4lp9)
> raised some concerns especially around KRaft (which did not exist at
> that time) and scalability.
>
> Since then, we got a new KRaft controller so I've been able to revisit
> this KIP. I kept the KIP number as it's essentially the same idea, but
> the proposal is significantly different:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-660%3A+Pluggable+ReplicaPlacer
>
> Please take a look and let me know if you have any feedback.
>
> Thanks,
> Mickael
>

[jira] [Resolved] (KAFKA-7077) KIP-318: Make Kafka Connect Source idempotent

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-7077.
--
Resolution: Fixed

> KIP-318: Make Kafka Connect Source idempotent
> -
>
> Key: KAFKA-7077
> URL: https://issues.apache.org/jira/browse/KAFKA-7077
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 2.0.0
>Reporter: Stephane Maarek
>Assignee: Stephane Maarek
>Priority: Major
> Fix For: 3.2.0
>
>
> KIP Link: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13742) Quota byte-rate/request metrics are loaded only when at least one quota is register

[Jorge Esteban Quilcate Otoya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Esteban Quilcate Otoya resolved KAFKA-13742.
--
Resolution: Not A Problem

Given that I got a better understanding of how quota metrics work 
(https://issues.apache.org/jira/browse/KAFKA-13744), I will close this one.

The only comment I may add is that there are no metrics at the moment to match 
client/users with topic/partitions. This information is only captured on the 
client-side as far as I know. I found quota metrics as a good proxy to get this 
mapping, though is still incomplete as the map to the topic is implicit at the 
moment on the broker-side.

Would be a nice addition to have this mapping available, but it should be 
discussed in another issue if there's interest in that.

> Quota byte-rate/request metrics are loaded only when at least one quota is 
> register
> ---
>
> Key: KAFKA-13742
> URL: https://issues.apache.org/jira/browse/KAFKA-13742
> Project: Kafka
>  Issue Type: Bug
>  Components: core, metrics
>Reporter: Jorge Esteban Quilcate Otoya
>Priority: Major
>  Labels: quotas
>
> Quota metrics are loaded only when at least one quota is present:
>  * Metrics: 
> [https://github.com/apache/kafka/blob/0b9a8bac36f16b5397e9ec3a0441758e4b60a384/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L552-L563]
>  * Reporting when quotas are enabled: 
> [https://github.com/apache/kafka/blob/0b9a8bac36f16b5397e9ec3a0441758e4b60a384/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L249-L256]
>  * Quotas enabled: `def quotasEnabled: Boolean = quotaTypesEnabled != 
> QuotaTypes.NoQuotas`
> Even though throttling is specific for quotas, byte-rate/request per 
> user/client-id is a valid metric for any deployment.
>  
> The current workaround is to add _any_ quota, as this will enable metrics for 
> *all* client-id/users.
> If these metrics are captured for all clients regardless of the quotas 
> created, it would be a better experience to have a config to opt-in into 
> these metrics instead of creating meaningless quotas just to get these 
> metrics.
> For threshold metrics, it makes sense to me to enable them only when quotas 
> are enabled.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13744) Quota metric tags are inconsistent

[Jorge Esteban Quilcate Otoya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Esteban Quilcate Otoya resolved KAFKA-13744.
--
Resolution: Not A Problem

> Quota metric tags are inconsistent
> --
>
> Key: KAFKA-13744
> URL: https://issues.apache.org/jira/browse/KAFKA-13744
> Project: Kafka
>  Issue Type: Bug
>  Components: core, metrics
>Reporter: Jorge Esteban Quilcate Otoya
>Priority: Major
>  Labels: quotas
> Attachments: image-2022-03-15-16-57-12-583.png
>
>
> When enabling metrics for quotas the metrics apply to _all_ clients (see 
> https://issues.apache.org/jira/browse/KAFKA-13742).
> Though, the tags are calculated depending on the quotas registered and 
> applied to all clients: 
> [https://github.com/apache/kafka/blob/0b9a8bac36f16b5397e9ec3a0441758e4b60a384/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L649-L694]
> This causes different metric tags result depending on which quota is 
> registered first.
> For instance, if a quota is registered with userId and clientId, then metrics 
> are tagged with both, though if then a quota is registered with only tagged 
> with clientId, then all metrics are only tagged by clientId — even though 
> user principal is available.
> !image-2022-03-15-16-57-12-583.png|width=1034,height=415!
> I managed to reproduce this behavior here:
>  * From 10:30 to 10:45, there was a quota with both client-id and user-id
>  * It was removed by 10:45, so no metrics were exposed.
>  * After, a quota with client id was created, and metrics were collected only 
> with client id, even though the user was available.
> I'd expect metrics to always contain both, if available — and simplify the 
> logic here 
> [https://github.com/apache/kafka/blob/0b9a8bac36f16b5397e9ec3a0441758e4b60a384/core/src/main/scala/kafka/server/ClientQuotaManager.scala#L649-L694].



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Hi all,

Thank you to everybody who contributed to this discussion, we've come a
long way and now the "rejected alternatives" section is almost as big as
the actual proposal (which indicates that it's a well thought through
solution :-)).  If there are no further considerations, I'll start voting
in the next couple of days.

-Artem

On Mon, Mar 14, 2022 at 6:19 PM Artem Livshits 
wrote:

> Hi Jun,
>
> 33. Sounds good.  Updated the KIP.
>
> -Artem
>
> On Mon, Mar 14, 2022 at 5:45 PM Jun Rao  wrote:
>
>> Hi, Artem,
>>
>> 33. We introduced onNewBatch() primarily for the sticky partitioner. It
>> seems to be a very subtle API to explain and to use properly. If we can't
>> find any convincing usage, it's probably better to deprecate it so that we
>> could keep the API clean.
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Mar 14, 2022 at 1:36 PM Artem Livshits
>>  wrote:
>>
>> > Hi Jun,
>> >
>> > 33.  That's an interesting point.  Technically, onNewBatch is just a
>> way to
>> > pass some signal to the partitioner, the sticky partitioner uses this
>> > signal that is suboptimal, in theory someone could use it for something
>> > else
>> >
>> > -Artem
>> >
>> > On Mon, Mar 14, 2022 at 9:11 AM Jun Rao 
>> wrote:
>> >
>> > > Hi, Artem,
>> > >
>> > > Thanks for the reply. A couple of more comments.
>> > >
>> > > 32. We could defer the metrics until we know what to add.
>> > >
>> > > 33. Since we are deprecating DefaultPartitioner and
>> > > UniformStickyPartitioner, should we depreciate
>> Partitioner.onNewBatch()
>> > too
>> > > given its unexpected side effect?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Thu, Mar 10, 2022 at 5:20 PM Artem Livshits
>> > >  wrote:
>> > >
>> > > > Hi Jun,
>> > > >
>> > > > 32. Good point.  Do you think it's ok to defer the metrics until we
>> run
>> > > > some benchmarks so that we get a better idea of what metrics we
>> need?
>> > > >
>> > > > -Artem
>> > > >
>> > > > On Thu, Mar 10, 2022 at 3:12 PM Jun Rao 
>> > > wrote:
>> > > >
>> > > > > Hi, Artem.
>> > > > >
>> > > > > Thanks for the reply. One more comment.
>> > > > >
>> > > > > 32. Do we need to add any new metric on the producer? For
>> example, if
>> > > > > partitioner.availability.timeout.ms is > 0, it might be useful to
>> > know
>> > > > the
>> > > > > number of unavailable partitions.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > > On Thu, Mar 10, 2022 at 12:46 PM Artem Livshits
>> > > > >  wrote:
>> > > > >
>> > > > > > Hi Jun,
>> > > > > >
>> > > > > > 30.  Clarified.
>> > > > > >
>> > > > > > 31. I plan to do some benchmarking once implementation is
>> finished,
>> > > > I'll
>> > > > > > update the KIP with the results once I have them.  The reason to
>> > make
>> > > > it
>> > > > > > default is that it won't be used otherwise and we won't know if
>> > it's
>> > > > good
>> > > > > > or not in practical workloads.
>> > > > > >
>> > > > > > -Artem
>> > > > > >
>> > > > > > On Thu, Mar 10, 2022 at 11:42 AM Jun Rao
>> > > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi, Artem,
>> > > > > > >
>> > > > > > > Thanks for the updated KIP. A couple of more comments.
>> > > > > > >
>> > > > > > > 30. For the 3 new configs, it would be useful to make it clear
>> > that
>> > > > > they
>> > > > > > > are only relevant when the partitioner class is null.
>> > > > > > >
>> > > > > > > 31. partitioner.adaptive.partitioning.enable : I am wondering
>> > > whether
>> > > > > it
>> > > > > > > should default to true. This is a more complex behavior than
>> > > "uniform
>> > > > > > > sticky" and may take some time to get right. If we do want to
>> > > enable
>> > > > it
>> > > > > > by
>> > > > > > > default, it would be useful to validate it with some test
>> > results.
>> > > > > > >
>> > > > > > > Jun
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Mar 9, 2022 at 10:05 PM Artem Livshits
>> > > > > > >  wrote:
>> > > > > > >
>> > > > > > > > Thank you for feedback, I've discussed this offline with
>> some
>> > of
>> > > > the
>> > > > > > > folks
>> > > > > > > > and updated the KIP.  The main change is that now instead of
>> > > using
>> > > > > > > > DefaultPartitioner and UniformStickyPartitioners as flags,
>> in
>> > the
>> > > > new
>> > > > > > > > proposal the default partitioner is null, so if no custom
>> > > > partitioner
>> > > > > > is
>> > > > > > > > specified then the partitioning logic is implemented in
>> > > > > KafkaProducer.
>> > > > > > > > Compatibility section is updated as well.  Also the
>> > configuration
>> > > > > > options
>> > > > > > > > are renamed to be more consistent.
>> > > > > > > >
>> > > > > > > > -Artem
>> > > > > > > >
>> > > > > > > > On Fri, Mar 4, 2022 at 10:38 PM Luke Chen <
>> show...@gmail.com>
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Artem,
>> > > > > > > > >
>> > > > > > > > > Thanks for your explanation and update to the KIP.
>> > > > > > > > > Some comments:
>> > > > > > > > >
>> > > > > > > > > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #780

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-653: Upgrade log4j to log4j2

[Ismael Juma]

Hi all,

The KIP compatibility section does not include enough detail. I am puzzled
how we voted +1 given that. I noticed that Colin indicated it would only be
acceptable in a major release unless the new version was fully compatible
(which it is not). Can we clarify what we actually voted for here?

Ismael

On Wed, Oct 21, 2020 at 6:41 PM Dongjin Lee  wrote:

> Hi All,
>
> As of present:
>
> - Binding: +3 (Gwen, John, Colin)
> - Non-binding: +1 (David, Tom)
>
> This KIP is now accepted. Thanks for your votes!
>
> @Colin Sure, I have some plan for providing a compatibility preview. Let's
> continue in the discussion thread.
>
> All other voters not in KIP-676 Vote thread: KIP-676 (by Tom) is a
> prerequisite of this KIP. Please have a look at that proposal and vote for
> it.
>
> Best,
> Dongjin
>
> On Wed, Oct 21, 2020 at 9:17 PM Colin McCabe  wrote:
>
> > +1 (binding).  I think we should consider doing this in 3.0 rather than
> > 2.8, though, unless we are really confident that it is 100% compatible.
> >
> > I wasn't able to find much information on how compatible the new API
> > bridge is, but the log4j website does have this:
> >
> > > Basic compatibility with Log4j 1.x is provided through the log4j12-api
> > component,
> > > however it does not implement some of the very implementation specific
> > > classes and methods
> >
> > best,
> > Colin
> >
> >
> > On Fri, Oct 9, 2020, at 02:51, Tom Bentley wrote:
> > > +1 non-binding.
> > >
> > > Thanks for your efforts on this Dongjin.
> > >
> > > Tom
> > >
> > > On Wed, Oct 7, 2020 at 6:45 AM Dongjin Lee  wrote:
> > >
> > > > As of present:
> > > >
> > > > - Binding: +2 (Gwen, John)
> > > > - Non-binding: +1 (David)
> > > >
> > > > Now we need one more binding +1.
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Wed, Oct 7, 2020 at 1:37 AM David Jacot 
> > wrote:
> > > >
> > > > > Thanks for driving this, Dongjin!
> > > > >
> > > > > The KIP looks good to me. I’m +1 (non-binding).
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > Le mar. 6 oct. 2020 à 17:23, Dongjin Lee  a
> > écrit :
> > > > >
> > > > > > As of present:
> > > > > >
> > > > > > - Binding: +2 (Gwen, John)
> > > > > > - Non-binding: 0
> > > > > >
> > > > > > Thanks,
> > > > > > Dongjin
> > > > > >
> > > > > > On Sat, Oct 3, 2020 at 10:51 AM John Roesler <
> vvcep...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks for the KIP, Dongjin!
> > > > > > >
> > > > > > > I’ve just reviewed the KIP document, and it looks good to me.
> > > > > > >
> > > > > > > I’m +1 (binding)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Fri, Oct 2, 2020, at 19:11, Gwen Shapira wrote:
> > > > > > > > +1 (binding)
> > > > > > > >
> > > > > > > > A very welcome update :)
> > > > > > > >
> > > > > > > > On Tue, Sep 22, 2020 at 9:09 AM Dongjin Lee <
> > dong...@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi devs,
> > > > > > > > >
> > > > > > > > > Here I open the vote for KIP-653: Upgrade log4j to log4j2.
> It
> > > > > > replaces
> > > > > > > the
> > > > > > > > > obsolete log4j logging library into the current standard,
> > log4j2,
> > > > > > with
> > > > > > > > > maintaining backward-compatibility.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dongjin
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Dongjin Lee*
> > > > > > > > >
> > > > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *github:  github.com/dongjinleekr
> > > > > > > > > keybase:
> > > > > > > https://keybase.io/dongjinleekr
> > > > > > > > > linkedin:
> > > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > > > speakerdeck:
> > > > > > > speakerdeck.com/dongjin
> > > > > > > > > *
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Gwen Shapira
> > > > > > > > Engineering Manager | Confluent
> > > > > > > > 650.450.2760 | @gwenshap
> > > > > > > > Follow us: Twitter | blog
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *github:  github.com/dongjinleekr
> > > > > > keybase:
> > > > > https://keybase.io/dongjinleekr
> > > > > > linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > speakerdeck:
> > > > > > speakerdeck.com/dongjin
> > > > > > *
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > >
> > > >
> > > >
> 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #189

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #779

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #89

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 503725 lines...]
[2022-03-17T19:03:43.567Z] > Task :core:classes UP-TO-DATE
[2022-03-17T19:03:43.567Z] > Task :core:compileTestJava NO-SOURCE
[2022-03-17T19:03:43.567Z] 
[2022-03-17T19:03:43.567Z] > Task :streams:processMessages
[2022-03-17T19:03:43.567Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2022-03-17T19:03:43.567Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-03-17T19:03:43.567Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2022-03-17T19:03:43.567Z] 
[2022-03-17T19:03:43.567Z] > Task :streams:compileJava UP-TO-DATE
[2022-03-17T19:03:43.567Z] > Task :streams:classes UP-TO-DATE
[2022-03-17T19:03:43.567Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2022-03-17T19:03:44.492Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2022-03-17T19:03:44.492Z] > Task :streams:jar UP-TO-DATE
[2022-03-17T19:03:44.492Z] > Task :core:compileTestScala UP-TO-DATE
[2022-03-17T19:03:44.492Z] > Task :core:testClasses UP-TO-DATE
[2022-03-17T19:03:44.492Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2022-03-17T19:03:50.189Z] > Task :connect:api:javadoc
[2022-03-17T19:03:50.189Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-03-17T19:03:50.189Z] > Task :connect:api:jar UP-TO-DATE
[2022-03-17T19:03:50.189Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-03-17T19:03:50.189Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-03-17T19:03:50.189Z] > Task :connect:json:jar UP-TO-DATE
[2022-03-17T19:03:50.189Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-03-17T19:03:50.189Z] > Task :connect:api:javadocJar
[2022-03-17T19:03:50.189Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-03-17T19:03:50.189Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-03-17T19:03:50.189Z] > Task :connect:api:testJar
[2022-03-17T19:03:50.189Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-03-17T19:03:50.189Z] > Task :connect:json:publishToMavenLocal
[2022-03-17T19:03:50.189Z] > Task :connect:api:testSrcJar
[2022-03-17T19:03:50.189Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-03-17T19:03:50.189Z] > Task :connect:api:publishToMavenLocal
[2022-03-17T19:03:55.345Z] > Task :streams:javadoc
[2022-03-17T19:03:56.477Z] > Task :streams:javadocJar
[2022-03-17T19:03:56.477Z] > Task :streams:compileTestJava UP-TO-DATE
[2022-03-17T19:03:56.477Z] > Task :streams:processTestResources UP-TO-DATE
[2022-03-17T19:03:56.477Z] > Task :streams:testClasses UP-TO-DATE
[2022-03-17T19:03:56.477Z] > Task :streams:testJar
[2022-03-17T19:03:57.306Z] 
[2022-03-17T19:03:57.306Z] > Task :clients:javadoc
[2022-03-17T19:03:57.306Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-03-17T19:03:57.306Z] 
[2022-03-17T19:03:57.306Z] > Task :streams:testSrcJar
[2022-03-17T19:03:57.306Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-03-17T19:03:57.306Z] > Task :streams:publishToMavenLocal
[2022-03-17T19:03:58.357Z] 
[2022-03-17T19:03:58.357Z] > Task :clients:javadoc
[2022-03-17T19:03:58.357Z] 1 warning
[2022-03-17T19:03:58.357Z] 
[2022-03-17T19:03:58.357Z] > Task :clients:javadocJar
[2022-03-17T19:03:59.438Z] 
[2022-03-17T19:03:59.438Z] > Task :clients:srcJar
[2022-03-17T19:03:59.438Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-03-17T19:03:59.438Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-03-17T19:03:59.438Z] 
[2022-03-17T19:03:59.438Z] > Task :clients:testJar
[2022-03-17T19:04:00.363Z] > Task :clients:testSrcJar

[jira] [Created] (KAFKA-13754) Follower should reject Fetch request while the leader is recovering

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-13754:
--

 Summary: Follower should reject Fetch request while the leader is 
recovering
 Key: KAFKA-13754
 URL: https://issues.apache.org/jira/browse/KAFKA-13754
 Project: Kafka
  Issue Type: Task
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio


In the PR for KIP-704 we removed leader recovery state validation from the 
FETCH. This is okay because the leader immediately recovers the partition.

We should enable this validation before implementing log recovery from unclean 
leader election.

The old implementation and test is in this commit: 
https://github.com/apache/kafka/pull/11733/commits/c7e54b8f6cef087deac119d61a46d3586ead72b9



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13753) Log cleaner should transaction metadata in index until corresponding marker is removed

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-13753:
---

 Summary: Log cleaner should transaction metadata in index until 
corresponding marker is removed
 Key: KAFKA-13753
 URL: https://issues.apache.org/jira/browse/KAFKA-13753
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Currently the log cleaner will remove aborted transactions from the index as 
soon as it detects that the data from the transaction is gone. It does not wait 
until the corresponding marker has also been removed. Although it is extremely 
unlikely, it seems possible today that a Fetch might fail to return the aborted 
transaction metadata correctly if a log cleaning occurs concurrently. This is 
because the collection of aborted transactions is only done after the reading 
data from the log. It would be safer to preserve the aborted transaction 
metadata in the index until the marker is also removed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] KIP-820: Extend KStream process with new Processor API

[Guozhang Wang]

Reviewed the updated docs, and recasting my +1 vote again, thanks Jorge for
pushing it through!

Guozhang

On Tue, Mar 15, 2022 at 4:28 PM Matthias J. Sax  wrote:

> +1 (binding)
>
> Thanks for pushing this through. Was a difficult discussion!
>
>
> -Matthias
>
> On 3/15/22 10:01 AM, John Roesler wrote:
> > Thanks for the update, Jorge!
> >
> > I’m still +1 (binding)
> >
> > Thanks,
> > John
> >
> > On Thu, Feb 17, 2022, at 12:57, Guozhang Wang wrote:
> >> Thanks Jorge, overall looks good to me.
> >>
> >> Maybe we can clarify a bit in the wiki that the reason we have to not
> >> include the additional `final String... stateStoreNames` params in the
> new
> >> `process` API is that we need to have overloaded functions which takes
> >> `ProcessorSupplier<...> ` where the output types are not `Void`, but
> due to
> >> type eraser we cannot distinguish the new overloaded function signatures
> >> with the old ones if they also include `final String...
> stateStoreNames`.
> >> And in javadocs explains that if users want to connect state stores to
> this
> >> processor, they could use the `connectState` API instead.
> >>
> >> Otherwise, I'm +1.
> >>
> >> Guozhang
> >>
> >> On Tue, Feb 15, 2022 at 11:54 AM John Roesler 
> wrote:
> >>
> >>> Thanks, Jorge!
> >>>
> >>> I'm +1 (binding)
> >>>
> >>> -John
> >>>
> >>> On Tue, 2022-02-15 at 19:16 +, Jorge Esteban Quilcate
> >>> Otoya wrote:
>  Hi all,
> 
>  I'd like to start a vote on KIP-820 which proposes extending KStream
> to
> >>> use
>  the new Processor API
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-820%3A+Extend+KStream+process+with+new+Processor+API
> 
>  Thanks,
>  Jorge
> >>>
> >>>
> >>
> >> --
> >> -- Guozhang
>


-- 
-- Guozhang

[jira] [Resolved] (KAFKA-13750) Client Compatability KafkaTest uses invalid idempotency configs

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13750.
-
Fix Version/s: 3.2.0
   3.1.1
   3.0.2
   Resolution: Fixed

> Client Compatability KafkaTest uses invalid idempotency configs
> ---
>
> Key: KAFKA-13750
> URL: https://issues.apache.org/jira/browse/KAFKA-13750
> Project: Kafka
>  Issue Type: Bug
>Reporter: Justine Olshan
>Assignee: Justine Olshan
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.2
>
>
> With the switch to idempotency as a default, some of our tests broke 
> including 
> ClientCompatibilityFeaturesTest.run_compatibility_test for versions prior to 
> 0.11 where EOS was enabled. We need to configure the producer correctly for 
> these earlier versions.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13509) Support max timestamp in GetOffsetShell

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13509.
-
  Reviewer: David Jacot
Resolution: Fixed

> Support max timestamp in GetOffsetShell
> ---
>
> Key: KAFKA-13509
> URL: https://issues.apache.org/jira/browse/KAFKA-13509
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: dengziming
>Assignee: dengziming
>Priority: Major
> Fix For: 3.2.0
>
>
> We would list offset with max timestamp using `kafka.tools.GetOffsetShell` :
> ```
> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server 
> localhost:9092 --topic topic1 --time -3
> ```
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13136) kafka-connect task.max : active task in consumer group is limited by the bigger topic to consume

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13136.
---
Resolution: Fixed

> kafka-connect task.max : active task in consumer group is limited by the 
> bigger topic to consume
> 
>
> Key: KAFKA-13136
> URL: https://issues.apache.org/jira/browse/KAFKA-13136
> Project: Kafka
>  Issue Type: Bug
>Reporter: raphaelauv
>Priority: Major
>
> In kafka-connect 2.7
> *The maximum number of active task for a sink connector is equal to the topic 
> with the biggest number of partitions to consume*
> An active task is a task with partitions attributed in the consumer-group of 
> the sink connector
> example :
> With 2 topics where each have 10 partitions ( 20 partitions in total )
> The maximum number of active task is 10 ( if I set task.max at 12 ,there is 
> 10 members of the consumer group consuming partitions and  2 members in the 
> consumer-group that do not have partitions to consume).
> If I add a third topic with 15 partitions to the connector conf then the 12 
> members of the consumer group are consuming partitions, and then if I set now 
> task.max at 17 only 15 members are active in the consumer-group.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

[Jorge Esteban Quilcate Otoya]

Thanks, Matthias. KIP is updated with the APIs.

On Tue, 15 Mar 2022 at 23:28, Matthias J. Sax  wrote:

> Thanks.
>
> It would be good to add the concrete interfaces of the new classed to
> the KIP, ie,
>
>   - FixedKeyProcessorSupplier
>   - FixedKeyProcessor
>   - FixedKeyProcessorContext
>   - FixedKeyRecord
>
>
> -Matthias
>
>
> On 3/10/22 3:15 PM, Jorge Esteban Quilcate Otoya wrote:
> > Thanks all!
> >
> > I agree with Matthias and Jon on going forward with the new
> > `FixedKeyRecord` approach.
> > The KIP has been updated accordingly.
> >
> > Feel free to add your vote or amend on the vote thread if needed.
> >
> > Cheers,
> >
> > On Mon, 7 Mar 2022 at 21:57, Matthias J. Sax  wrote:
> >
> >> I think it's ok that we cannot prevent users from mutating a given
> >> read-only object. We have similar issues "all over the place" in the
> >> API, because it's just how Java works unfortunately (eg,
> >> `ValueMapperWithKey` and similar interfaces).
> >>
> >> The point being is, that the API clearly expresses that the key should
> >> not be changes, as `FixedKeyRecord` as not `withKey()` method, what is
> >> much better then having `Record.withKey()` and thus incorrectly
> >> indicating to user that it would be ok to set a new key.
> >>
> >> I think it's worth to add the new interfaces.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 3/7/22 11:46 AM, Guozhang Wang wrote:
> >>> Thanks John! I feel a bit ashamed of just thinking loud here without
> >> trying
> >>> out prototypes myself :P
> >>>
> >>> I think the FixedKeyProcessor/Record looks very good -- like you said,
> >>> since we are making a new set of APIs then why don't we reconsider more
> >>> bolderly -- but at the same time I'd also like to make sure we agree on
> >> how
> >>> much "safety" we can achieve in runtime: even with the proposed APIs,
> we
> >>> cannot prevent users doing something like:
> >>>
> >>> ---
> >>> process(FixedKeyRecord inputRecord) {
> >>>   inputRecord.key().modifyField(...); // this is not preventable
> with
> >>> runtime key validations either since we just check the key object
> itself
> >> is
> >>> not replaced
> >>>   context.forward(inputRecord);
> >>> }
> >>>
> >>> ---
> >>>
> >>> I.e. in either type-safety or runtime validation, we cannot be 100%
> safe
> >>> that users would not do anything wrong. This drives me to think, how
> much
> >>> we'd like to pay to "remind" (instead of say "enforce", since we cannot
> >>> really do it) users the semantics of "processValue". Personally I felt
> >> that
> >>> adding the new set of APIs for that purpose only is a bit overkill, and
> >>> hence was leaning towards just the runtime validation. But I admit this
> >> is
> >>> purely subjective so I'm willing to yield to the group if others feel
> >> it's
> >>> worthy to do so.
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>>
> >>> On Mon, Mar 7, 2022 at 10:32 AM Jorge Esteban Quilcate Otoya <
> >>> quilcate.jo...@gmail.com> wrote:
> >>>
>  Thanks, John!
>  This looks very promising.
> 
>  I will familiarize this approach and update the KIP accordingly. From
> >> what
>  I can see so far, this should cover most of the open issues in this
>  proposal.
> 
>  PS.
> 
> > Just as a reminder, the current approach with transformers
> > is NOT enforced at compile time. Transformers have access to
> > a "forwarding disabled" processor context, which still has
> > the forward methods that throw a runtime exception when
> > invoked.
> 
>  Agree. I was referring to the value transformers where `readOnlyKey`
> is
>  passed but not forwarded internally. Though about the "forwarding
> >> disabled"
>  approach, you're totally right that is a runtime validation.
>  Regardless, the approach proposed here will be a much better one.
> 
> 
>  On Sun, 6 Mar 2022 at 18:59, John Roesler 
> wrote:
> 
> > Hello all,
> >
> > It seems like we're making good progress on this discussion.
> > If I'm keeping track correctly, if we can resolve this
> > question about how to handle processValues(), then we should
> > be able to finalize the vote, right?
> >
> > I share Matthias's preference for having a type-safe API.
> >
> > Just as a reminder, the current approach with transformers
> > is NOT enforced at compile time. Transformers have access to
> > a "forwarding disabled" processor context, which still has
> > the forward methods that throw a runtime exception when
> > invoked.
> >
> > However, the spirit of the "new processor api" line of work
> > is to clean up a lot of the cruft around the original
> > processor API, so this is a good opportunity to introduce a
> > type-safe version if we can.
> >
> > Based on my experience adding the new processor API, I felt
> > like it should be possible to do what he suggests, but it
> > would be more involved than 

[jira] [Created] (KAFKA-13752) Using `equals` instead of `==` when Uuid compare in Java

[Xiaobing Fang (Jira)]

Xiaobing Fang created KAFKA-13752:
-

 Summary: Using `equals` instead of `==` when Uuid compare in Java
 Key: KAFKA-13752
 URL: https://issues.apache.org/jira/browse/KAFKA-13752
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Xiaobing Fang


`Uuid.ZERO_UUID == new Uuid(0L, 0L)` is true in scala, but in java is false.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] KIP-660: Pluggable ReplicaPlacer

[Mickael Maison]

Hi,

I'd like to start a new discussion on KIP-660. I originally wrote this
KIP in 2020 and the initial discussion
(https://lists.apache.org/thread/xn7xyb74nyt281brto4x28r9rzxm4lp9)
raised some concerns especially around KRaft (which did not exist at
that time) and scalability.

Since then, we got a new KRaft controller so I've been able to revisit
this KIP. I kept the KIP number as it's essentially the same idea, but
the proposal is significantly different:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-660%3A+Pluggable+ReplicaPlacer

Please take a look and let me know if you have any feedback.

Thanks,
Mickael