Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2583

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 2765 lines...]
one error found

> Task :core:compileTestScala FAILED
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/KafkaMetricReporterClusterIdTest.scala:23:21:
 imported `QuorumTestHarness` is permanently hidden by definition of object 
QuorumTestHarness in package server
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala:32:21:
 imported `QuorumTestHarness` is permanently hidden by definition of object 
QuorumTestHarness in package server
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/LogRecoveryTest.scala:25:21:
 imported `QuorumTestHarness` is permanently hidden by definition of object 
QuorumTestHarness in package server
[Error] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala:6178:23:
 value latest is not a member of object 
org.apache.kafka.server.common.MetadataVersion
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/AdminClientWithPoliciesIntegrationTest.scala:107:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/AuthorizerIntegrationTest.scala:1156:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/AuthorizerIntegrationTest.scala:1193:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/AuthorizerIntegrationTest.scala:1226:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/ConsumerBounceTest.scala:391:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/PlaintextAdminIntegrationTest.scala:2437:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/PlaintextAdminIntegrationTest.scala:2618:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/PlaintextAdminIntegrationTest.scala:2687:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/TransactionsBounceTest.scala:76:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/api/TransactionsTest.scala:305:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala:622:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala:1512:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala:1539:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala:1548:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala:717:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala:342:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/log/LogConfigTest.scala:405:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/log/OffsetIndexTest.scala:52:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:201:4:
 @nowarn annotation does not suppress any warnings
[Warn] 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #68

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #138

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2582

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 2764 lines...]
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala:1539:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/DynamicBrokerReconfigurationTest.scala:1548:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala:717:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala:342:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/log/LogConfigTest.scala:405:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/log/OffsetIndexTest.scala:52:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:201:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:680:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala:459:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:591:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:738:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:765:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/server/LogDirFailureTest.scala:74:4:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/tools/ConsoleProducerTest.scala:263:2:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/tools/MirrorMakerTest.scala:29:2:
 @nowarn annotation does not suppress any warnings
[Warn] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/test/scala/unit/kafka/utils/TestUtils.scala:409:4:
 @nowarn annotation does not suppress any warnings
35 warnings found
one error found

> Task :core:compileTestScala FAILED

> Task :core:compileScala
Unexpected javac output: warning: [options] bootstrap class path not set in 
conjunction with -source 8
warning: [options] source value 8 is obsolete and will be removed in a future 
release
warning: [options] target value 8 is obsolete and will be removed in a future 
release
warning: [options] To suppress warnings about obsolete options, use 
-Xlint:-options.
/home/jenkins/workspace/Kafka_kafka_trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java:235:
 warning: [removal] AccessController in java.security has been deprecated and 
marked for removal
return java.security.AccessController.doPrivileged(new 
PrivilegedAction() {
^
/home/jenkins/workspace/Kafka_kafka_trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java:257:
 warning: [removal] AccessController in java.security has been deprecated and 
marked for removal
return java.security.AccessController.doPrivileged(new 
PrivilegedAction() {
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: 
/home/jenkins/workspace/Kafka_kafka_trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java
 uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
6 warnings.

> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :shell:compileJava
> Task :shell:classes
> Task :core:checkstyleMain
> Task :shell:checkstyleMain
> Task :shell:spotbugsMain

> Task :core:compileScala
[Warn] /home/jenkins/.gradle/workers/warning:[options] bootstrap class path not 
set in conjunction with -source 8

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2581

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #67

[Apache Jenkins Server]

See 

Re: [VOTE] 3.7.0 RC2

[Matthias J. Sax]

Stan, thanks for driving this all forward! Excellent job.

About


StreamsStandbyTask - https://issues.apache.org/jira/browse/KAFKA-16141
StreamsUpgradeTest - https://issues.apache.org/jira/browse/KAFKA-16139


For `StreamsUpgradeTest` it was a test setup issue and should be fixed 
now in trunk and 3.7 (and actually also in 3.6...)


For `StreamsStandbyTask` the failing test exposes a regression bug, so 
it's a blocker. I updated the ticket accordingly. We already have an 
open PR that reverts the code introducing the regression.



-Matthias

On 1/17/24 9:44 AM, Proven Provenzano wrote:

We have another blocking issue for the RC :
https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar to
https://issues.apache.org/jira/browse/KAFKA-14616. The new issue however
can lead to the new topic having partitions that a producer cannot write to.

--Proven

On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano 
wrote:



I have a PR https://github.com/apache/kafka/pull/15197 for
https://issues.apache.org/jira/browse/KAFKA-16131 that is building now.
--Proven

On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz  wrote:


*> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
blocker bug because it *
*> will generate huge amount of logspam. I guess we didn't find it in
junit
tests *
*> since logspam doesn't fail the automated tests. But certainly it's not
suitable *
*> for production. Did you file a JIRA yet?*

Hi Colin,

I opened https://issues.apache.org/jira/browse/KAFKA-16131.

Thanks & Regards
Jakub

On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe  wrote:


Hi Stanislav,

Thanks for making the first RC. The fact that it's titled RC2 is messing
with my mind a bit. I hope this doesn't make people think that we're
farther along than we are, heh.

On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:

*> Nice catch! It does seem like we should have gated this behind the
metadata> version as KIP-858 implies. Is the cluster configured with
multiple log> dirs? What is the impact of the error messages?*

I did not observe any obvious impact. I was able to send and receive
messages as normally. But to be honest, I have no idea what else
this might impact, so I did not try anything special.

I think everyone upgrading an existing KRaft cluster will go through

this

stage (running Kafka 3.7 with an older metadata version for at least a
while). So even if it is just a logged exception without any other

impact I

wonder if it might scare users from upgrading. But I leave it to

others

to

decide if this is a blocker or not.



Hi Jakub,

Thanks for trying the RC. I think what you found is a blocker bug

because

it will generate huge amount of logspam. I guess we didn't find it in

junit

tests since logspam doesn't fail the automated tests. But certainly it's
not suitable for production. Did you file a JIRA yet?


On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
 wrote:


Hey Luke,

This is an interesting problem. Given the fact that the KIP for

having a

3.8 release passed, I think it weights the scale towards not calling

this a

blocker and expecting it to be solved in 3.7.1.

It is unfortunate that it would not seem safe to migrate to KRaft in

3.7.0

(given the inability to rollback safely), but if that's true - the

same

case would apply for 3.6.0. So in any case users w\ould be expected

to

use a

patch release for this.


Hi Luke,

Thanks for testing rollback. I think this is a case where the
documentation is wrong. The intention was to for the steps to basically

be:


1. roll all the brokers into zk mode, but with migration enabled
2. take down the kraft quorum
3. rmr /controller, allowing a hybrid broker to take over.
4. roll all the brokers into zk mode without migration enabled (if

desired)


With these steps, there isn't really unavailability since a ZK

controller

can be elected quickly after the kraft quorum is gone.


Further, since we will have a 3.8 release - it is
likely we will ultimately recommend users upgrade from that version

given

its aim is to have strategic KRaft feature parity with ZK.
That being said, I am not 100% on this. Let me know whether you think

this

should block the release, Luke. I am also tagging Colin and David to

weigh

in with their opinions, as they worked on the migration logic.


The rollback docs are new in 3.7 so the fact that they're wrong is a

clear

blocker, I think. But easy to fix, I believe. I will create a PR.

best,
Colin



Hey Kirk and Chris,

Unless I'm missing something - KAFKALESS-16029 is simply a bad log

due

to

improper closing. And the PR description implies this has been

present

since 3.5. While annoying, I don't see a strong reason for this to

block

the release.

Hey Jakub,

Nice catch! It does seem like we should have gated this behind the

metadata

version as KIP-858 implies. Is the cluster configured with multiple

log

dirs? What is the impact of the error messages?

Tagging Igor (the author of the KIP) to weigh 

[jira] [Resolved] (KAFKA-16078) Be more consistent about getting the latest MetadataVersion

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-16078.
--
Fix Version/s: 3.7.0
 Reviewer: Colin Patrick McCabe
   Resolution: Fixed

> Be more consistent about getting the latest MetadataVersion
> ---
>
> Key: KAFKA-16078
> URL: https://issues.apache.org/jira/browse/KAFKA-16078
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.7.0
>
>
> The InterBrokerProtocolVersion currently defaults to a non-production 
> MetadataVersion. We should be more consistent about getting the latest 
> MetadataVersion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16161) Avoid creating remote log metadata snapshot file in partition data directory.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-16161:
--

 Summary: Avoid creating remote log metadata snapshot file in 
partition data directory.
 Key: KAFKA-16161
 URL: https://issues.apache.org/jira/browse/KAFKA-16161
 Project: Kafka
  Issue Type: Improvement
Reporter: Satish Duggana


Avoid creating remote log metadata snapshot file in a partition data directory. 
This can be added when the snapshots implementation related functionality is 
enabled end to end. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16160) AsyncKafkaConsumer is trying to connect to a disconnected note in a tight loop

[Philip Nee (Jira)]

Philip Nee created KAFKA-16160:
--

 Summary: AsyncKafkaConsumer is trying to connect to a disconnected 
note in a tight loop
 Key: KAFKA-16160
 URL: https://issues.apache.org/jira/browse/KAFKA-16160
 Project: Kafka
  Issue Type: Task
  Components: consumer
Reporter: Philip Nee


Observing some excessive logging running AsyncKafkaConsumer and observing 
excessive logging of :
{code:java}
1271 [2024-01-15 09:43:36,627] DEBUG [Consumer clientId=console-consumer, 
groupId=concurrent_consumer] Node is not ready, handle the request in the next 
event loop: node=worker4:9092 (id: 2147483644 rack: null), 
request=UnsentRequest{requestBuil     
der=ConsumerGroupHeartbeatRequestData(groupId='concurrent_consumer', 
memberId='laIqS789StuhXFpTwjh6hA', memberEpoch=1, instanceId=null, rackId=null, 
rebalanceTimeoutMs=30, subscribedTopicNames=[output-topic], 
serverAssignor=null, topicP     
artitions=[TopicPartitions(topicId=I5P5lIXvR1Cjc8hfoJg5bg, partitions=[0])]), 
handler=org.apache.kafka.clients.consumer.internals.NetworkClientDelegate$FutureCompletionHandler@918925b,
 node=Optional[worker4:9092 (id: 2147483644 rack: null)]     , 
timer=org.apache.kafka.common.utils.Timer@55ed4733} 
(org.apache.kafka.clients.consumer.internals.NetworkClientDelegate) {code}
This seems to be triggered by a tight poll loop of the network thread.  The 
right thing to do is to backoff a bit for that given node and retry later.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16131) Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-16131.
--
Resolution: Fixed

> Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 
> KRaft cluster with metadata version 3.6
> 
>
> Key: KAFKA-16131
> URL: https://issues.apache.org/jira/browse/KAFKA-16131
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Jakub Scholz
>Assignee: Proven Provenzano
>Priority: Blocker
> Fix For: 3.7.0
>
>
> When running Kafka 3.7.0-RC2 as a KRaft cluster with metadata version set to 
> 3.6-IV2 metadata version, it throws repeated errors like this in the 
> controller logs:
> {quote}2024-01-13 16:58:01,197 INFO [QuorumController id=0] 
> assignReplicasToDirs: event failed with UnsupportedVersionException in 15 
> microseconds. (org.apache.kafka.controller.QuorumController) 
> [quorum-controller-0-event-handler]
> 2024-01-13 16:58:01,197 ERROR [ControllerApis nodeId=0] Unexpected error 
> handling request RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS, apiVersion=0, 
> clientId=1000, correlationId=14, headerVersion=2) – 
> AssignReplicasToDirsRequestData(brokerId=1000, brokerEpoch=5, 
> directories=[DirectoryData(id=w_uxN7pwQ6eXSMrOKceYIQ, 
> topics=[TopicData(topicId=bvAKLSwmR7iJoKv2yZgygQ, 
> partitions=[PartitionData(partitionIndex=2), 
> PartitionData(partitionIndex=1)]), TopicData(topicId=uNe7f5VrQgO0zST6yH1jDQ, 
> partitions=[PartitionData(partitionIndex=0)])])]) with context 
> RequestContext(header=RequestHeader(apiKey=ASSIGN_REPLICAS_TO_DIRS, 
> apiVersion=0, clientId=1000, correlationId=14, headerVersion=2), 
> connectionId='172.16.14.219:9090-172.16.14.217:53590-7', 
> clientAddress=/[172.16.14.217|http://172.16.14.217/], 
> principal=User:CN=my-cluster-kafka,O=io.strimzi, 
> listenerName=ListenerName(CONTROLPLANE-9090), securityProtocol=SSL, 
> clientInformation=ClientInformation(softwareName=apache-kafka-java, 
> softwareVersion=3.7.0), fromPrivilegedListener=false, 
> principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@71004ad2])
>  (kafka.server.ControllerApis) [quorum-controller-0-event-handler]
> java.util.concurrent.CompletionException: 
> org.apache.kafka.common.errors.UnsupportedVersionException: Directory 
> assignment is not supported yet.
> at 
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
>  at 
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
>  at 
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
>  at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
>  at 
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
>  at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:880)
>  at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:871)
>  at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>  at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>  at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>  at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>  at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: 
> Directory assignment is not supported yet.
> {quote}
>  
> With the metadata version set to 3.6-IV2, it makes sense that the request is 
> not supported. But the request should in such case not be sent at all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16159) Prune excessive logging from Telemetry Reporter

[Philip Nee (Jira)]

Philip Nee created KAFKA-16159:
--

 Summary: Prune excessive logging from Telemetry Reporter
 Key: KAFKA-16159
 URL: https://issues.apache.org/jira/browse/KAFKA-16159
 Project: Kafka
  Issue Type: Task
  Components: consumer, log
Reporter: Philip Nee
Assignee: Apoorv Mittal


While running system tests locally, I've noticed excessive logging of the 
Telemtry Reporter.  This I believe was introduced in KIP-714.
{code:java}
[2024-01-15 09:44:16,911] DEBUG For telemetry state SUBSCRIPTION_NEEDED, 
returning the value 224678 ms; the client will wait before submitting the next 
GetTelemetrySubscriptions network API request 
(org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter) {code}
This is logged several times per ms - Also, given the amount of log being 
emitted, can we also check the CPU profile to see if there's a process running 
a tight loop?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Raman Verma]

The KIP is accepted with 3 binding votes (Jun, Justine and Jason).
Thank you all !

On Wed, Jan 17, 2024 at 1:49 PM Raman Verma  wrote:
>
> Thanks Jason,
> I have added a public constructor to TransactionDescription class and
> updated the KIP.
>
> On Thu, Jan 11, 2024 at 9:33 AM Jason Gustafson
>  wrote:
> >
> > HI Raman,
> >
> > Thanks for the KIP! +1 from me.
> >
> > One small thing: we will probably have to overload the constructor for
> > TransactionDescription in order to add the new update time field to avoid
> > breaking the API. We might consider whether we need the overload to be
> > public or not.
> >
> > Best,
> > Jason
> >
> > On Tue, Jan 9, 2024 at 10:41 AM Justine Olshan 
> > 
> > wrote:
> >
> > > Thanks Raman.
> > >
> > > +1 (binding) from me as well.
> > >
> > > Justine
> > >
> > > On Tue, Jan 9, 2024 at 10:12 AM Jun Rao  wrote:
> > >
> > > > Hi, Raman,
> > > >
> > > > Thanks for the KIP. +1 from me.
> > > >
> > > > Jun
> > > >
> > > > On Tue, Dec 26, 2023 at 11:32 AM Raman Verma 
> > > >  > > >
> > > > wrote:
> > > >
> > > > > I would like to start a Vote on KIP-994
> > > > >
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs
> > > > >
> > > >
> > >

Re: [VOTE] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Raman Verma]

Thanks Jason,
I have added a public constructor to TransactionDescription class and
updated the KIP.

On Thu, Jan 11, 2024 at 9:33 AM Jason Gustafson
 wrote:
>
> HI Raman,
>
> Thanks for the KIP! +1 from me.
>
> One small thing: we will probably have to overload the constructor for
> TransactionDescription in order to add the new update time field to avoid
> breaking the API. We might consider whether we need the overload to be
> public or not.
>
> Best,
> Jason
>
> On Tue, Jan 9, 2024 at 10:41 AM Justine Olshan 
> wrote:
>
> > Thanks Raman.
> >
> > +1 (binding) from me as well.
> >
> > Justine
> >
> > On Tue, Jan 9, 2024 at 10:12 AM Jun Rao  wrote:
> >
> > > Hi, Raman,
> > >
> > > Thanks for the KIP. +1 from me.
> > >
> > > Jun
> > >
> > > On Tue, Dec 26, 2023 at 11:32 AM Raman Verma  > >
> > > wrote:
> > >
> > > > I would like to start a Vote on KIP-994
> > > >
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs
> > > >
> > >
> >

[jira] [Resolved] (KAFKA-16139) StreamsUpgradeTest fails consistently in 3.7.0

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-16139.
-
Fix Version/s: 3.7.0
   3.6.1
   Resolution: Fixed

> StreamsUpgradeTest fails consistently in 3.7.0
> --
>
> Key: KAFKA-16139
> URL: https://issues.apache.org/jira/browse/KAFKA-16139
> Project: Kafka
>  Issue Type: Test
>Affects Versions: 3.7.0
>Reporter: Stanislav Kozlovski
>Assignee: Bruno Cadonna
>Priority: Major
> Fix For: 3.7.0, 3.6.1
>
>
> h1. 
> kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest#test_rolling_upgrade_with_2_bouncesArguments:\{
>  “from_version”: “3.5.1”, “to_version”: “3.7.0-SNAPSHOT”}
>  
> {{TimeoutError('Could not detect Kafka Streams version 3.7.0-SNAPSHOT on 
> ubuntu@worker2')}}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16158) Cleanup usage of `TimestampedBytesStore` interface

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-16158:
---

 Summary: Cleanup usage of `TimestampedBytesStore` interface
 Key: KAFKA-16158
 URL: https://issues.apache.org/jira/browse/KAFKA-16158
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


We added `TimestampedBytesStore` interface many release ago. It's purpose is to 
indicate if a byte-store's binary value contains a "plain value" or a 
"" format. Stores with "" format should implement the 
interface, however not all stores which this format do.

We tried to fix one occurrence via 
https://issues.apache.org/jira/browse/KAFKA-15629 by adding 
`TimestampedBytesStore` to `KeyValueToTimestampedKeyValueByteStoreAdapter`, 
whoever this change broke the restore code path (cf 
https://issues.apache.org/jira/browse/KAFKA-16141) and thus we reverted the 
change.

During the investigation, we also notices that 
`InMemoryTimestampedKeyValueStoreMarker` implements `TimestampedBytesStore` but 
does not do a byte-array translation (it's unclear why no byte array 
translation happens) – and it's also unclear if in-memory store is testes 
properly.

We should try to clean this all up, adding `TimestampedBytesStore` to 
`KeyValueToTimestampedKeyValueByteStoreAdapter` and figure out how avoid 
breaking the restore code path. In addition, we should verify if 
`InMemoryTimestampedKeyValueStoreMarker` is correct or not, and if the restore 
code path (and maybe also IQv2 code path) is tested properly and correct.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16151) Fix PlaintextConsumerTest.testPerPartitionLeadMetricsCleanUpWithSubscribe

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16151.
---
Resolution: Duplicate

> Fix PlaintextConsumerTest.testPerPartitionLeadMetricsCleanUpWithSubscribe
> -
>
> Key: KAFKA-16151
> URL: https://issues.apache.org/jira/browse/KAFKA-16151
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer, unit tests
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848-client-support
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16150) Fix PlaintextConsumerTest.testPerPartitionLagMetricsCleanUpWithSubscribe

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16150.
---
Resolution: Duplicate

> Fix PlaintextConsumerTest.testPerPartitionLagMetricsCleanUpWithSubscribe
> 
>
> Key: KAFKA-16150
> URL: https://issues.apache.org/jira/browse/KAFKA-16150
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer, unit tests
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848-client-support
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan]

Hey Jun,

I'm glad we are getting to convergence on the design. :)

While I understand it seems a little "weird". I'm not sure what the benefit
of writing an extra record to the log.
Is the concern a tool to describe transactions won't work (ie, the complete
state is needed to calculate the time since the transaction completed?)
If we have a reason like this, it is enough to convince me we need such an
extra record. It seems like it would be replacing the record written on
InitProducerId. Is this correct?

Thanks,
Justine

On Tue, Jan 16, 2024 at 5:14 PM Jun Rao  wrote:

> Hi, Justine,
>
> Thanks for the explanation. I understand the intention now. In the overflow
> case, we set the non-tagged field to the old pid (and the max epoch) in the
> prepare marker so that we could correctly write the marker to the data
> partition if the broker downgrades. When writing the complete marker, we
> know the marker has already been written to the data partition. We set the
> non-tagged field to the new pid to avoid InvalidPidMappingException in the
> client if the broker downgrades.
>
> The above seems to work. It's just a bit inconsistent for a prepare marker
> and a complete marker to use different pids in this special case. If we
> downgrade with the complete marker, it seems that we will never be able to
> write the complete marker with the old pid. Not sure if it causes any
> issue, but it seems a bit weird. Instead of writing the complete marker
> with the new pid, could we write two records: a complete marker with the
> old pid followed by a TransactionLogValue with the new pid and an empty
> state? We could make the two records in the same batch so that they will be
> added to the log atomically.
>
> Thanks,
>
> Jun
>
>
> On Fri, Jan 12, 2024 at 5:40 PM Justine Olshan
> 
> wrote:
>
> > (1) the prepare marker is written, but the endTxn response is not
> received
> > by the client when the server downgrades
> > (2)  the prepare marker is written, the endTxn response is received by
> the
> > client when the server downgrades.
> >
> > I think I am still a little confused. In both of these cases, the
> > transaction log has the old producer ID. We don't write the new producer
> ID
> > in the prepare marker's non tagged fields.
> > If the server downgrades now, it would read the records not in tagged
> > fields and the complete marker will also have the old producer ID.
> > (If we had used the new producer ID, we would not have transactional
> > correctness since the producer id doesn't match the transaction and the
> > state would not be correct on the data partition.)
> >
> > In the overflow case, I'd expect the following to happen on the client
> side
> > Case 1  -- we retry EndTxn -- it is the same producer ID and epoch - 1
> this
> > would fence the producer
> > Case 2 -- we don't retry EndTxn and use the new producer id which would
> > result in InvalidPidMappingException
> >
> > Maybe we can have special handling for when a server downgrades. When it
> > reconnects we could get an API version request showing KIP-890 part 2 is
> > not supported. In that case, we can call initProducerId to abort the
> > transaction. (In the overflow case, this correctly gives us a new
> producer
> > ID)
> >
> > I guess the corresponding case would be where the *complete marker *is
> > written but the endTxn is not received by the client and the server
> > downgrades? This would result in the transaction coordinator having the
> new
> > ID and not the old one.  If the client retries, it will receive an
> > InvalidPidMappingException. The InitProducerId scenario above would help
> > here too.
> >
> > To be clear, my compatibility story is meant to support downgrades server
> > side in keeping the transactional correctness. Keeping the client from
> > fencing itself is not the priority.
> >
> > Hope this helps. I can also add text in the KIP about InitProducerId if
> we
> > think that fixes some edge cases.
> >
> > Justine
> >
> > On Fri, Jan 12, 2024 at 4:10 PM Jun Rao 
> wrote:
> >
> > > Hi, Justine,
> > >
> > > Thanks for the reply.
> > >
> > > I agree that we don't need to optimize for fencing during downgrades.
> > > Regarding consistency, there are two possible cases: (1) the prepare
> > marker
> > > is written, but the endTxn response is not received by the client when
> > the
> > > server downgrades; (2)  the prepare marker is written, the endTxn
> > response
> > > is received by the client when the server downgrades. In (1), the
> client
> > > will have the old produce Id and in (2), the client will have the new
> > > produce Id. If we downgrade right after the prepare marker, we can't be
> > > consistent to both (1) and (2) since we can only put one value in the
> > > existing produce Id field. It's also not clear which case is more
> likely.
> > > So we could probably be consistent with either case. By putting the new
> > > producer Id in the prepare marker, we are consistent with case (2) and
> it
> > > also has the 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2580

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-974: Docker Image for GraalVM based Native Kafka Broker

[Krishna Agarwal]

Hi,

The voting process for KIP was temporarily paused due to ongoing
discussions on the KIP.
With the conclusion of these discussions and no further active
conversations, I propose to resume the voting thread.

Regards,
Krishna

On Mon, Nov 20, 2023 at 11:53 AM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> I'd like to call a vote on KIP-974 which aims to publish a docker image
> for GraalVM based Native Kafka Broker.
>
> KIP -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
>
> Discussion thread -
> https://lists.apache.org/thread/98wnx4w92fqj5wymkqlqyjsvzxz277hk
>
> Regards,
> Krishna
>

Re: [VOTE] 3.7.0 RC2

[Proven Provenzano]

We have another blocking issue for the RC :
https://issues.apache.org/jira/browse/KAFKA-16157. This bug is similar to
https://issues.apache.org/jira/browse/KAFKA-14616. The new issue however
can lead to the new topic having partitions that a producer cannot write to.

--Proven

On Tue, Jan 16, 2024 at 12:04 PM Proven Provenzano 
wrote:

>
> I have a PR https://github.com/apache/kafka/pull/15197 for
> https://issues.apache.org/jira/browse/KAFKA-16131 that is building now.
> --Proven
>
> On Mon, Jan 15, 2024 at 5:03 AM Jakub Scholz  wrote:
>
>> *> Hi Jakub,> > Thanks for trying the RC. I think what you found is a
>> blocker bug because it *
>> *> will generate huge amount of logspam. I guess we didn't find it in
>> junit
>> tests *
>> *> since logspam doesn't fail the automated tests. But certainly it's not
>> suitable *
>> *> for production. Did you file a JIRA yet?*
>>
>> Hi Colin,
>>
>> I opened https://issues.apache.org/jira/browse/KAFKA-16131.
>>
>> Thanks & Regards
>> Jakub
>>
>> On Mon, Jan 15, 2024 at 8:57 AM Colin McCabe  wrote:
>>
>> > Hi Stanislav,
>> >
>> > Thanks for making the first RC. The fact that it's titled RC2 is messing
>> > with my mind a bit. I hope this doesn't make people think that we're
>> > farther along than we are, heh.
>> >
>> > On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote:
>> > > *> Nice catch! It does seem like we should have gated this behind the
>> > > metadata> version as KIP-858 implies. Is the cluster configured with
>> > > multiple log> dirs? What is the impact of the error messages?*
>> > >
>> > > I did not observe any obvious impact. I was able to send and receive
>> > > messages as normally. But to be honest, I have no idea what else
>> > > this might impact, so I did not try anything special.
>> > >
>> > > I think everyone upgrading an existing KRaft cluster will go through
>> this
>> > > stage (running Kafka 3.7 with an older metadata version for at least a
>> > > while). So even if it is just a logged exception without any other
>> > impact I
>> > > wonder if it might scare users from upgrading. But I leave it to
>> others
>> > to
>> > > decide if this is a blocker or not.
>> > >
>> >
>> > Hi Jakub,
>> >
>> > Thanks for trying the RC. I think what you found is a blocker bug
>> because
>> > it will generate huge amount of logspam. I guess we didn't find it in
>> junit
>> > tests since logspam doesn't fail the automated tests. But certainly it's
>> > not suitable for production. Did you file a JIRA yet?
>> >
>> > > On Sun, Jan 14, 2024 at 10:17 PM Stanislav Kozlovski
>> > >  wrote:
>> > >
>> > >> Hey Luke,
>> > >>
>> > >> This is an interesting problem. Given the fact that the KIP for
>> having a
>> > >> 3.8 release passed, I think it weights the scale towards not calling
>> > this a
>> > >> blocker and expecting it to be solved in 3.7.1.
>> > >>
>> > >> It is unfortunate that it would not seem safe to migrate to KRaft in
>> > 3.7.0
>> > >> (given the inability to rollback safely), but if that's true - the
>> same
>> > >> case would apply for 3.6.0. So in any case users w\ould be expected
>> to
>> > use a
>> > >> patch release for this.
>> >
>> > Hi Luke,
>> >
>> > Thanks for testing rollback. I think this is a case where the
>> > documentation is wrong. The intention was to for the steps to basically
>> be:
>> >
>> > 1. roll all the brokers into zk mode, but with migration enabled
>> > 2. take down the kraft quorum
>> > 3. rmr /controller, allowing a hybrid broker to take over.
>> > 4. roll all the brokers into zk mode without migration enabled (if
>> desired)
>> >
>> > With these steps, there isn't really unavailability since a ZK
>> controller
>> > can be elected quickly after the kraft quorum is gone.
>> >
>> > >> Further, since we will have a 3.8 release - it is
>> > >> likely we will ultimately recommend users upgrade from that version
>> > given
>> > >> its aim is to have strategic KRaft feature parity with ZK.
>> > >> That being said, I am not 100% on this. Let me know whether you think
>> > this
>> > >> should block the release, Luke. I am also tagging Colin and David to
>> > weigh
>> > >> in with their opinions, as they worked on the migration logic.
>> >
>> > The rollback docs are new in 3.7 so the fact that they're wrong is a
>> clear
>> > blocker, I think. But easy to fix, I believe. I will create a PR.
>> >
>> > best,
>> > Colin
>> >
>> > >>
>> > >> Hey Kirk and Chris,
>> > >>
>> > >> Unless I'm missing something - KAFKALESS-16029 is simply a bad log
>> due
>> > to
>> > >> improper closing. And the PR description implies this has been
>> present
>> > >> since 3.5. While annoying, I don't see a strong reason for this to
>> block
>> > >> the release.
>> > >>
>> > >> Hey Jakub,
>> > >>
>> > >> Nice catch! It does seem like we should have gated this behind the
>> > metadata
>> > >> version as KIP-858 implies. Is the cluster configured with multiple
>> log
>> > >> dirs? What is the impact of the error messages?
>> > >>
>> > >> Tagging Igor (the 

[jira] [Created] (KAFKA-16157) Topic recreation with offline disk doesn't update leadership/shrink ISR correctly

[Gaurav Narula (Jira)]

Gaurav Narula created KAFKA-16157:
-

 Summary: Topic recreation with offline disk doesn't update 
leadership/shrink ISR correctly
 Key: KAFKA-16157
 URL: https://issues.apache.org/jira/browse/KAFKA-16157
 Project: Kafka
  Issue Type: Bug
  Components: jbod, kraft
Affects Versions: 3.7.1
Reporter: Gaurav Narula


In a cluster with 4 brokers, `broker-1..broker-4` with 2 disks `d1` and `d2` in 
each broker, we perform the following operations:

 
 # Create a topic `foo.test` with 10 replicas and RF 4. Let's assume the topic 
was created with id `rAujIqcjRbu_-E4UxgQT8Q`.
 # Start a producer in the background to produce to `foo.test`.
 # Break disk `d1` in `broker-1`. We simulate this by marking the log dir 
read-only.
 # Delete topic `foo.test`
 # Recreate topic `foo.test`. Let's assume the topic was created with id 
`bgdrsv-1QjCLFEqLOzVCHg`.
 # Wait for 5 minutes
 # Describe the recreated topic `foo.test`.

 

We observe that `broker-1` is the leader and in-sync for few partitions

 

 
{code:java}
 
Topic: foo.test TopicId: bgdrsv-1QjCLFEqLOzVCHg PartitionCount: 10      
ReplicationFactor: 4    Configs: 
min.insync.replicas=1,unclean.leader.election.enable=false
        Topic: foo.test Partition: 0    Leader: 101     Replicas: 
101,102,103,104       Isr: 101,102,103,104
        Topic: foo.test Partition: 1    Leader: 102     Replicas: 
102,103,104,101       Isr: 102,103,104
        Topic: foo.test Partition: 2    Leader: 103     Replicas: 
103,104,101,102       Isr: 103,104,102
        Topic: foo.test Partition: 3    Leader: 104     Replicas: 
104,101,102,103       Isr: 104,102,103
        Topic: foo.test Partition: 4    Leader: 104     Replicas: 
104,102,101,103       Isr: 104,102,103
        Topic: foo.test Partition: 5    Leader: 102     Replicas: 
102,101,103,104       Isr: 102,103,104
        Topic: foo.test Partition: 6    Leader: 101     Replicas: 
101,103,104,102       Isr: 101,103,104,102
        Topic: foo.test Partition: 7    Leader: 103     Replicas: 
103,104,102,101       Isr: 103,104,102
        Topic: foo.test Partition: 8    Leader: 101     Replicas: 
101,102,104,103       Isr: 101,102,104,103
        Topic: foo.test Partition: 9    Leader: 102     Replicas: 
102,104,103,101       Isr: 102,104,103
{code}
 

 

In this example, it is the leader of partitions `0, 6 and 8`.

 

Consider `foo.test-8`. It is present in the following brokers/disks:

 

 
{code:java}
$ fd foo.test-8
broker-1/d1/foo.test-8/
broker-2/d2/foo.test-8/
broker-3/d2/foo.test-8/
broker-4/d1/foo.test-8/{code}
 

 

`broker-1/d1` still refers to the topic id which is pending deletion because 
the log dir is marked offline.

 

 
{code:java}
$ cat broker-1/d1/foo.test-8/partition.metadata
version: 0
topic_id: rAujIqcjRbu_-E4UxgQT8Q{code}
 

 

However, other brokers have the correct topic-id

 

 
{code:java}
$ cat broker-2/d2/foo.test-8/partition.metadata
version: 0
topic_id: bgdrsv-1QjCLFEqLOzVCHg%{code}
 

 

Now, let's consider `foo.test-0`. We observe that the replica isn't present in 
`broker-1`:




{code:java}
$ fd foo.test-0
broker-2/d1/foo.test-0/
broker-3/d1/foo.test-0/
broker-4/d2/foo.test-0/{code}



In both cases, `broker-1` shouldn't be the leader or in-sync replica for the 
partitions.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1016 Make MM2 heartbeats topic name configurable

[Viktor Somogyi-Vass]

Hi Bertalan,

Thanks for creating this KIP.
A couple of observations/questions:
1. If I have multiple source->target pairs, can I set this property per
cluster by prefixing with "source->target" as many other configs or is it
global?
2. The replication policy must be set in MirrorClient as well. Is your
change applicable to both MirrorClient and the connectors as well?
3. It might be worth pointing out (both in the docs and the KIP) that if
the user overrides the replication policy to any other than
DefaultReplicationPolicy, then this config has no effect.
4. With regards to integration tests, I tend to lean towards that we don't
need them if we can cover this well with unit tests and mocking.

Thanks,
Viktor

On Wed, Jan 17, 2024 at 12:23 AM Ryanne Dolan  wrote:

> Makes sense to me, +1.
>
> On Tue, Jan 16, 2024 at 5:04 PM Kondrát Bertalan 
> wrote:
>
>> Hey Team,
>>
>> I would like to start a discussion thread about the *KIP-1016 Make MM2
>> heartbeats topic name configurable
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1016+Make+MM2+heartbeats+topic+name+configurable
>> >*
>> .
>>
>> This KIP aims to make the default heartbeat topic name (`heartbeats`) in
>> the DefaultReplicationPolicy configurable via a property.
>> Since this is my first KIP and the change is small, I implemented it in
>> advance so, I can include the PR
>>  as well.
>>
>> I appreciate all your feedbacks and comments.
>>
>> Special thanks to Viktor Somogyi-Vass  and
>> Daniel
>> Urban  for the original idea and help.
>> Thank you,
>> Berci
>>
>> --
>> *Bertalan Kondrat* | Founder, SWE
>> servy.hu 
>>
>>
>>
>> 
>> --
>>
>

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

Hi Christo,
Thanks for volunteering to contribute to the KIP discussion. I suggest
considering this KIP for both ZK and KRaft as it will be helpful for
this feature to be available in 3.8.0 running with ZK clusters.

Thanks,
Satish.

On Wed, 17 Jan 2024 at 19:04, Christo Lolov  wrote:
>
> Hello!
>
> I volunteer to get this KIP moving forward and implemented in Apache Kafka
> 3.8.
>
> I have caught up with Mehari offline and we have agreed that given Apache
> Kafka 4.0 being around the corner we would like to propose this feature
> only for KRaft clusters.
>
> Any and all reviews and comments are welcome!
>
> Best,
> Christo
>
> On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
> wrote:
>
> > Hi everyone, any progress on the status of this KIP? Overall looks good to
> > me but I wonder whether we still need to support it for Zookeeper mode
> > given that it will be deprecated in the next 3 months.
> >
> > On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> > > Hi everyone,
> > > I would like to start a discussion on KIP-950: Tiered Storage Disablement
> > (
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> > ).
> > >
> > > This KIP proposes adding the ability to disable and re-enable tiered
> > storage on a topic.
> > >
> > > Thanks,
> > > Mehari
> > >
> >

[jira] [Created] (KAFKA-16156) System test failing on endOffsets with negative timestamps

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16156:
--

 Summary: System test failing on endOffsets with negative timestamps
 Key: KAFKA-16156
 URL: https://issues.apache.org/jira/browse/KAFKA-16156
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Reporter: Lianet Magrans


TransactionalMessageCopier run with 3.7 new consumer fails with "Invalid 
negative timestamp".
Trace:
[2024-01-15 07:42:33,933] ERROR Shutting down after unexpected error in event 
loop (org.apache.kafka.tools.TransactionalMessageCopier)
org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: 
Invalid negative timestamp
at 
org.apache.kafka.clients.consumer.internals.ConsumerUtils.maybeWrapAsKafkaException(ConsumerUtils.java:234)
at 
org.apache.kafka.clients.consumer.internals.ConsumerUtils.getResult(ConsumerUtils.java:212)
at 
org.apache.kafka.clients.consumer.internals.events.CompletableApplicationEvent.get(CompletableApplicationEvent.java:44)
at 
org.apache.kafka.clients.consumer.internals.events.ApplicationEventHandler.addAndGet(ApplicationEventHandler.java:113)
at 
org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.beginningOrEndOffset(AsyncKafkaConsumer.java:1134)
at 
org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.endOffsets(AsyncKafkaConsumer.java:1113)
at 
org.apache.kafka.clients.consumer.internals.AsyncKafkaConsumer.endOffsets(AsyncKafkaConsumer.java:1108)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.endOffsets(KafkaConsumer.java:1651)
at 
org.apache.kafka.tools.TransactionalMessageCopier.messagesRemaining(TransactionalMessageCopier.java:246)
at 
org.apache.kafka.tools.TransactionalMessageCopier.runEventLoop(TransactionalMessageCopier.java:342)
at 
org.apache.kafka.tools.TransactionalMessageCopier.main(TransactionalMessageCopier.java:292)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16155) Investigate testAutoCommitIntercept

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-16155:
--

 Summary: Investigate testAutoCommitIntercept
 Key: KAFKA-16155
 URL: https://issues.apache.org/jira/browse/KAFKA-16155
 Project: Kafka
  Issue Type: Bug
  Components: clients, consumer
Reporter: Lucas Brutschy


Even with KAFKA-15942, the test PlaintextConsumerTest.testAutoCommitIntercept 
flakes on the the initial setup (before using interceptors, so interceptors are 
unrelated here, except for being used later in the test).

The problem is that we are seeking two topic partitions to offset 10 and 20, 
respectively, but when we commit, we seem to have lost one of the offsets, 
likely due to a race condition. 

When I output `subscriptionState.allConsumed` repeatedly, I get this output:

 
{code:java}
allConsumed: {topic-0=OffsetAndMetadata{offset=100, leaderEpoch=0, 
metadata=''}, topic-1=OffsetAndMetadata{offset=0, leaderEpoch=null, 
metadata=''}}
seeking topic-0 to FetchPosition{offset=10, offsetEpoch=Optional.empty, 
currentLeader=LeaderAndEpoch{leader=Optional[localhost:58298 (id: 0 rack: 
null)], epoch=0}}
seeking topic-1 to FetchPosition{offset=20, offsetEpoch=Optional.empty, 
currentLeader=LeaderAndEpoch{leader=Optional[localhost:58301 (id: 1 rack: 
null)], epoch=0}}
allConsumed: {topic-0=OffsetAndMetadata{offset=10, leaderEpoch=null, 
metadata=''}, topic-1=OffsetAndMetadata{offset=20, leaderEpoch=null, 
metadata=''}} 
allConsumed: {topic-0=OffsetAndMetadata{offset=10, leaderEpoch=null, 
metadata=''}} allConsumed: {topic-0=OffsetAndMetadata{offset=10, 
leaderEpoch=null, metadata=''}, topic-1=OffsetAndMetadata{offset=0, 
leaderEpoch=null, metadata=''}} autocommit start 
{topic-0=OffsetAndMetadata{offset=10, leaderEpoch=null, metadata=''}, 
topic-1=OffsetAndMetadata{offset=0, leaderEpoch=null, metadata=''}}
{code}
So we after we seek to 10 / 20, we lose one of the offsets, maybe because we 
haven't reconciled the assignment yet. Later, we get the second topic partition 
assigned, but the offset is initialized to 0.

We should investigate whether this can be made more like the behavior in the 
original consumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Christo Lolov]

Hello!

I volunteer to get this KIP moving forward and implemented in Apache Kafka
3.8.

I have caught up with Mehari offline and we have agreed that given Apache
Kafka 4.0 being around the corner we would like to propose this feature
only for KRaft clusters.

Any and all reviews and comments are welcome!

Best,
Christo

On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
wrote:

> Hi everyone, any progress on the status of this KIP? Overall looks good to
> me but I wonder whether we still need to support it for Zookeeper mode
> given that it will be deprecated in the next 3 months.
>
> On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> > Hi everyone,
> > I would like to start a discussion on KIP-950: Tiered Storage Disablement
> (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> ).
> >
> > This KIP proposes adding the ability to disable and re-enable tiered
> storage on a topic.
> >
> > Thanks,
> > Mehari
> >
>

[jira] [Created] (KAFKA-16154) Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP

[Christo Lolov (Jira)]

Christo Lolov created KAFKA-16154:
-

 Summary: Make broker changes to return an offset for 
LATEST_TIERED_TIMESTAMP
 Key: KAFKA-16154
 URL: https://issues.apache.org/jira/browse/KAFKA-16154
 Project: Kafka
  Issue Type: Sub-task
Reporter: Christo Lolov
Assignee: Christo Lolov
 Fix For: 3.8.0


A broker should start returning offsets when given a timestamp of -5, which 
signifies a LATEST_TIERED_TIMESTAMP.

There are 3 cases.

Tiered Storage is not enabled. In such a situation asking for 
LATEST_TIERED_TIMESTAMP should always return no offset.

Tiered Storage is enabled and there is nothing in remote storage. In such a 
situation the offset returned should be 0.

Tiered Storage is enabled and there is something in remote storage. In such a 
situation the offset returned should be the highest offset the broker is aware 
of.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

[Yash Mayya]

Hi Ashwin,

Thanks for the KIP.

> If Connect runtime encounters an error in any of these steps,
> it will cleanup (if required) and return an error response

Could you please elaborate on the cleanup steps? For instance, if we
encounter an error after wiping existing offsets but before writing the new
offsets, there's not really any good way to "revert" the wiped offsets.
It's definitely extremely unlikely that a user would expect the previous
offsets for a connector to still be present (by creating a new connector
with the same name but without initial offsets for instance) after such a
failed operation, but it would still be good to call this out explicitly. I
presume that we'd want to wipe the newly written initial offsets if we fail
while writing the connector's config however?

> Validate the offset using the same checks performed while
> altering connector offsets (PATCH /$connector/offsets ) as
> specified in KIP-875

The `PATCH /connectors/{connector}/offsets` and `DELETE
/connectors/{connector}/offsets` endpoints have two possible success
messages in the response depending on whether or not the connector plugin
has implemented the `alterOffsets` connector method. Since we're proposing
to utilize the same offset validation during connector creation if initial
offsets are specified, I think it would be valuable to surface similar
information to users here as well. Thoughts?

Thanks,
Yash

On Wed, Jan 17, 2024 at 3:31 PM Ashwin  wrote:

> Hi All ,
>
> Can I please get one more binding vote, so that the KIP is approved ?
> Thanks for the votes Chris and Mickael !
>
>
> - Ashwin
>
>
> On Thu, Jan 11, 2024 at 3:55 PM Mickael Maison 
> wrote:
>
> > Hi Ashwin,
> >
> > +1 (binding), thanks for the KIP
> >
> > Mickael
> >
> > On Tue, Jan 9, 2024 at 4:54 PM Chris Egerton 
> > wrote:
> > >
> > > Thanks for the KIP! +1 (binding)
> > >
> > > On Mon, Jan 8, 2024 at 9:35 AM Ashwin 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I would like to start  a vote on KIP-995.
> > > >
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors
> > > >
> > > > Discussion thread -
> > > > https://lists.apache.org/thread/msorbr63scglf4484yq764v7klsj7c4j
> > > >
> > > > Thanks!
> > > >
> > > > Ashwin
> > > >
> >
>

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

[Christo Lolov]

Thank you everyone for casting your votes!

KIP-1005 passes with 6 +1 (three binding and three not binding) - I will
get down to implementing it :)

Best,
Christo

On Tue, 16 Jan 2024 at 01:35, Luke Chen  wrote:

> +1 binding from me.
>
> Thanks for the KIP!
> Luke
>
> On Fri, Jan 12, 2024 at 5:41 PM Federico Valeri 
> wrote:
>
> > +1 non binding
> >
> > Thanks
> >
> > On Fri, Jan 12, 2024 at 1:31 AM Boudjelda Mohamed Said
> >  wrote:
> > >
> > > +1 (binding)
> > >
> > >
> > > On Fri, Jan 12, 2024 at 1:21 AM Satish Duggana <
> satish.dugg...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Thu, 11 Jan 2024 at 17:52, Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Dec 26, 2023 at 7:05 AM Kamal Chandraprakash <
> > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > >
> > > > > > +1 (non-binding). Thanks for the KIP!
> > > > > >
> > > > > > --
> > > > > > Kamal
> > > > > >
> > > > > > On Thu, Dec 21, 2023 at 2:23 PM Christo Lolov <
> > christolo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Heya all!
> > > > > > >
> > > > > > > KIP-1005 (
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Expose+EarliestLocalOffset+and+TieredOffset
> > > > > > > )
> > > > > > > has been open for around a month with no further comments - I
> > would
> > > > like
> > > > > > to
> > > > > > > start a voting round on it!
> > > > > > >
> > > > > > > Best,
> > > > > > > Christo
> > > > > > >
> > > > > >
> > > >
> >
>

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

[Ashwin]

Hi All ,

Can I please get one more binding vote, so that the KIP is approved ?
Thanks for the votes Chris and Mickael !


- Ashwin


On Thu, Jan 11, 2024 at 3:55 PM Mickael Maison 
wrote:

> Hi Ashwin,
>
> +1 (binding), thanks for the KIP
>
> Mickael
>
> On Tue, Jan 9, 2024 at 4:54 PM Chris Egerton 
> wrote:
> >
> > Thanks for the KIP! +1 (binding)
> >
> > On Mon, Jan 8, 2024 at 9:35 AM Ashwin 
> wrote:
> >
> > > Hi All,
> > >
> > > I would like to start  a vote on KIP-995.
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors
> > >
> > > Discussion thread -
> > > https://lists.apache.org/thread/msorbr63scglf4484yq764v7klsj7c4j
> > >
> > > Thanks!
> > >
> > > Ashwin
> > >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2579

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #66

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16153) kraft_upgrade_test system test is broken

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-16153:
--

 Summary: kraft_upgrade_test system test is broken
 Key: KAFKA-16153
 URL: https://issues.apache.org/jira/browse/KAFKA-16153
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Reporter: Mickael Maison


I get the following failure from all `from_kafka_version` versions:


Command '/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server 
ducker05:9092,ducker06:9092,ducker07:9092 upgrade --metadata 3.8' returned 
non-zero exit status 1. Remote error message: b'SLF4J: Class path contains 
multiple SLF4J bindings.\nSLF4J: Found binding in 
[jar:file:/opt/kafka-dev/tools/build/dependant-libs-2.13.12/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
 Found binding in 
[jar:file:/opt/kafka-dev/trogdor/build/dependant-libs-2.13.12/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
 See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.\nSLF4J: Actual binding is of type 
[org.slf4j.impl.Reload4jLoggerFactory]\nUnsupported metadata version 3.8. 
Supported metadata versions are 3.3-IV0, 3.3-IV1, 3.3-IV2, 3.3-IV3, 3.4-IV0, 
3.5-IV0, 3.5-IV1, 3.5-IV2, 3.6-IV0, 3.6-IV1, 3.6-IV2, 3.7-IV0, 3.7-IV1, 
3.7-IV2, 3.7-IV3, 3.7-IV4, 3.8-IV0\n'



--
This message was sent by Atlassian Jira
(v8.20.10#820010)