[jira] [Created] (KAFKA-16181) Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-16181:
---

 Summary: Use incrementalAlterConfigs when updating broker configs 
by kafka-configs.sh
 Key: KAFKA-16181
 URL: https://issues.apache.org/jira/browse/KAFKA-16181
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming
Assignee: Deng Ziming






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15619) Deleted topics will come back again

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15619.
-
  Assignee: Deng Ziming
Resolution: Invalid

This has been fixed after setting auto.create.topics.enable=false.

> Deleted topics will come back again
> ---
>
> Key: KAFKA-15619
> URL: https://issues.apache.org/jira/browse/KAFKA-15619
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 3.5.0, 3.5.1
>    Reporter: Deng Ziming
>    Assignee: Deng Ziming
>Priority: Major
>
> Deleted topics will come back again in Apache Spark structured streaming 
> stress test after upgrade Kafka from 3.4.0 to 3.5.0, related ticket is: 
> https://issues.apache.org/jira/browse/SPARK-45529 , the test randomly 
> starts/stops/adds data/add partitions/delete topic/add topic/checks the 
> result in a loop, I finally found that a deleted topic will come back again 
> after some time.
> By constantly reseting the head of branch-3.5 and using {{gradlew install}} 
> to repackage and rerunning of the stress test, I have basically inferred that 
> this bug comes from [https://github.com/apache/kafka/pull/12590]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13005) Support JBOD in kraft mode

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-13005.
-
Resolution: Duplicate

> Support JBOD in kraft mode
> --
>
> Key: KAFKA-13005
> URL: https://issues.apache.org/jira/browse/KAFKA-13005
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin McCabe
>    Assignee: Deng Ziming
>Priority: Major
>  Labels: kip-500
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15390) FetchResponse.preferredReplica may contains fenced replica in KRaft mode

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15390.
-
Fix Version/s: 3.6.0
   Resolution: Fixed

> FetchResponse.preferredReplica may contains fenced replica in KRaft mode
> 
>
> Key: KAFKA-15390
> URL: https://issues.apache.org/jira/browse/KAFKA-15390
> Project: Kafka
>  Issue Type: Bug
>        Reporter: Deng Ziming
>    Assignee: Deng Ziming
>Priority: Major
> Fix For: 3.6.0
>
>
> `KRaftMetadataCache.getPartitionReplicaEndpoints` will return a fenced broker 
> id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15619) Kafka

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15619:
---

 Summary: Kafka
 Key: KAFKA-15619
 URL: https://issues.apache.org/jira/browse/KAFKA-15619
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 3.5.1, 3.5.0
Reporter: Deng Ziming


Deleted topics will come back again in Spark structured streaming test after 
upgrade Kafka from 3.4.0 to 3.5.0, related ticket is:  
https://issues.apache.org/jira/browse/SPARK-45529

 

I have basically inferred that this bug comes from 
https://github.com/apache/kafka/pull/12590



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15566) Klaky tests in FetchRequestTest.scala in KRaft mode

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15566:
---

 Summary: Klaky tests in FetchRequestTest.scala in KRaft mode
 Key: KAFKA-15566
 URL: https://issues.apache.org/jira/browse/KAFKA-15566
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming


|[https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/#showFailuresLink]
[Build / JDK 11 and Scala 2.13 /

kafka.server.FetchRequestTest.testLastFetchedEpochValidation(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidation_String__quorum_kraft/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testLastFetchedEpochValidationV12(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidationV12_String__quorum_kraft/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testFetchWithPartitionsWithIdError(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testFetchWithPartitionsWithIdError_String__quorum_kraft_2/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testLastFetchedEpochValidation(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidation_String__quorum_kraft_2/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testLastFetchedEpochValidationV12(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidationV12_String__quorum_kraft_2/]|
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15315) Use getOrDefault rather than get

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15315.
-
Resolution: Fixed

> Use getOrDefault rather than get
> 
>
> Key: KAFKA-15315
> URL: https://issues.apache.org/jira/browse/KAFKA-15315
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: roon
>Priority: Trivial
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15140) Improve TopicCommandIntegrationTest to be less flaky

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming reopened KAFKA-15140:
-

This is still flaky, it seems the internal topic is also under min isr 
partitions.

https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13562/23/testReport/junit/kafka.admin/TopicCommandIntegrationTest/Build___JDK_8_and_Scala_2_12___testDescribeUnderMinIsrPartitionsMixed_String__quorum_kraft/

> Improve TopicCommandIntegrationTest to be less flaky
> 
>
> Key: KAFKA-15140
> URL: https://issues.apache.org/jira/browse/KAFKA-15140
> Project: Kafka
>  Issue Type: Test
>  Components: unit tests
>Reporter: Divij Vaidya
>Assignee: Lan Ding
>Priority: Minor
>  Labels: newbie
> Fix For: 3.6.0, 3.5.1
>
>
> *This is a good Jira for folks who are new to contributing to Kafka.*
> Tests in TopicCommandIntegrationTest get flaky from time to time. The 
> objective of the task is to make them more robust by doing the following:
> 1. Replace the usage {-}createAndWaitTopic{-}() adminClient.createTopics() 
> method and other places where were are creating a topic (without waiting) 
> with 
> TestUtils.createTopicWithAdmin(). The latter method already contains the 
> functionality to create a topic and wait for metadata to sync up.
> 2. Replace the number 6 at places such as 
> "adminClient.createTopics(
> Collections.singletonList(new NewTopic("foo_bar", 1, 6.toShort)))" with a 
> meaningful constant.
> 3. Add logs if an assertion fails, for example, lines such as "
> assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), output)" should 
> have a third argument which prints the actual output printed so that we can 
> observe in the test logs on what was the output when assertion failed.
> 4. Replace occurrences of "\n" with System.lineSeparator() which is platform 
> independent
> 5. We should wait for reassignment to complete whenever we are re-assigning 
> partitions using alterconfig before we call describe to validate it. We could 
> use 
> TestUtils.waitForAllReassignmentsToComplete()
> *Motivation of this task*
> Try to fix the flaky test behaviour such as observed in 
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13924/5/testReport/junit/kafka.admin/TopicCommandIntegrationTest/Build___JDK_11_and_Scala_2_13___testDescribeUnderMinIsrPartitionsMixed_String__quorum_zk/]
>  
> {noformat}
> org.opentest4j.AssertionFailedError: expected:  but was: 
>   at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>   at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>   at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
>   at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
>   at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
>   at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
>   at 
> app//kafka.admin.TopicCommandIntegrationTest.testDescribeUnderMinIsrPartitionsMixed(TopicCommandIntegrationTest.scala:794){noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15286) Migrate ApiVersion related code to kraft

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15286.
-
Resolution: Fixed

> Migrate ApiVersion related code to kraft
> 
>
> Key: KAFKA-15286
> URL: https://issues.apache.org/jira/browse/KAFKA-15286
> Project: Kafka
>  Issue Type: Task
>        Reporter: Deng Ziming
>    Assignee: Deng Ziming
>Priority: Major
>
> In many places involving ApiVersion, we only support zk, we should move it 
> forward to kraft.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15390) FetchResponse.preferredReplica may contains fenced replica in KRaft mode

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15390:
---

 Summary: FetchResponse.preferredReplica may contains fenced 
replica in KRaft mode
 Key: KAFKA-15390
 URL: https://issues.apache.org/jira/browse/KAFKA-15390
 Project: Kafka
  Issue Type: Bug
Reporter: Deng Ziming
Assignee: Deng Ziming


`KRaftMetadataCache.getPartitionReplicaEndpoints` will return a fenced broker 
id.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15371) MetadataShell is stuck when bootstrapping

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15371:
---

 Summary: MetadataShell is stuck when bootstrapping
 Key: KAFKA-15371
 URL: https://issues.apache.org/jira/browse/KAFKA-15371
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.1
Reporter: Deng Ziming
 Attachments: image-2023-08-17-10-35-01-039.png, 
image-2023-08-17-10-35-36-067.png

I  downloaded the 3.5.1 package and startup it, then use metadata shell to 
inspect the data

 
{code:java}
// shell
bin/kafka-metadata-shell.sh --snapshot 
/tmp/kraft-combined-logs/__cluster_metadata-0/.log  {code}
Then process will stuck at loading.

 

 

!image-2023-08-17-10-35-36-067.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15354) Partition leader is not evenly distributed in kraft mode

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15354:
---

 Summary: Partition leader is not evenly distributed in kraft mode
 Key: KAFKA-15354
 URL: https://issues.apache.org/jira/browse/KAFKA-15354
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming


In StripedReplicaPlacerTest, we can create a test below to reproduce this bug.
{code:java}
// code placeholder
@Test
public void testReplicaDistribution() {
MockRandom random = new MockRandom();
StripedReplicaPlacer placer = new StripedReplicaPlacer(random);
TopicAssignment assignment = place(placer, 0, 4, (short) 2, Arrays.asList(
new UsableBroker(0, Optional.of("0"), false),
new UsableBroker(1, Optional.of("0"), false),
new UsableBroker(2, Optional.of("1"), false),
new UsableBroker(3, Optional.of("1"), false)));
System.out.println(assignment);
} {code}
In StripedReplicaPlacer, we only ensure leader are distributed evenly across 
racks, but we didn't ensure leader are evenly distributed across nodes. in the 
test above, we have 4 node: 1 2 3 4, and create 4 partitions but the leaders 
are  1 2 1 2. while in zk mode, this is ensured, see 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13836) Improve KRaft broker heartbeat logic

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-13836.
-
Resolution: Won't Fix

> Improve KRaft broker heartbeat logic
> 
>
> Key: KAFKA-13836
> URL: https://issues.apache.org/jira/browse/KAFKA-13836
> Project: Kafka
>  Issue Type: Improvement
>        Reporter: Deng Ziming
>    Assignee: Deng Ziming
>Priority: Major
>
> # Don't advertise an offset to the controller until it has been published
>  # only unfence a broker when it has seen it's own registration



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15289) Support KRaft mode in RequestQuotaTest

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15289.
-
Resolution: Fixed

> Support KRaft mode in RequestQuotaTest
> --
>
> Key: KAFKA-15289
> URL: https://issues.apache.org/jira/browse/KAFKA-15289
> Project: Kafka
>  Issue Type: Sub-task
>        Reporter: Deng Ziming
>Priority: Major
>  Labels: newbee
>
> we are calling `zkBrokerApis` in RequestQuotaTest, we should ensure kraft 
> broker apis are also supported, so use clientApis as far as possible.use 
> zkBrokerApis.clientApis instead of ApiKeys.zkBrokerApis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15340) Test request quota for kraft controller apis

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15340:
---

 Summary: Test request quota for kraft controller apis
 Key: KAFKA-15340
 URL: https://issues.apache.org/jira/browse/KAFKA-15340
 Project: Kafka
  Issue Type: Improvement
  Components: kraft, unit tests
Reporter: Deng Ziming


The RequestQuotaTest only tests request quota for kraft broker apis and zk 
broker apis, we should also test kraft controller apis. 

further more, maybe there are others tests we need to complement for kraft 
controller apis(not only broker apis).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15334) DescribeQuorum should not be seen as cluster action

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15334:
---

 Summary: DescribeQuorum should not be seen as cluster action 
 Key: KAFKA-15334
 URL: https://issues.apache.org/jira/browse/KAFKA-15334
 Project: Kafka
  Issue Type: Bug
Reporter: Deng Ziming






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15287) Change NodeApiVersions.create() to contains both apis of zk and kraft broker

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15287.
-
Resolution: Fixed

> Change NodeApiVersions.create() to contains both apis of zk and kraft broker 
> -
>
> Key: KAFKA-15287
> URL: https://issues.apache.org/jira/browse/KAFKA-15287
> Project: Kafka
>  Issue Type: Sub-task
>        Reporter: Deng Ziming
>Priority: Major
>  Labels: newbee
>
> We are using ApiKeys.zkBrokerApis() when calling NodeApiVersions.create(), 
> this means we only support zk broker apis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15288) Change BrokerApiVersionsCommandTest to support kraft mode

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15288.
-
Resolution: Fixed

> Change BrokerApiVersionsCommandTest to support kraft mode
> -
>
> Key: KAFKA-15288
> URL: https://issues.apache.org/jira/browse/KAFKA-15288
> Project: Kafka
>  Issue Type: Sub-task
>        Reporter: Deng Ziming
>Priority: Minor
>
> Currently we only test zk mode for BrokerApiVersionsCommand



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15289) Use zkBrokerApis.clientApis instead of ApiKeys.zkBrokerApis in most cases

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15289:
---

 Summary: Use zkBrokerApis.clientApis instead of 
ApiKeys.zkBrokerApis in most cases
 Key: KAFKA-15289
 URL: https://issues.apache.org/jira/browse/KAFKA-15289
 Project: Kafka
  Issue Type: Sub-task
Reporter: Deng Ziming


In most Test cases, we are calling `zkBrokerApis`, we should ensure kraft 
broker apis are also supported, so use clientApis as far as possible.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15288) Change BrokerApiVersionsCommandTest to support kraft mode

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15288:
---

 Summary: Change BrokerApiVersionsCommandTest to support kraft mode
 Key: KAFKA-15288
 URL: https://issues.apache.org/jira/browse/KAFKA-15288
 Project: Kafka
  Issue Type: Sub-task
Reporter: Deng Ziming


Currently 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15287) Change NodeApiVersions.create() to contains both apis of zk and kraft broker

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15287:
---

 Summary: Change NodeApiVersions.create() to contains both apis of 
zk and kraft broker 
 Key: KAFKA-15287
 URL: https://issues.apache.org/jira/browse/KAFKA-15287
 Project: Kafka
  Issue Type: Sub-task
Reporter: Deng Ziming


We are using ApiKeys.zkBrokerApis() when calling NodeApiVersions.create(), this 
means we only support zk broker apis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15286) Migrate ApiVersion related code to kraft

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15286:
---

 Summary: Migrate ApiVersion related code to kraft
 Key: KAFKA-15286
 URL: https://issues.apache.org/jira/browse/KAFKA-15286
 Project: Kafka
  Issue Type: Task
Reporter: Deng Ziming
Assignee: Deng Ziming


In many places involving ApiVersion, we only support zk, we should move it 
forward to kraft.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15036) Kraft leader change fails when invoking getFinalizedFeatures

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-15036.
-
Resolution: Fixed

> Kraft leader change fails when invoking getFinalizedFeatures
> 
>
> Key: KAFKA-15036
> URL: https://issues.apache.org/jira/browse/KAFKA-15036
> Project: Kafka
>  Issue Type: Improvement
>        Reporter: Deng Ziming
>    Assignee: Deng Ziming
>Priority: Major
>
> When kraft leader changes, we can receiving a error as follows:
>  
> {{[2023-05-24 18:00:02,898] WARN [QuorumController id=3002] 
> getFinalizedFeatures: failed with unknown server exception RuntimeException 
> in 271 us.  The controller is already in standby mode. 
> (org.apache.kafka.controller.QuorumController)
> java.lang.RuntimeException: No in-memory snapshot for epoch 9. Snapshot 
> epochs are: 
>   at 
> org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:173)
>   at 
> org.apache.kafka.timeline.SnapshotRegistry.iterator(SnapshotRegistry.java:131)
>   at org.apache.kafka.timeline.TimelineObject.get(TimelineObject.java:69)
>   at 
> org.apache.kafka.controller.FeatureControlManager.finalizedFeatures(FeatureControlManager.java:303)
>   at 
> org.apache.kafka.controller.QuorumController.lambda$finalizedFeatures$16(QuorumController.java:2016)
>   at 
> org.apache.kafka.controller.QuorumController$ControllerReadEvent.run(QuorumController.java:546)
>   at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
>   at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>   at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>   at java.base/java.lang.Thread.run(Thread.java:829)}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15065) ApiVersionRequest is not properly handled in Sasl ControllerServer

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15065:
---

 Summary: ApiVersionRequest is not properly handled in Sasl 
ControllerServer
 Key: KAFKA-15065
 URL: https://issues.apache.org/jira/browse/KAFKA-15065
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming


In KAFKA-14291 we add finalizedFeatures in ApiVersionResponse, also change the 
`apiVersionResponse` method to throw exception:
{code:java}
override def apiVersionResponse(requestThrottleMs: Int): ApiVersionsResponse = {
throw new UnsupportedOperationException("This method is not supported in 
SimpleApiVersionManager, use apiVersionResponse(throttleTimeMs, 
finalizedFeatures, epoch) instead")
} {code}
but this method is used in SocketServer:
{code:java}
private[network] val selector = createSelector(
ChannelBuilders.serverChannelBuilder(
listenerName,
listenerName == config.interBrokerListenerName,
securityProtocol,
config,
credentialProvider.credentialCache,
credentialProvider.tokenCache,
time,
logContext,
() => apiVersionManager.apiVersionResponse(throttleTimeMs = 0)
)
) {code}
 

And this method will be invoked in `SaslServerAuthenticator.authenticate` and 
will stop the process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15036) Kraft leader change fails when invoking getFinalizedFeatures

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-15036:
---

 Summary: Kraft leader change fails when invoking 
getFinalizedFeatures
 Key: KAFKA-15036
 URL: https://issues.apache.org/jira/browse/KAFKA-15036
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming


When kraft leader changes, we can receiving a error as follows:
 
{{[2023-05-24 18:00:02,898] WARN [QuorumController id=3002] 
getFinalizedFeatures: failed with unknown server exception RuntimeException in 
271 us.  The controller is already in standby mode. 
(org.apache.kafka.controller.QuorumController)
java.lang.RuntimeException: No in-memory snapshot for epoch 9. Snapshot epochs 
are: 
at 
org.apache.kafka.timeline.SnapshotRegistry.getSnapshot(SnapshotRegistry.java:173)
at 
org.apache.kafka.timeline.SnapshotRegistry.iterator(SnapshotRegistry.java:131)
at org.apache.kafka.timeline.TimelineObject.get(TimelineObject.java:69)
at 
org.apache.kafka.controller.FeatureControlManager.finalizedFeatures(FeatureControlManager.java:303)
at 
org.apache.kafka.controller.QuorumController.lambda$finalizedFeatures$16(QuorumController.java:2016)
at 
org.apache.kafka.controller.QuorumController$ControllerReadEvent.run(QuorumController.java:546)
at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
at java.base/java.lang.Thread.run(Thread.java:829)}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14989) Flaky test TransactionsTest.testFailureToFenceEpoch

[Deng Ziming (Jira)]

Deng Ziming created KAFKA-14989:
---

 Summary: Flaky test TransactionsTest.testFailureToFenceEpoch
 Key: KAFKA-14989
 URL: https://issues.apache.org/jira/browse/KAFKA-14989
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14291) KRaft: ApiVersionsResponse doesn't have finalizedFeatures and finalizedFeatureEpoch in KRaft mode

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming reopened KAFKA-14291:
-
  Assignee: Deng Ziming

> KRaft: ApiVersionsResponse doesn't have finalizedFeatures and 
> finalizedFeatureEpoch in KRaft mode
> -
>
> Key: KAFKA-14291
> URL: https://issues.apache.org/jira/browse/KAFKA-14291
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft
>Reporter: Akhilesh Chaganti
>Assignee: Deng Ziming
>Priority: Critical
>
> https://github.com/apache/kafka/blob/d834947ae7abc8a9421d741e742200bb36f51fb3/core/src/main/scala/kafka/server/ApiVersionManager.scala#L53
> ```
> class SimpleApiVersionManager(
>   val listenerType: ListenerType,
>   val enabledApis: collection.Set[ApiKeys],
>   brokerFeatures: Features[SupportedVersionRange]
> ) extends ApiVersionManager {
>   def this(listenerType: ListenerType) = {
> this(listenerType, ApiKeys.apisForListener(listenerType).asScala, 
> BrokerFeatures.defaultSupportedFeatures())
>   }
>   private val apiVersions = 
> ApiVersionsResponse.collectApis(enabledApis.asJava)
>   override def apiVersionResponse(requestThrottleMs: Int): 
> ApiVersionsResponse = {
> ApiVersionsResponse.createApiVersionsResponse(requestThrottleMs, 
> apiVersions, brokerFeatures)
>   }
> }
> ```
> ApiVersionManager for KRaft doesn't add the finalizedFeatures and 
> finalizedFeatureEpoch to the ApiVersionsResponse.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14689) Kafka won't start on Windows in KRaft mode

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-14689.
-
Resolution: Duplicate

This is the same with KAFKA-14273

> Kafka won't start on Windows in KRaft mode
> --
>
> Key: KAFKA-14689
> URL: https://issues.apache.org/jira/browse/KAFKA-14689
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Microsoft Windows 10 Pro
> 10.0.19045 Build 19045
> OpenJDK build 17.0.2
>Reporter: Kedar Joshi
>Priority: Major
> Attachments: kafka-log.txt, server.properties
>
>
> Kafka 3.4 is unable to start in KRaft mode on Windows 10.
> Kafka is started in Kafka directory with -
> {code:java}
> bin\windows\kafka-server-start.bat .\config\kraft\server.properties {code}
> but it always fails with following exception -
> {code:java}
> Caused by: java.nio.file.FileSystemException: 
> D:\LocationGuru\Servers\Kafka-3.4\.\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state.tmp
>  -> 
> D:\LocationGuru\Servers\Kafka-3.4\.\tmp\kraft-combined-logs\__cluster_metadata-0\quorum-state:
>  The process cannot access the file because it is being used by another 
> process
>         at 
> java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
>         at 
> java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
>         at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:403)
>         at 
> java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:293)
>         at java.base/java.nio.file.Files.move(Files.java:1430)
>         at 
> org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:949)
>         at 
> org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:932)
>         at 
> org.apache.kafka.raft.FileBasedStateStore.writeElectionStateToFile(FileBasedStateStore.java:152)
>         ... 15 more {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-893: The Kafka protocol should support nullable structs

[deng ziming]

+1 (binding)

--
Thanks,
Ziming


> On Dec 6, 2022, at 10:48, John Roesler  wrote:
> 
> +1 (binding)
> 
> Thanks,
> -John
> 
> On Mon, Dec 5, 2022, at 16:57, Kirk True wrote:
>> +1 (non-binding)
>> 
>> On Mon, Dec 5, 2022, at 10:05 AM, Colin McCabe wrote:
>>> +1 (binding)
>>> 
>>> best,
>>> Colin
>>> 
>>> On Mon, Dec 5, 2022, at 10:03, David Jacot wrote:
 Hi all,
 
 As this KIP-893 is trivial and non-controversial, I would like to
 start the vote on it. The KIP is here:
 https://cwiki.apache.org/confluence/x/YJIODg
 
 Thanks,
 David
>>> 

[jira] [Resolved] (KAFKA-14380) consumer should refresh preferred read replica on metadata update

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-14380.
-
Resolution: Duplicate

duplicated with KAFKA-14379

> consumer should refresh preferred read replica on metadata update
> -
>
> Key: KAFKA-14380
> URL: https://issues.apache.org/jira/browse/KAFKA-14380
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> The consumer (fetcher) clears the preferred read replica only on two 
> conditions:
>  # the consumer receives an OFFSET_OUT_OF_RANGE error
>  # the follower does not exist in the client's metadata (i.e., offline)
>  # preferred read replica value expires (5 minutes)
> For other errors, it will continue to reach to the possibly unavailable 
> follower and only after 5 minutes will it clear preferred read replica and go 
> back to the leader.
> A specific example is when a partition is reassigned. the consumer will get 
> NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred 
> read replica will not be refreshed as the follower is still online. it will 
> continue to reach out to the old follower until the preferred read replica 
> expires.
> the consumer can instead clear its preferred read replica whenever it updates 
> its metadata. so when the consumer receives NOT_LEADER_OR_FOLLOWER in the 
> scenario above it can find the new preferred read replica by fetching from 
> the leader without waiting for the old value to expire.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14378) consumer should refresh preferred read replica on update metadata

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-14378.
-
Resolution: Duplicate

duplicated with KAFKA-14379

> consumer should refresh preferred read replica on update metadata
> -
>
> Key: KAFKA-14378
> URL: https://issues.apache.org/jira/browse/KAFKA-14378
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> The consumer (fetcher) refreshes the preferred read replica only on three 
> conditions:
>  # the consumer receives an OFFSET_OUT_OF_RANGE error
>  # the follower does not exist in the client's metadata (i.e., offline)
>  # after metadata.max.age.ms (5 min default)
> For other errors, it will continue to reach to the possibly unavailable 
> follower and only after 5 minutes will it refresh the preferred read replica 
> and go back to the leader.
> A specific example is when a partition is reassigned. the consumer will get 
> NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred 
> read replica will not be refreshed as the follower is still online. it will 
> continue to reach out to the old follower until the preferred read replica 
> expires.
> the consumer can instead refresh its preferred read replica whenever it makes 
> a metadata update request. so when the consumer receives i.e. 
> NOT_LEADER_OR_FOLLOWER it can find the new preferred read replica without 
> waiting for the expiration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

[deng ziming]

Congrats!

--
Ziming

> On Nov 2, 2022, at 3:36 AM, Guozhang Wang  wrote:
> 
> Hi everyone,
> 
> I'd like to introduce our new Kafka PMC member, Bruno.
> 
> Bruno has been a committer since April. 2021 and has been very active in
> the community. He's a key contributor to Kafka Streams, and also helped
> review a lot of horizontal improvements such as Mockito. It is my pleasure
> to announce that Bruno has agreed to join the Kafka PMC.
> 
> Congratulations, Bruno!
> 
> -- Guozhang Wang, on behalf of Apache Kafka PMC

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

[deng ziming]

Thanks for this KIP,

+1 for this(binding).

--
Best,
Ziming

> On Oct 14, 2022, at 8:11 AM, José Armando García Sancio 
>  wrote:
> 
> Hello all,
> 
> I would like to start voting for "KIP-876: Time based cluster metadata
> snapshots."
> 
> KIP: https://cwiki.apache.org/confluence/x/MY3GDQ
> Discussion thread:
> https://lists.apache.org/thread/ww67h9d4xvgw1f7jn4zxwydmt8x1mq72
> 
> Thanks!
> -- 
> -José

Re: question

[deng ziming]

Hello jincheng,
Kafka provides Java Producer/Consumer/Admin public api, so you can access
Kafka if you are using Java to develop an Android App even though it's not
common. for example, you can develop an Android App to get the Kafka
metadata, send bury-point event log to Kafka, however, a better solution is
to do these jobs in a server and you access your server from your App.

--
Best,
Ziming

On Tue, Oct 11, 2022 at 12:41 AM 关锦成 <865617...@qq.com.invalid> wrote:

> Does kafka support Android client？

Re: [ANNOUNCE] New committer: Deng Ziming

[deng ziming]

Thank you all!

--
Best,
Ziming

On Tue, Oct 11, 2022 at 9:43 AM Luke Chen  wrote:

> Congratulations Ziming!!
> Very well deserved!
> 恭喜! :)
>
> Luke
>
> On Tue, Oct 11, 2022 at 9:22 AM Sagar  wrote:
>
> > Congratulations ziming!
> >
> > On Tue, 11 Oct 2022 at 4:16 AM, Bill Bejeck  wrote:
> >
> > > Congrats Ziming!
> > >
> > > Regards,
> > > Bill
> > >
> > > On Mon, Oct 10, 2022 at 5:32 PM Ismael Juma  wrote:
> > >
> > > > Congratulations Ziming!
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Oct 10, 2022 at 9:30 AM Jason Gustafson
> > >  > > > >
> > > > wrote:
> > > >
> > > > > Hi All
> > > > >
> > > > > The PMC for Apache Kafka has invited Deng Ziming to become a
> > committer,
> > > > > and we are excited to announce that he has accepted!
> > > > >
> > > > > Ziming has been contributing to Kafka for about three years. He has
> > > > > authored
> > > > > more than 100 patches and helped to review nearly as many. In
> > > particular,
> > > > > he made significant contributions to the KRaft project which had a
> > big
> > > > part
> > > > > in reaching our production readiness goal in the 3.3 release:
> > > > > https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in.
> > > > >
> > > > > Please join me in congratulating Ziming! Thanks for all of your
> > > > > contributions!
> > > > >
> > > > > -- Jason, on behalf of the Apache Kafka PMC
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-868 Metadata Transactions

[deng ziming]

Thanks for this KIP,

+1(non-binding) from me.

--
Best,
Ziming

> On Sep 28, 2022, at 10:05 PM, David Jacot  wrote:
> 
> +1 (binding). Thanks for the KIP. I really like the approach!
> 
> Best,
> David
> 
> On Wed, Sep 28, 2022 at 4:20 AM Luke Chen  wrote:
>> 
>> Hi David,
>> 
>> +1 (binding) from me.
>> 
>> Thank you
>> Luke
>> 
>> On Tue, Sep 27, 2022 at 11:02 PM David Arthur 
>> wrote:
>> 
>>> Hey folks, I'd like to start a vote on KIP-868. This proposal adds
>>> lightweight transactions to the KRaft controller. These transactions
>>> will allow us to generate atomic batches of records that exceed the
>>> current limits of the metadata layer.
>>> 
>>> Here is the KIP:
>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
>>> 
>>> And previous discussion:
>>> https://lists.apache.org/thread/895pgb85l08g2l63k99cw5dt2qpjkxb9
>>> 
>>> Thanks to all who have reviewed so far!
>>> 
>>> Cheers,
>>> David
>>> 

Re: [jira] [Created] (KAFKA-14257) Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response

[deng ziming]

Hello jianbin,
This error happens when the clusterIds are inconsistent among all kraft
voters, have you checked the cluster.id in your meta.properties(meta.properties
is generated using `kafka-storage.sh`) ?

--
Ziming

On Fri, Sep 23, 2022 at 9:41 AM jianbin.chen (Jira)  wrote:

> jianbin.chen created KAFKA-14257:
> 
>
>  Summary: Unexpected error INCONSISTENT_CLUSTER_ID in VOTE
> response
>  Key: KAFKA-14257
>  URL: https://issues.apache.org/jira/browse/KAFKA-14257
>  Project: Kafka
>   Issue Type: Bug
>   Components: kraft
> Affects Versions: 3.2.3
> Reporter: jianbin.chen
>
>
> Please help me see why the error message is output indefinitely
>
> broker1：
> {code:java}
> process.roles=broker,controller
>
> listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
> node.id=1
> listeners=PLAINTEXT://192.168.6.57:9092,CONTROLLER://192.168.6.57:9093
> inter.broker.listener.name=PLAINTEXT
> advertised.listeners=PLAINTEXT://192.168.6.57:9092
> controller.listener.names=CONTROLLER
> num.io.threads=8
> num.network.threads=5
> controller.quorum.voters=1@192.168.6.57:9093,2@192.168.6.56:9093,
> 3@192.168.6.55:9093
> log.dirs=/data01/kafka323-logs{code}
> broker2
> {code:java}
> process.roles=broker,controller
> controller.listener.names=CONTROLLER
> num.io.threads=8
> num.network.threads=5
>
> listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
> node.id=2
> listeners=PLAINTEXT://192.168.6.56:9092,CONTROLLER://192.168.6.56:9093
> inter.broker.listener.name=PLAINTEXT
> controller.quorum.voters=1@192.168.6.57:9093,2@192.168.6.56:9093,
> 3@192.168.6.55:9093
> log.dirs=/data01/kafka323-logs{code}
> broker3
> {code:java}
> process.roles=broker,controller
> controller.listener.names=CONTROLLER
> num.io.threads=8
> num.network.threads=5
> node.id=3
> listeners=PLAINTEXT://192.168.6.55:9092,CONTROLLER://192.168.6.55:9093
> inter.broker.listener.name=PLAINTEXT
> controller.quorum.voters=1@192.168.6.57:9093,2@192.168.6.56:9093,
> 3@192.168.6.55:9093
> log.dirs=/data01/kafka323-logs
>
> {code}
> error msg：
> {code:java}
> [2022-09-22 18:44:01,601] ERROR [RaftManager nodeId=2] Unexpected error
> INCONSISTENT_CLUSTER_ID in VOTE response:
> InboundResponse(correlationId=378, data=VoteResponseData(errorCode=104,
> topics=[]), sourceId=1) (org.apache.kafka.raft.KafkaRaftClient)
> [2022-09-22 18:44:01,625] ERROR [RaftManager nodeId=2] Unexpected error
> INCONSISTENT_CLUSTER_ID in VOTE response:
> InboundResponse(correlationId=380, data=VoteResponseData(errorCode=104,
> topics=[]), sourceId=1) (org.apache.kafka.raft.KafkaRaftClient)
> [2022-09-22 18:44:01,655] ERROR [RaftManager nodeId=2] Unexpected error
> INCONSISTENT_CLUSTER_ID in VOTE response:
> InboundResponse(correlationId=382, data=VoteResponseData(errorCode=104,
> topics=[]), sourceId=1) (org.apache.kafka.raft.KafkaRaftClient)
> [2022-09-22 18:44:01,679] ERROR [RaftManager nodeId=2] Unexpected error
> INCONSISTENT_CLUSTER_ID in VOTE response:
> InboundResponse(correlationId=384, data=VoteResponseData(errorCode=104,
> topics=[]), sourceId=1) (org.apache.kafka.raft.KafkaRaftClient)
> [2022-09-22 18:44:01,706] ERROR [RaftManager nodeId=2] Unexpected error
> INCONSISTENT_CLUSTER_ID in VOTE response:
> InboundResponse(correlationId=386, data=VoteResponseData(errorCode=104,
> topics=[]), sourceId=1) (org.apache.kafka.raft.KafkaRaftClient)
> [2022-09-22 18:44:01,729] ERROR [RaftManager nodeId=2] Unexpected error
> INCONSISTENT_CLUSTER_ID in VOTE response:
> InboundResponse(correlationId=388, data=VoteResponseData(errorCode=104,
> topics=[]), sourceId=1) (org.apache.kafka.raft.KafkaRaftClient){code}
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)
>

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

[deng ziming]

David,
Thanks for the feedback about #2 and #3, I'm OK with them.
I also mentioned the visibility in the MetadataShell in #1, do you have any
thoughts?

--
Best,
Ziming

On Wed, Sep 21, 2022 at 10:56 PM David Arthur  wrote:

> Ziming, thanks for the feedback! Let me know your thoughts on #2 and #3
>
> 1. Good idea. I consolidated all the details of record visibility into
> that section.
>
> 2. I'm not sure we can always know the number of records ahead of time
> for a transaction. One future use case is likely for the ZK data
> migration which will have an undetermined number of records. I would
> be okay with some short textual fields like "name" for the Begin
> record and "reason" for the Abort record. These could also be tagged
> fields if we don't want to always include them in the records.
>
> 3. The metadata records end up in org.apache.kafka.common.metadata, so
> maybe we can avoid Metadata in the name since it's kind of implicit.
> I'd be okay with [Begin|End|Abort]TransactionRecord.
>
> -David
>
> On Mon, Sep 19, 2022 at 10:58 PM deng ziming 
> wrote:
> >
> > Hello David,
> > Thanks for the KIP, certainly it makes sense, I left some minor
> questions.
> >
> > 1. In “Record Visibility” section you declare visibility in the
> controller, in “Broker Support” you mention visibility in the broker, we
> can put them together, and I think we can also describe visibility in the
> MetadataShell since it is also a public interface.
> >
> > 2. In “Public interfaces” section, I found that the “BeginMarkerRecord”
> has no fields, should we include some auxiliary attributes to help parse
> the transaction, for example, number of records in this transaction.
> >
> > 3. The record name seems vague, and we already have a
> `EndTransactionMarker` class in `org.apache.kafka.common.record`, how about
> `BeginMetadataTransactionRecord`?
> >
> > - -
> > Best,
> > Ziming
> >
> > > On Sep 10, 2022, at 1:13 AM, David Arthur 
> wrote:
> > >
> > > Starting a new thread to avoid issues with mail client threading.
> > >
> > > Original thread follows:
> > >
> > > Hey folks, I'd like to start a discussion on the idea of adding
> > > transactions in the KRaft controller. This will allow us to overcome
> > > the current limitation of atomic batch sizes in Raft which lets us do
> > > things like create topics with a huge number of partitions.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
> > >
> > > Thanks!
> > >
> > > ---
> > >
> > > Colin McCabe said:
> > >
> > > Thanks for this KIP, David!
> > >
> > > In the "motivation" section, it might help to give a concrete example
> > > of an operation we want to be atomic. My favorite one is probably
> > > CreateTopics since it's easy to see that we want to create all of a
> > > topic or none of it, and a topic could be a potentially unbounded
> > > number of records (although hopefully people have reasonable create
> > > topic policy classes in place...)
> > >
> > > In "broker support", it would be good to clarify that we will buffer
> > > the records in the MetadataDelta and not publish a new MetadataImage
> > > until the transaction is over. This is an implementation detail, but
> > > it's a simple one and I think it will make it easier to understand how
> > > this works.
> > >
> > > In the "Raft Transactions" section of "Rejected Alternatives," I'd add
> > > that managing buffering in the Raft layer would be a lot less
> > > efficient than doing it in the controller / broker layer. We would end
> > > up accumulating big lists of records which would then have to be
> > > applied when the transaction completed, rather than building up a
> > > MetadataDelta (or updating the controller state) incrementally.
> > >
> > > Maybe we want to introduce the concept of "last stable offset" to be
> > > the last committed offset that is NOT part of an ongoing transaction?
> > > Just a nomenclature suggestion...
> > >
> > > best,
> > > Colin
> >
>
>
> --
> David Arthur
>

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

[deng ziming]

Hello David,
Thanks for the KIP, certainly it makes sense, I left some minor questions.

1. In “Record Visibility” section you declare visibility in the controller, in 
“Broker Support” you mention visibility in the broker, we can put them 
together, and I think we can also describe visibility in the MetadataShell 
since it is also a public interface.

2. In “Public interfaces” section, I found that the “BeginMarkerRecord” has no 
fields, should we include some auxiliary attributes to help parse the 
transaction, for example, number of records in this transaction.

3. The record name seems vague, and we already have a `EndTransactionMarker` 
class in `org.apache.kafka.common.record`, how about 
`BeginMetadataTransactionRecord`?

- -
Best,
Ziming

> On Sep 10, 2022, at 1:13 AM, David Arthur  wrote:
> 
> Starting a new thread to avoid issues with mail client threading.
> 
> Original thread follows:
> 
> Hey folks, I'd like to start a discussion on the idea of adding
> transactions in the KRaft controller. This will allow us to overcome
> the current limitation of atomic batch sizes in Raft which lets us do
> things like create topics with a huge number of partitions.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
> 
> Thanks!
> 
> ---
> 
> Colin McCabe said:
> 
> Thanks for this KIP, David!
> 
> In the "motivation" section, it might help to give a concrete example
> of an operation we want to be atomic. My favorite one is probably
> CreateTopics since it's easy to see that we want to create all of a
> topic or none of it, and a topic could be a potentially unbounded
> number of records (although hopefully people have reasonable create
> topic policy classes in place...)
> 
> In "broker support", it would be good to clarify that we will buffer
> the records in the MetadataDelta and not publish a new MetadataImage
> until the transaction is over. This is an implementation detail, but
> it's a simple one and I think it will make it easier to understand how
> this works.
> 
> In the "Raft Transactions" section of "Rejected Alternatives," I'd add
> that managing buffering in the Raft layer would be a lot less
> efficient than doing it in the controller / broker layer. We would end
> up accumulating big lists of records which would then have to be
> applied when the transaction completed, rather than building up a
> MetadataDelta (or updating the controller state) incrementally.
> 
> Maybe we want to introduce the concept of "last stable offset" to be
> the last committed offset that is NOT part of an ongoing transaction?
> Just a nomenclature suggestion...
> 
> best,
> Colin

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[deng ziming]

Hi Igor,

I think this KIP can solve the current problems, I have some problems relating 
to the migration section.

Since we have bumped broker RPC version and metadata record version, there will 
be some problems between brokers/controllers of different versions. In ZK mode 
we use IBP as a flag to help solve this, in KRaft mode we use a feature 
flag(metadata.version) as a flag for using new RPC/metadata or not. 

Assuming that we are upgrading from 3.3 to 3.4, firstly the finalized version 
of metadata.version is 3.3, brokers will use version 1 of 
`BrokerRegistrationRequest` which contains no `OfflineLogDirectories`, finally 
the finalized version of metadata.version is 3.4, but brokers will no longer 
send `BrokerRegistrationRequest` unless we restart the broker, so controllers 
can’t be aware of the `OfflineLogDirectories` of each broker, so we should 
reconsider the suggestion of Jason to use `BrokerHeartbeatRequest` to 
communicate `OfflineLogDirectories`.

Of course this problem can be solved through a double roll(restart broker twice 
when upgrading), but we should trying to avoid it since we now have feature 
flag.

One solution is that we include `OfflineLogDirectories` both in 
`BrokerRegistrationRequest` and `BrokerHeartbeatRequest` requests, when 
upgrading from 3.3 to 3.4 the `BrokerRegistrationRequest.OfflineLogDirectories` 
is empty whereas when upgrading from 3.4 to 3.5 it will not be empty. And maybe 
we can also remove `LogDirectoriesOfflineRequest` you proposed in this KIP.

--
Best,
Ziming


> On Aug 18, 2022, at 11:24 PM, Igor Soarez  wrote:
> 
> Hi Ziming,
> 
> I'm sorry it took me a while to reply.
> 
> Thank you for having a look at this KIP and providing feedback.
> 
>> 1. We have a version field in meta.properties, currently it’s 1, and we can
>> set it to 2 in this KIP, and we can give an example of server.properties and
>> it’s corresponding meta.properties generated by the storage command tool.
> 
>> 2. When using storage format tool we should specify cluster-id, it will be an
>> arduous work if we should manually specify directory-ids for all log dirs,
>> I think you can make it more clear about this change that the directory-ids
>> are generated automatically.
> 
> Thank you for these suggestions. I've updated the KIP to:
> 
> * include an example
> * clarify that the version field would be changed to 2
> * clarify that the log directory UUIDs are automatically generated
> 
>> 3. When controller place a replica, it will select a broker as well as a log
>> directory, the latter is currently accomplished in the broker side, so this
>> will be a big change?
> 
> I think there can be benefits, as Jason described previously, if we change how
> log directories are assigned as follow-up work.
> 
> From a codebase surface area perspective, it is definitely a big change
> because there are many models, types and interfaces that assume replicas are
> identified solely by a broker id.
> That will have to change to include the directory UUID, many lines of code 
> will
> be affected.
> 
> But in terms of behavior it shouldn't be a big change at all. Brokers 
> currently
> select the log directory with the least logs in it. This isn't a very nice
> policy, as logs can have wildly different sizes, and log directories can have
> different capacities. But it is a policy that the controller can keep.
> 
> If we decide to extend the selection policy and keep it in the broker,
> the broker will continue to be able to override the controller's selection
> of log directory, using the `AssignReplicasToDirectories` RPC.
> 
>> 4. When handling log directory failures we will update Leader and ISR using
>> the existing replica state machine, what is this state machine referring to,
>> do you mean the AlterPartition RPC?
> 
> No, I mean that it will use the same rules and mechanism
> (`PartitionChangeBuilder`) that is used when a broker is fenced, shutdown or
> unregistered.
> 
> I think maybe the term "replica state machine" isn't the very appropriate.
> I've updated the KIP to rephrase this section.
> 
> Thanks,
> 
> --
> Igor

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[deng ziming]

Congrats, Sophie!

--
Best,
Ziming

> On Aug 9, 2022, at 3:20 PM, Bruno Cadonna  wrote:
> 
> Congrats, Sophie!

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[deng ziming]

Hi, Igor,
Thanks for this great work, left some questions,

1. We have a version field in meta.properties, currently it’s 1, and we can set 
it to 2 in this KIP, and we can give an example of server.properties and it’s 
corresponding meta.properties generated by the storage command tool.

2. When using storage format tool we should specify cluster-id, it will be an 
arduous work if we should manually specify directory-ids for all log dirs, I 
think you can make it more clear about this change that the directory-ids are 
generated automatically.

3. When controller place a replica, it will select a broker as well as a log 
directory, the latter is currently accomplished in the broker side, so this 
will be a big change?

4. When handling log directory failures we will update Leader and ISR using the 
existing replica state machine, what is this state machine referring to, do you 
mean the AlterPartition RPC?

--
Best,
Ziming


> On Aug 9, 2022, at 12:10 AM, Jason Gustafson  
> wrote:
> 
> Hi Igor,
> 
> Thanks for the KIP. It looks like it's on a good track. I have a few
> suggestions to throw into the mix:
> 
> 1. (nit): Instead of "storage id," maybe we could call it "directory id"?
> It seems a little clear since each log dir gets a unique id.
> 2. Rather than introducing a new RPC to communicate offline directories,
> would it be reasonable to add it to BrokerHeartbeat? My thinking is that we
> can let broker registration include the complete list of log dirs. The
> heartbeat can then be used to communicate online/offline status of each log
> dir.
> 3. I think we have an opportunity to consolidate normal partition
> reassignment and log dir reassignment here. Since we are modifying the
> `Assignment` to consist of both replica id and storage id, couldn't we make
> a similar change to the `AlterPartitionReassignment` API? Basically that
> would let us treat all reassignments as moving from one log dir to another.
> The case handled by `AlterReplicaLogDir` currently would just become a
> special case where the replica id happens to be the same. This might also
> let us get rid of all the custom logic surrounding JBOD, such as the
> additional fetcher, which has proven difficult to maintain.
> 
> Thanks,
> Jason
> 
> On Tue, Aug 2, 2022 at 2:24 AM Igor Soarez  wrote:
> 
>> Hi José,
>> 
>> Thanks for having a look at this KIP and thanks for pointing this out,
>> I've had a look at KIP-856.
>> 
>> It's good to see there's some overlap in our proposals. we're both
>> proposing:
>> 
>> - Identifying log directories with a UUID
>> - Extending the storage tool to ensure each log directory has a UUID
>> assigned
>> - Expanding the topic partition identity to include the log directory UUID
>> 
>> There were differences in our proposals as to how the UUID is to be
>> persisted, but I've changed my proposal to match yours — I think adding
>> storage.id to meta.properties makes sense.
>> 
>> --
>> Igor
>> 
>> 
>> On Wed, Jul 27, 2022, at 4:42 PM, José Armando García Sancio wrote:
>>> Hi Igor,
>>> 
>>> Thanks for the KIP. Looking forward to this improvement. I'll review
>> your KIP.
>>> 
>>> I should mention that I started a discussion thread on KIP-856: KRaft
>>> Disk Failure Recovery at
>>> https://lists.apache.org/thread/ytv0t18cplwwwqcp77h6vry7on378jzj
>>> 
>>> Both keep introducing similar concepts. For example both KIP introduce
>>> a storage uuid that is stored in meta.properties. At first glance
>>> there are some minor differences. I suggest that we review each
>>> other's KIP so that we can remove these minor differences. What do you
>>> think?
>>> 
>>> Thanks!
>>> --
>>> -José
>> 

Re: [ANNOUNCE] New Committer: Chris Egerton

[deng ziming]

Congratulations Chris !

--
Ziming

> On Jul 26, 2022, at 5:01 AM, Matthias J. Sax  wrote:
> 
> Congrats! Well deserved!
> 
> -Matthias
> 
> On 7/25/22 1:08 PM, Bill Bejeck wrote:
>> Congrats Chris!
>> -Bill
>> On Mon, Jul 25, 2022 at 3:58 PM Jorge Esteban Quilcate Otoya <
>> quilcate.jo...@gmail.com> wrote:
>>> Congratulations Chris!
>>> 
>>> On Mon, 25 Jul 2022 at 20:27, Robin Moffatt 
>>> wrote:
>>> 
 Congrats Chris!
 
 
 --
 
 Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |
>>> @rmoff
 
 
 On Mon, 25 Jul 2022 at 17:26, Mickael Maison 
>>> wrote:
 
> Hi all,
> 
> The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> we are excited to announce that he accepted!
> 
> Chris has been contributing to Kafka since 2017. He has made over 80
> commits mostly around Kafka Connect. His most notable contributions
> include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> Exactly-Once Support for Source Connectors.
> 
> He has been an active participant in discussions and reviews on the
> mailing lists and on Github.
> 
> Thanks for all of your contributions Chris. Congratulations!
> 
> -- Mickael, on behalf of the Apache Kafka PMC
> 
 
>>> 

Re: [VOTE] KIP-851: : Add requireStable flag into ListConsumerGroupOffsetsOptions

[deng ziming]

Thanks for this KIP,
we have a kafka-consumer-groups.sh shell which is based on the API you proposed 
to change, is it worth update it as well?

--
Best,
Ziming

> On Jul 1, 2022, at 9:04 AM, Guozhang Wang  wrote:
> 
> Hello folks,
> 
> I'd like to call out for a vote for the following KIP to expose the
> requireStable flag inside admin client's options as well:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-851%3A+Add+requireStable+flag+into+ListConsumerGroupOffsetsOptions
> 
> Any feedback as well as your votes are welcome.
> 
> -- Guozhang

[DISCUSS] KIP-849: Expose logdirs total and usable space via kafka-log-dirs.sh

[deng ziming]

Hi all,

I'd like to propose a small KIP to expose logdirs total and usable space via 
kafka-log-dirs.sh, most of the work has been finished in KIP-827, I just want 
to show these 2 fields in the output of afka-log-dirs.sh

Details can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-849%3A+Expose+logdirs+total+and+usable+space+via+kafka-log-dirs.sh
 


Any feedback is appreciated.

--
Thank you.
Ziming

Re: [VOTE] KIP-842: Add richer group offset reset mechanisms

[deng ziming]

Thank you for this KIP,
+1 (non-binding)

--
Ziming

> On Jun 15, 2022, at 8:54 PM, hudeqi <16120...@bjtu.edu.cn> wrote:
> 
> Hi all,
> 
> I'd like to start a vote on KIP-842to add some group offset reset mechanisms. 
> Details can be found here: https://cwiki.apache.org/confluence/x/xhyhD
> 
> Any feedback is appreciated.
> 
> Thank you.
> 
> hudeqi
> 
> 

Re: [VOTE] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[deng ziming]

Thank you for this KIP,

+1 (non-binding)

-- 
Best,
Ziming

> On Jun 7, 2022, at 8:53 PM, Alexandre Garnier  wrote:
> 
> Hi!
> 
> A little reminder to vote for this KIP.
> 
> Thanks.
> 
> 
> Le mer. 1 juin 2022 à 10:58, Alexandre Garnier  a écrit :
>> 
>> Hi everyone!
>> 
>> I propose to start voting for KIP-840:
>> https://cwiki.apache.org/confluence/x/bBqhD
>> 
>> Thanks,
>> --
>> Alex

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[deng ziming]

Thank you for your reply.

According to your description, strategy=nearest is redundant, right? We only 
rely on nearest.offset.reset=true/false to handle OffsetOutOfRange, I think we 
can directly remove this enum value, WDYT?

--
Best,
Ziming

> On May 27, 2022, at 5:19 PM, hudeqi <16120...@bjtu.edu.cn> wrote:
> 
> Thank you for your attention and reply. Here are my reply to your questions:
> 
> 1. If strategy=safe_latest and there is not committed offset, whether the 
> group is newly started is determined in this way: when the group is started, 
> a timestamp "createTimeStamp" will be passed as the start time of the group. 
> When the offset needs to be reset, the timestamp will be added to 
> "ListOffsetsRequest" as a new field. The server compares the timestamp 
> "createTimeStamp" with the timestamp "logStartTime", which is the first 
> message time for each partition. If "createTimeStamp" "greater than 
> "logStartTime" means that the group is newly started for this partition and 
> consumed from the latest, otherwise it means that the partition is newly 
> expanded and needs to be consumed from the earliest. For details, you can see 
> related jira and pr.
> 
> 
> 
> 2. Strictly speaking, nearest is not a level strategy with 
> earlyliest/latest/safe_latest/earliest_on_start/latest_on_start. It is more 
> like an auxiliary strategy, which is only dealt with out-of-range. So if you 
> set nearest.offset.reset=true, no matter what strategy "auto.offset.reset" is 
> set to, it will be performed according to the strategy of nearest when 
> out-of-range (to the earliest if it was under the range , or to the latest if 
> it was over the range), and for the case where no committed offset, nearest 
> will naturally have no effect, instead, it is determined by the main 
> strategy(auto.offset.reset).
> 
> 
> 
> 
> 3. This I have added to the form of module "Proposed Changes" in kip-842.
> 
> 
> 
> 
> 4. The meaning of nearest.offset.reset has been clearly expressed in point 2, 
> this configuration is disabled default, that is to say, when out-of-range, 
> reset strategy is performed according to the main strategy 
> (auto.offset.reset).
> 
> 
> 
>> -原始邮件-
>> 发件人: "deng ziming" 
>> 发送时间: 2022-05-27 16:02:53 (星期五)
>> 收件人: dev@kafka.apache.org
>> 抄送: 
>> 主题: Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms
>> 
>> Thank you for this KIP, the motivation makes sense to me, left some 
>> questions:
>> 
>> 1. If strategy=safe_latest and there is not committed offset, we have 2 
>> choices based on whether the group is started newly, can you elaborate on 
>> how can we decide the group is started newly? It would be clear.
>> 
>> 2. If strategy=nearest and there is not committed offset, its behavior is 
>> determined by the earliest, or latest, or safe_latest used together. can you 
>> elaborate on it more clearly?
>> 
>> 3. Can you also add a column "current reset behavior” and change "reset 
>> behavior” to “proposed reset behavior”, then we can be clear that this has 
>> no effect on current behavior.
>> 
>> 4. You added a new config “nearest.offset.reset” and only explain what will 
>> happen when we set it true, you’d better explain what will happen it it is 
>> false
>> 
>> --
>> Best,
>> Ziming
>> 
>> 
>>> On May 26, 2022, at 10:54 AM, 胡德祺 <16120...@bjtu.edu.cn> wrote:
>>> 
>>> Hi all,
>>> Why is no one talking about this?
>>> best
>>> hudeqi
>>> 
>>> 2022-05-23 17:45:53"胡德祺" <16120...@bjtu.edu.cn>写道：
>>> 
>>> Hi all,
>>> 
>>> I have written a new KIPto add some group offset reset mechanisms. Please 
>>> take a look here: https://cwiki.apache.org/confluence/x/xhyhD
>>> 
>>> besthudeqi
>> 
> 
> 

Re: [DISCUSS] KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

[deng ziming]

Thanks for the KIP, this is a good improvement. I only have one minor 
suggestion.

Currently many command line tools supports config file argument, but their name 
style is not unified, for example, most newly added tools are using 
--command-config, but ConsoleConsumer use —consumer.config。 I think we should 
unify the naming style from now on, I recommend us to use --reader-config and 
--formatter-config for the newly added arguments.

--
Best,
Ziming


> On May 26, 2022, at 4:36 PM, Alexandre Garnier  wrote:
> 
> Hello everyone,
> 
> Any feedback on this KIP https://cwiki.apache.org/confluence/x/bBqhD?
> It is a straightforward improvement without any impact on existing users,
> so not much to discuss besides maybe the option name.
> 
> -- 
> Alex
> 
> 
> Le mer. 18 mai 2022 à 10:44, Alexandre Garnier  a écrit :
> 
>> Hi everyone,
>> 
>> I created a KIP to add a config file option of reader/formatter for
>> kafka-console-(consumer|producer).sh tools.
>> https://cwiki.apache.org/confluence/x/bBqhD
>> 
>> Thanks for your feedback,
>> --
>> Alex
>> 

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[deng ziming]

Thank you for this KIP, the motivation makes sense to me, left some questions:

1. If strategy=safe_latest and there is not committed offset, we have 2 choices 
based on whether the group is started newly, can you elaborate on how can we 
decide the group is started newly? It would be clear.

2. If strategy=nearest and there is not committed offset, its behavior is 
determined by the earliest, or latest, or safe_latest used together. can you 
elaborate on it more clearly?

3. Can you also add a column "current reset behavior” and change "reset 
behavior” to “proposed reset behavior”, then we can be clear that this has no 
effect on current behavior.

4. You added a new config “nearest.offset.reset” and only explain what will 
happen when we set it true, you’d better explain what will happen it it is false

--
Best,
Ziming


> On May 26, 2022, at 10:54 AM, 胡德祺 <16120...@bjtu.edu.cn> wrote:
> 
> Hi all,
> Why is no one talking about this?
> best
> hudeqi
> 
> 2022-05-23 17:45:53"胡德祺" <16120...@bjtu.edu.cn>写道：
> 
> Hi all,
> 
> I have written a new KIPto add some group offset reset mechanisms. Please 
> take a look here: https://cwiki.apache.org/confluence/x/xhyhD
> 
> besthudeqi

Re: [VOTE] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[deng ziming]

+1(non-binding)

Thanks for this KIP.
--
Best,
Ziming

> On May 17, 2022, at 1:14 AM, Niket Goel  wrote:
> 
> Hi all,
> 
> I would like to start a vote for KIP-836:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Addition+of+Information+in+DescribeQuorumResponse+about+Voter+Lag
> 
> Thanks
> Niket Goel

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[deng ziming]

Hello Niket,


1. I find the DescribeQuorumResult still contains an 
DescribeQuorumResponseData, which is not allowed as Jose commented, have you 
forgot to change it?

2. You only add an Handle in AdminClient, can you also add an 
`kafka-metadata-quorum.sh` tool to help this?


> On May 17, 2022, at 9:50 AM, Niket Goel  wrote:
> 
> Thanks for the call out David. We will populate these fields for the
> Observers as well. I will clarify this in the KIP.
> 
> On Mon, May 16, 2022 at 1:50 PM David Arthur  wrote:
> 
>> Niket, thanks for the KIP!
>> 
>> Sorry for the late feedback on this, but I just had a quick question. The
>> KIP indicates the two new fields will be set for voters only, however this
>> ReplicaState struct is also used by the Observers in
>> DescribeQuorumResponse. Will we simply fill in -1 for these values, or do
>> we intend to report the last fetch and caught-up time of the observers as
>> well?
>> 
>> Thanks!
>> David
>> 
>> 
>> On Mon, May 16, 2022 at 1:46 PM Niket Goel 
>> wrote:
>> 
>>> Hi all,
>>> 
>>> Thank you for the feedback on this. I have started a voting thread for
>>> this KIP here:
>>> https://lists.apache.org/thread/bkb7gsbxpljh5qh014ztffq7bldjrb2x
>>> 
>>> Thanks
>>> Niket Goel
>>> 
>>> 
>>> From: Niket Goel 
>>> Date: Thursday, May 12, 2022 at 5:25 PM
>>> To: dev@kafka.apache.org 
>>> Subject: Re: [DISCUSS] KIP-836: Addition of Information in
>>> DescribeQuorumResponse about Voter Lag
>>> Appreciate the careful review Jose.!
>>> 
>>> Ack on 1 and 2. Will fix.
>>> 
>>> For number 3 (and I am using [1] as a reference for this discussion), I
>>> think the correct language to use would be:
>>> 
>>> "Whenever a new fetch request
>>> comes in the replica's last caught up time is updated to the time of
>>> this fetch request if it requests an offset greater than or equal to the
>>> leader's
>>> current end offset"
>>> Does that sound right now?
>>> 
>>> Although I think I will go ahead and rewrite the explanation in a way
>> that
>>> is more understandable. Thanks for pointing this out.
>>> 
>>> Thanks
>>> 
>>> [1]
>>> 
>> https://github.com/apache/kafka/blob/fa59be4e770627cd34cef85986b58ad7f606928d/core/src/main/scala/kafka/cluster/Replica.scala#L97
>>> 
>>> 
>>> 
>>> On Thu, May 12, 2022 at 3:20 PM José Armando García Sancio
>>>  wrote:
>>> Thanks for the Kafka improvement Niket.
>>> 
>>> 1. For the fields `LastFetchTime` and `LastCaughtUpTime`, Kafka tends
>>> to use the suffix "Timestamp" when the value is an absolute wall clock
>>> value.
>>> 
>>> 2. The method `result()` for the type `DescribeQuorumResult` returns
>>> the type `DescribeQuorumResponseData`. The types generated from the
>>> RPC JSON schema are internal to Kafka and not exposed to clients. For
>>> the admin client we should use a different type that is explicitly
>>> public. See `org.apache.kafka.client.admin.DescribeTopicsResult` for
>>> an example.
>>> 
>>> 3. The proposed section has his sentence "Whenever a new fetch request
>>> comes in the replica's last caught up time is updated to the time of
>>> the fetch request if it requests an offset greater than the leader's
>>> current end offset." Did you mean "previous fetch time" instead of
>>> "last caught up time"? What do you mean by "requests an offset greater
>>> than the leader's current end offset.?" Excluding diverging logs the
>>> follower fetch offset should never be greater than the leader LEO.
>>> 
>>> Thanks,
>>> -José
>>> 
>>> 
>>> --
>>> - Niket
>>> 
>> 
> 
> 
> -- 
> - Niket

Re: CI on Kafka seems to be failing even on trunk branch

[deng ziming]

You can help to solve this bug first if you are free, 
https://issues.apache.org/jira/browse/KAFKA-13907 
, since this bug can be 
reproduced locally, it won't be hard to investigated.

> On May 17, 2022, at 12:34 AM, Lim Qingwei  wrote:
> 
> Hi, I am a newbie.
> 
> I noticed Jenkins builds are failing, is this expected?
> My PR has some failures, but I don't think they are relevant, how should I
> proceed?
> 
> Can I still request for reviews?
> 
> Kind regards

Re: [VOTE] KIP-831: Add metric for log recovery progress

[deng ziming]

Hello Luke, thanks for this KIP,

+1 (non-binding)

--
Best,
Ziming

> On May 16, 2022, at 3:59 PM, Divij Vaidya  wrote:
> 
> +1 (non-binding)
> 
> 
> Divij Vaidya
> 
> 
> 
> On Mon, May 16, 2022 at 10:12 AM Luke Chen  wrote:
> 
>> Hi all,
>> 
>> I'd like to start a vote on KIP to expose metrics for log recovery
>> progress. These metrics would let the admins have a way to monitor the log
>> recovery progress.
>> 
>> Details can be found here:
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
>> 
>> Any feedback is appreciated.
>> 
>> Thank you.
>> Luke
>> 

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[deng ziming]

Hello Niket, currently DescribeQuorumResponse is not a public API, we don’t 
have a Admin api or shell script to get DescribeQuorumResponse, so it’s 
unnecessary to submit a KIP to change it, you can just submit a PR to 
accomplish this.

--
Thanks
Ziming

> On May 10, 2022, at 1:33 AM, Niket Goel  wrote:
> 
> Hi all,
> 
> I created a KIP to add some more information to `DesscribeQuorumResponse` to 
> enable ascertaining voter lag in the quorum a little better.
> Please see KIP -- 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Additional+Information+in+DescribeQuorumResponse+about+Voter+Lag
> 
> Thanks for your feedback,
> Niket Goel

Re: Request for pull request review

[deng ziming]

Hello Igor,
Thank you for driving this, currently we have more discussion about this, the 
main concern is that we don't want to add an RPC which will be 
O(num_partitions_in_dir), please see 
https://issues.apache.org/jira/browse/KAFKA-9837 
 for more details.

—
Thanks,
Ziming

> On Mar 9, 2022, at 8:53 PM, Igor Soarez  wrote:
> 
> Can someone review this pull request please?
> 
> https://github.com/apache/kafka/pull/9577
> 
> It has been open for quite some time, and it seems pinging folks directly on 
> GitHub has not worked.
> 
> Thanks,
> 
> --
> Igor

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[deng ziming]

Thank you Luke for this work,
I’m +1(non-binding)

--
Best,
Ziming Deng

> On Dec 1, 2021, at 8:36 AM, Luke Chen  wrote:
> 
> Hi all,
> 
> I'd like to start the vote for KIP-792: Add "generation" field into
> consumer protocol.
> 
> The goal of this KIP is to allow the assignor/consumer coordinator to have
> a way to identify the out-of-date members/assignments, to avoid rebalance
> stuck issues in current protocol.
> 
> Detailed description can be found here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> 
> Any feedback is welcome.
> 
> Thank you.
> Luke

Re: [DISCUSS] Apache Kafka 3.2.0 release

[deng ziming]

Hey Bruno,

Can we add KIP-815 to the plan? 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-815%3A++Support+max-timestamp+in+GetOffsetShell
 

The vote has passed just 2 days ago.

—
Thanks,
Ziming Deng

> On Mar 2, 2022, at 12:41 AM, Bruno Cadonna  wrote:
> 
> Hi all,
> 
> A quick reminder that KIP freeze for the Apache 3.2.0 is tomorrow. Please 
> make sure to close your votes if you want to add a KIP to the release plan.
> 
> Best,
> Bruno
> 
> On 15.02.22 12:37, Bruno Cadonna wrote:
>> Hi all,
>> I published a release plan for the Apache Kafka 3.2.0 release here:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.0
>> KIP Freeze: 2 March 2022
>> Feature Freeze: 16 March 2022
>> Code Freeze:30 March 2022
>> At least two weeks of stabilization will follow Code Freeze.
>> Please let me know if should add or remove KIPs from the plan or if you have 
>> any other objections.
>> Best,
>> Bruno
>> On 04.02.22 16:03, Bruno Cadonna wrote:
>>> Hi,
>>> 
>>> I'd like to volunteer to be the release manager for our next
>>> feature release, 3.2.0. If there are no objections, I'll send
>>> out the release plan soon.
>>> 
>>> Best,
>>> Bruno

Re: [VOTE] KIP-815: Support max-timestamp in GetOffsetShell

[deng ziming]

Hi all,

Since it’s a pretty minor KIP, I think we can pass the vote with:
- 4 +1(binding) votes (Luke, David, Mickael and John)

Thanks to all that participated in the discussion and voting,

-- 
Ziming Deng


> On Feb 22, 2022, at 2:56 PM, David Jacot  wrote:
> 
> For reference, here is the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-815%3A++Support+max-timestamp+in+GetOffsetShell
> 
> Thanks for the KIP! +1 (binding)
> 
> Best,
> David
> 
> Le mar. 22 févr. 2022 à 04:03, deng ziming  a
> écrit :
> 
>> Hey all, I'm starting the voting on KIP-815.
>> 
>> This supports a new OffsetSpec in GetOffsetShell so that we can easily
>> determine the offset and timestamp of the message with the largest
>> timestamp on a partition. This seems a simple change but replaced
>> KafkaConsumer with AdminClient in GetOffsetShell.
>> 
>> I recreated this vote thread since I changed the KIP title when
>> discussing.
>> 
>> Thanks,
>> Ziming Deng

Re: [VOTE] KIP-815: Support max-timestamp in GetOffsetShell

[deng ziming]

Hello Maison,
I think you suggestion makes sense, it’s better to support querying by 
"earliest" "latest" "max-timestamp”, I will add this to the KIP.


> On Feb 23, 2022, at 8:54 PM, Mickael Maison  wrote:
> 
> Hi,
> 
> I'm +1 (binding) too.
> 
> Just one minor comment:
> Could we make the "time" argument also accept "earliest", "latest" and
> "max-timestamp" alongside -1, -2, -3. I think it's a bit confusing to
> use these negative numbers to specify the desired timestamps.
> 
> Thanks
> 
> On Tue, Feb 22, 2022 at 8:05 AM Luke Chen  wrote:
>> 
>> Hi Ziming,
>> 
>> Thanks for the KIP!
>> I'm +1 (binding)
>> 
>> Luke
>> 
>> On Tue, Feb 22, 2022 at 2:56 PM David Jacot  wrote:
>> 
>>> For reference, here is the KIP:
>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-815%3A++Support+max-timestamp+in+GetOffsetShell
>>> 
>>> Thanks for the KIP! +1 (binding)
>>> 
>>> Best,
>>> David
>>> 
>>> Le mar. 22 févr. 2022 à 04:03, deng ziming  a
>>> écrit :
>>> 
>>>> Hey all, I'm starting the voting on KIP-815.
>>>> 
>>>> This supports a new OffsetSpec in GetOffsetShell so that we can easily
>>>> determine the offset and timestamp of the message with the largest
>>>> timestamp on a partition. This seems a simple change but replaced
>>>> KafkaConsumer with AdminClient in GetOffsetShell.
>>>> 
>>>> I recreated this vote thread since I changed the KIP title when
>>>> discussing.
>>>> 
>>>> Thanks,
>>>> Ziming Deng
>>> 

Re: [VOTE] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Thank you David, I started a new one and let’s close this.

Best,
Ziming Deng

> On Feb 21, 2022, at 5:31 PM, David Jacot  wrote:
> 
> Thanks for the update. LGTM.
> 
> The vote thread looks weird. Could you check or
> perhaps create a new one with the latest title?
> 
> Best,
> David
> 
> On Fri, Feb 11, 2022 at 2:57 PM deng ziming  wrote:
>> 
>> David, Thank you for pointing out this.
>> 
>> After some time, I sorted out the main configurations, the main difference 
>> are default.api.timeout.ms <http://default.api.timeout.ms/>, retries, 
>> request.timeout.ms <http://request.timeout.ms/>,metadata.max.age.ms 
>> <http://metadata.max.age.ms/>.  Please take a look again when you are free.
>> 
>>> On Feb 2, 2022, at 3:41 PM, David Jacot  wrote:
>>> 
>>> Hey,
>>> 
>>> Thanks for updating the KIP. I think that there are a few more configs
>>> which could
>>> be used. e.g. all the network related configs - they are in both
>>> consumer and admin
>>> configurations as well. Is `session.timeout.ms` relevant in our
>>> context? It does not
>>> seem to be used when querying offsets.
>>> 
>>> Regarding the usage of `request.timeout.ms` in the KafkaConsumer, it would 
>>> be
>>> great if we could be clearer in the KIP. When I read "Will use
>>> default.api.timeout.ms
>>> instead of request.timeout.ms , This is a small bug and will be fixed
>>> in a separate PR",
>>> it is not clear what will be fixed where. We could say that the 
>>> KafkaConsumer is
>>> inconsistent in its usage of the timeouts so the AdminClient will
>>> behave slightly
>>> differently. However, as it seems to be a bug, we will fix the Consumer and 
>>> we
>>> can add a link to the Jira.
>>> 
>>> It is important to get this section as clear as possible because this
>>> is where questions
>>> will be.
>>> 
>>> Cheers,
>>> David
>>> 
>>> On Wed, Feb 2, 2022 at 7:23 AM deng ziming  wrote:
>>>> 
>>>> Hey David,
>>>> I rechecked the ConsumerConfig and split the Compatibility section into 2 
>>>> sub-sections
>>>> regarding AdminClientConfig and ConsumerConfig, I also find a small bug 
>>>> that we use
>>>> different timeout in `KafkaConsumer.beginningOffsets` and 
>>>> `KafkaConsumer.endOffsets`,
>>>> I will fix this.
>>>> 
>>>> Please help to review the KIP and the bug, thank you.
>>>> 
>>>> Best,
>>>> Ziming Deng
>>>> 
>>>> 
>>>>> On Feb 1, 2022, at 6:31 PM, David Jacot  
>>>>> wrote:
>>>>> 
>>>>> Thanks for the updated KIP.
>>>>> 
>>>>> Regarding the compatibility section, I think that it would be
>>>>> great if we could really stress that the configurations that
>>>>> could reasonably be used to configure the tool are actually
>>>>> all supported by the admin client. Regarding the retry mechanism,
>>>>> the consumer will retry until `default.api.timeout.ms` is reached
>>>>> and it seems that the admin client does the same by default. Do
>>>>> you confirm this?
>>>>> 
>>>>> Best,
>>>>> David
>>>>> 
>>>>> On Mon, Jan 31, 2022 at 11:12 AM David Jacot  wrote:
>>>>>> 
>>>>>> Hey,
>>>>>> 
>>>>>> Thanks for the KIP. I have a few comments:
>>>>>> 
>>>>>> 1. I think that it would be better to name the KIP: "GetOffsetShell
>>>>>> should support max-timestamp"
>>>>>> or something like that as this is the initial intent of the change.
>>>>>> 
>>>>>> 2. There is a typo: `OffsetSpce` -> `OffsetSpec`.
>>>>>> 
>>>>>> 3. It would be great if we could further expand the compatibility
>>>>>> section. It seems that the number
>>>>>> of consumer configurations which could reasonably be used by
>>>>>> `GetOffsetShell` is quite small (timeout,
>>>>>> retries, etc.) and it seems that most of them (if not all) are
>>>>>> supported by the admin client as well. I wonder
>>>>>> if we could be explicit here and argue that the transition won't be
>>>>>> noticed. I might be speculating here.
>>>>>> 
>>>>>> 4. For completeness, I think that we should mention extending the
>>>>>> consumer to support max-timestamp
>>>>>> as well in the rejected alternatives. That would be another way to
>>>>>> address the issue. However, I agree
>>>>>> with you that using the admin client is better in the admin tools.
>>>>>> 
>>>>>> Best,
>>>>>> David
>>>>>> 
>>>>>> On Sun, Jan 30, 2022 at 2:09 PM deng ziming  
>>>>>> wrote:
>>>>>>> 
>>>>>>> Sorry all, I mean KIP-851 not KIP-734.
>>>>>>> In KIP-734 we add a new OffsetSpec to AdminClient, in this KIP I just 
>>>>>>> extend this OffsetSpec to GetOffsetShell.
>>>>>>> 
>>>>>>>> On Jan 30, 2022, at 6:29 PM, deng ziming  
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hey all, I'm starting the voting on KIP-734.
>>>>>>>> 
>>>>>>>> This supports a new OffsetSpec in GetOffsetShell so that we can easily 
>>>>>>>> determine the offset and timestamp of the message with the largest 
>>>>>>>> timestamp on a partition. This seems a simple change but replaced 
>>>>>>>> KafkaConsumer with AdminClient in GetOffsetShell.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Ziming Deng
>>>>>>> 
>>>> 
>> 

[VOTE] KIP-815: Support max-timestamp in GetOffsetShell

[deng ziming]

Hey all, I'm starting the voting on KIP-815.

This supports a new OffsetSpec in GetOffsetShell so that we can easily 
determine the offset and timestamp of the message with the largest timestamp on 
a partition. This seems a simple change but replaced KafkaConsumer with 
AdminClient in GetOffsetShell.

 I recreated this vote thread since I changed the KIP title when discussing.

Thanks,
Ziming Deng

Re: [VOTE] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

David, Thank you for pointing out this.

After some time, I sorted out the main configurations, the main difference are 
default.api.timeout.ms <http://default.api.timeout.ms/>, retries, 
request.timeout.ms <http://request.timeout.ms/>,metadata.max.age.ms 
<http://metadata.max.age.ms/>.  Please take a look again when you are free.

> On Feb 2, 2022, at 3:41 PM, David Jacot  wrote:
> 
> Hey,
> 
> Thanks for updating the KIP. I think that there are a few more configs
> which could
> be used. e.g. all the network related configs - they are in both
> consumer and admin
> configurations as well. Is `session.timeout.ms` relevant in our
> context? It does not
> seem to be used when querying offsets.
> 
> Regarding the usage of `request.timeout.ms` in the KafkaConsumer, it would be
> great if we could be clearer in the KIP. When I read "Will use
> default.api.timeout.ms
> instead of request.timeout.ms , This is a small bug and will be fixed
> in a separate PR",
> it is not clear what will be fixed where. We could say that the KafkaConsumer 
> is
> inconsistent in its usage of the timeouts so the AdminClient will
> behave slightly
> differently. However, as it seems to be a bug, we will fix the Consumer and we
> can add a link to the Jira.
> 
> It is important to get this section as clear as possible because this
> is where questions
> will be.
> 
> Cheers,
> David
> 
> On Wed, Feb 2, 2022 at 7:23 AM deng ziming  wrote:
>> 
>> Hey David,
>> I rechecked the ConsumerConfig and split the Compatibility section into 2 
>> sub-sections
>> regarding AdminClientConfig and ConsumerConfig, I also find a small bug that 
>> we use
>> different timeout in `KafkaConsumer.beginningOffsets` and 
>> `KafkaConsumer.endOffsets`,
>> I will fix this.
>> 
>> Please help to review the KIP and the bug, thank you.
>> 
>> Best,
>> Ziming Deng
>> 
>> 
>>> On Feb 1, 2022, at 6:31 PM, David Jacot  wrote:
>>> 
>>> Thanks for the updated KIP.
>>> 
>>> Regarding the compatibility section, I think that it would be
>>> great if we could really stress that the configurations that
>>> could reasonably be used to configure the tool are actually
>>> all supported by the admin client. Regarding the retry mechanism,
>>> the consumer will retry until `default.api.timeout.ms` is reached
>>> and it seems that the admin client does the same by default. Do
>>> you confirm this?
>>> 
>>> Best,
>>> David
>>> 
>>> On Mon, Jan 31, 2022 at 11:12 AM David Jacot  wrote:
>>>> 
>>>> Hey,
>>>> 
>>>> Thanks for the KIP. I have a few comments:
>>>> 
>>>> 1. I think that it would be better to name the KIP: "GetOffsetShell
>>>> should support max-timestamp"
>>>> or something like that as this is the initial intent of the change.
>>>> 
>>>> 2. There is a typo: `OffsetSpce` -> `OffsetSpec`.
>>>> 
>>>> 3. It would be great if we could further expand the compatibility
>>>> section. It seems that the number
>>>> of consumer configurations which could reasonably be used by
>>>> `GetOffsetShell` is quite small (timeout,
>>>> retries, etc.) and it seems that most of them (if not all) are
>>>> supported by the admin client as well. I wonder
>>>> if we could be explicit here and argue that the transition won't be
>>>> noticed. I might be speculating here.
>>>> 
>>>> 4. For completeness, I think that we should mention extending the
>>>> consumer to support max-timestamp
>>>> as well in the rejected alternatives. That would be another way to
>>>> address the issue. However, I agree
>>>> with you that using the admin client is better in the admin tools.
>>>> 
>>>> Best,
>>>> David
>>>> 
>>>> On Sun, Jan 30, 2022 at 2:09 PM deng ziming  
>>>> wrote:
>>>>> 
>>>>> Sorry all, I mean KIP-851 not KIP-734.
>>>>> In KIP-734 we add a new OffsetSpec to AdminClient, in this KIP I just 
>>>>> extend this OffsetSpec to GetOffsetShell.
>>>>> 
>>>>>> On Jan 30, 2022, at 6:29 PM, deng ziming  
>>>>>> wrote:
>>>>>> 
>>>>>> Hey all, I'm starting the voting on KIP-734.
>>>>>> 
>>>>>> This supports a new OffsetSpec in GetOffsetShell so that we can easily 
>>>>>> determine the offset and timestamp of the message with the largest 
>>>>>> timestamp on a partition. This seems a simple change but replaced 
>>>>>> KafkaConsumer with AdminClient in GetOffsetShell.
>>>>>> 
>>>>>> Thanks,
>>>>>> Ziming Deng
>>>>> 
>> 

Re: [ANNOUNCE] New committer: Luke Chen

[deng ziming]

Congratulations, Luke!

Thanks,
Ziming Deng

> On Feb 10, 2022, at 9:39 AM, John Roesler  wrote:
> 
> Congratulations, Luke!
> -John 
> 
> On Wed, Feb 9, 2022, at 19:33, Mayuresh Gharat wrote:
>> Congratulations Luke!
>> 
>> Thanks,
>> 
>> Mayuresh
>> 
>> On Wed, Feb 9, 2022, 5:24 PM Ismael Juma  wrote:
>> 
>>> Congratulations Luke!
>>> 
>>> On Wed, Feb 9, 2022 at 3:22 PM Guozhang Wang  wrote:
>>> 
 The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer
>>> and
 we are pleased to announce that he has accepted!
 
 Luke has been actively contributing to Kafka since early 2020. He has
 made more than 120 commits on various components of Kafka, with notable
 contributions to the rebalance protocol in Consumer and Streams (KIP-766,
 KIP-726, KIP-591, KAFKA-12675 and KAFKA12464, to just name a few), as
>>> well
 as making an impact on improving test stability of the project. Aside
>>> from
 all his code contributions, Luke has been a great participant in
 discussions across the board, a very active and helpful reviewer of other
 contributors' works, all of which are super valuable and highly
>>> appreciated
 by the community.
 
 
 Thanks for all of your contributions Luke. Congratulations!
 
 -- Guozhang, on behalf of the Apache Kafka PMC
 
>>> 

Re: [VOTE] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Hey David,
I rechecked the ConsumerConfig and split the Compatibility section into 2 
sub-sections
regarding AdminClientConfig and ConsumerConfig, I also find a small bug that we 
use
 different timeout in `KafkaConsumer.beginningOffsets` and 
`KafkaConsumer.endOffsets`,
I will fix this.

Please help to review the KIP and the bug, thank you.

Best,
Ziming Deng


> On Feb 1, 2022, at 6:31 PM, David Jacot  wrote:
> 
> Thanks for the updated KIP.
> 
> Regarding the compatibility section, I think that it would be
> great if we could really stress that the configurations that
> could reasonably be used to configure the tool are actually
> all supported by the admin client. Regarding the retry mechanism,
> the consumer will retry until `default.api.timeout.ms` is reached
> and it seems that the admin client does the same by default. Do
> you confirm this?
> 
> Best,
> David
> 
> On Mon, Jan 31, 2022 at 11:12 AM David Jacot  wrote:
>> 
>> Hey,
>> 
>> Thanks for the KIP. I have a few comments:
>> 
>> 1. I think that it would be better to name the KIP: "GetOffsetShell
>> should support max-timestamp"
>> or something like that as this is the initial intent of the change.
>> 
>> 2. There is a typo: `OffsetSpce` -> `OffsetSpec`.
>> 
>> 3. It would be great if we could further expand the compatibility
>> section. It seems that the number
>> of consumer configurations which could reasonably be used by
>> `GetOffsetShell` is quite small (timeout,
>> retries, etc.) and it seems that most of them (if not all) are
>> supported by the admin client as well. I wonder
>> if we could be explicit here and argue that the transition won't be
>> noticed. I might be speculating here.
>> 
>> 4. For completeness, I think that we should mention extending the
>> consumer to support max-timestamp
>> as well in the rejected alternatives. That would be another way to
>> address the issue. However, I agree
>> with you that using the admin client is better in the admin tools.
>> 
>> Best,
>> David
>> 
>> On Sun, Jan 30, 2022 at 2:09 PM deng ziming  wrote:
>>> 
>>> Sorry all, I mean KIP-851 not KIP-734.
>>> In KIP-734 we add a new OffsetSpec to AdminClient, in this KIP I just 
>>> extend this OffsetSpec to GetOffsetShell.
>>> 
>>>> On Jan 30, 2022, at 6:29 PM, deng ziming  wrote:
>>>> 
>>>> Hey all, I'm starting the voting on KIP-734.
>>>> 
>>>> This supports a new OffsetSpec in GetOffsetShell so that we can easily 
>>>> determine the offset and timestamp of the message with the largest 
>>>> timestamp on a partition. This seems a simple change but replaced 
>>>> KafkaConsumer with AdminClient in GetOffsetShell.
>>>> 
>>>> Thanks,
>>>> Ziming Deng
>>> 

[VOTE] KIP-815: Support max-timestamp in GetOffsetShell

[deng ziming]

Thank you David,
I retitled this KIP to be more accurate and supplemented the Compatibility and 
Rejected Alternatives sections, please help to review this again.

Best,
Ziming Deng

> On Jan 31, 2022, at 6:12 PM, David Jacot  wrote:
> 
> Hey,
> 
> Thanks for the KIP. I have a few comments:
> 
> 1. I think that it would be better to name the KIP: "GetOffsetShell
> should support max-timestamp"
> or something like that as this is the initial intent of the change.
> 
> 2. There is a typo: `OffsetSpce` -> `OffsetSpec`.
> 
> 3. It would be great if we could further expand the compatibility
> section. It seems that the number
> of consumer configurations which could reasonably be used by
> `GetOffsetShell` is quite small (timeout,
> retries, etc.) and it seems that most of them (if not all) are
> supported by the admin client as well. I wonder
> if we could be explicit here and argue that the transition won't be
> noticed. I might be speculating here.
> 
> 4. For completeness, I think that we should mention extending the
> consumer to support max-timestamp
> as well in the rejected alternatives. That would be another way to
> address the issue. However, I agree
> with you that using the admin client is better in the admin tools.
> 
> Best,
> David
> 
> On Sun, Jan 30, 2022 at 2:09 PM deng ziming  wrote:
>> 
>> Sorry all, I mean KIP-851 not KIP-734.
>> In KIP-734 we add a new OffsetSpec to AdminClient, in this KIP I just extend 
>> this OffsetSpec to GetOffsetShell.
>> 
>>> On Jan 30, 2022, at 6:29 PM, deng ziming  wrote:
>>> 
>>> Hey all, I'm starting the voting on KIP-734.
>>> 
>>> This supports a new OffsetSpec in GetOffsetShell so that we can easily 
>>> determine the offset and timestamp of the message with the largest 
>>> timestamp on a partition. This seems a simple change but replaced 
>>> KafkaConsumer with AdminClient in GetOffsetShell.
>>> 
>>> Thanks,
>>> Ziming Deng
>> 

Re: [VOTE] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Sorry all, I mean KIP-851 not KIP-734.
In KIP-734 we add a new OffsetSpec to AdminClient, in this KIP I just extend 
this OffsetSpec to GetOffsetShell.

> On Jan 30, 2022, at 6:29 PM, deng ziming  wrote:
> 
>  Hey all, I'm starting the voting on KIP-734.
> 
> This supports a new OffsetSpec in GetOffsetShell so that we can easily 
> determine the offset and timestamp of the message with the largest timestamp 
> on a partition. This seems a simple change but replaced KafkaConsumer with 
> AdminClient in GetOffsetShell.
> 
> Thanks,
> Ziming Deng

[VOTE] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

  Hey all, I'm starting the voting on KIP-734.

 This supports a new OffsetSpec in GetOffsetShell so that we can easily 
determine the offset and timestamp of the message with the largest timestamp on 
a partition. This seems a simple change but replaced KafkaConsumer with 
AdminClient in GetOffsetShell.

Thanks,
Ziming Deng

Re: [DISCUSS] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Hi everyone, I will start a voting progress soon if no body has more concerns.

> On Jan 25, 2022, at 1:04 PM, deng ziming  wrote:
> 
> Thank you Luke, I already changed the file name.
> 
>> On Jan 20, 2022, at 10:08 AM, Luke Chen  wrote:
>> 
>> Hi Ziming,
>> 
>> Thanks for the update! It looks good now.
>> 
>> Only 1 minor comment:
>> The file name in example can change to `kafka_admin_client.properties`,
>> which should be much clear.
>> 
>> Otherwise LGTM!
>> Thanks for the improvement!
>> 
>> Thank you.
>> Luke
> 

Re: [DISCUSS] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Thank you Luke, I already changed the file name.

> On Jan 20, 2022, at 10:08 AM, Luke Chen  wrote:
> 
> Hi Ziming,
> 
> Thanks for the update! It looks good now.
> 
> Only 1 minor comment:
> The file name in example can change to `kafka_admin_client.properties`,
> which should be much clear.
> 
> Otherwise LGTM!
> Thanks for the improvement!
> 
> Thank you.
> Luke

[DISCUSS] KIP-815: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Hi everyone,

I would like to  restart a discussion for KIP-815 since the old KIP number 
conflict with another KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-815%3A+Replace+KafkaConsumer+with+AdminClient+in+GetOffsetShell
 


The direct intention of this is to support max timestamp in GetOffsetShell, 
This seems like a simple change but there are some things to consider since it 
will change the --command-config parameter

Let me know what you think.


Thanks,
Ziming Deng

Re: [DISCUSS] KIP-813: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Thank you Daan for your reminders, I already changed the KIP number to KIP-815


> On Jan 17, 2022, at 4:39 PM, Daan Gertis  wrote:
> 
> Hey Deng,
> 
> Seems like we ran into a bit of a split-brain situation here. I just created 
> KIP-813 based on the last number in the wiki page and only now see your 
> proposal. Should have checked here as well, sorry.
> 
> Would you be able/willing to move to KIP-814 and register it on the KIP page 
> as well? I just finished linking it through the wiki so would be a bit work 
> to change it (but doable).
> 
> Tell me what you think!
> 
> Cheers,
> D.
> 
> From: deng ziming 
> Date: Friday, 14 January 2022 at 14:42
> To: dev@kafka.apache.org 
> Subject: [DISCUSS] KIP-813: Replace KafkaConsumer with AdminClient in 
> GetOffsetShell
> Hi everyone,
> 
> I would like to start a discussion for KIP-813
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-813%3A+Replace+KafkaConsumer+with+AdminClient+in+GetOffsetShell
>  
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-813:+Replace+KafkaConsumer+with+AdminClient+in+GetOffsetShell>
> 
> The direct intention of this is to support max timestamp in GetOffsetShell, 
> This seems like a simple change but there are some things to consider since 
> it will change the --command-config parameter
> 
> Let me know what you think.
> 
> 
> Thanks,
> Ziming Deng

Re: [DISCUSS] KIP-813: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Thank you Luke,

I fleshed my KIP according to your suggestions, mostly in the Public Interfaces 
section. 

Please take a review again.

Ziming Deng

> On Jan 15, 2022, at 11:09 AM, Luke Chen  wrote:
> 
> Hi Ziming,
> 
> Thanks for the KIP!
> It's good to support fetch max timestamp in GetOffsetShell.
> 
> Some comments to the KIP:
> 1. It might be good and clear to list (some) current options in
> `GetOffsetShell`, and the changes you want to make. You can refer to the
> KIP-635 here
> 
> .
> 2. It's good to add some examples to show how we can get the max timestamp.
> The examples in your PR is good start. You can add simple comment to
> explain what the command is doing. Ex:
> 
> # fetch max timestamp
> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --bootstrap-server
> localhost:9092 --topic topic1 --time -3
> topic1:0:9979
> 
> and add 1 or 2 examples to show how to use the changed `--command-config`
> config.
> 
> Thank you.
> Luke

[DISCUSS] KIP-813: Replace KafkaConsumer with AdminClient in GetOffsetShell

[deng ziming]

Hi everyone,

I would like to start a discussion for KIP-813
https://cwiki.apache.org/confluence/display/KAFKA/KIP-813%3A+Replace+KafkaConsumer+with+AdminClient+in+GetOffsetShell
 


The direct intention of this is to support max timestamp in GetOffsetShell, 
This seems like a simple change but there are some things to consider since it 
will change the --command-config parameter

Let me know what you think.


Thanks,
Ziming Deng

Re: [ANNOUNCE] New Kafka PMC member: David Jacot

[deng ziming]

Congrats David!

--
Ziming Deng

> On Dec 18, 2021, at 7:08 AM, Gwen Shapira  wrote:
> 
> Hi everyone,
> 
> David Jacot has been an Apache Kafka committer since Oct 2020 and has been 
> contributing to the community consistently this entire time - especially 
> notable the fact that he reviewed around 150 PRs in the last year. It is my 
> pleasure to announce that David agreed to join the Kafka PMC.
> 
> Congratulations, David!
> 
> Gwen Shapira, on behalf of Apache Kafka PMC

Re: [VOTE] KIP-778 KRaft upgrades

[deng ziming]

Hi, David

Looking forwarding to this feature

+1 (non-binding)

Thanks!

Ziming Deng

> On Dec 11, 2021, at 4:49 AM, David Arthur  wrote:
> 
> Hey everyone, I'd like to start a vote for KIP-778 which adds support for
> KRaft to KRaft upgrades.
> 
> Notably in this KIP is the first use case of KIP-584 feature flags. As
> such, there are some addendums to KIP-584 included.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-778%3A+KRaft+Upgrades
> 
> Thanks!
> David

Re: [DISCUSS] KIP-786: Use localhost:9092 as default bootstrap-server/broker-list in client tools

[deng ziming]

Thank you Maison

I have update the KIP number to KIP-789, the url has been updated to:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335433 
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335433>

Thanks

Deng Ziming

> On Oct 31, 2021, at 10:39 PM, Mickael Maison  wrote:
> 
> Hi,
> 
> Can you adjust the KIP number as there is already another KIP using 786?
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-786%3A+Emit+Metric+Client+Quota+Values
> 
> Thanks
> 
> On Mon, Oct 25, 2021 at 2:33 PM deng ziming  wrote:
>> 
>> Hey all,
>> I’d like to start the discussion for proposal, KIP-786: Use localhost:9092 
>> as default bootstrap-server/broker-list in client tools.
>> 
>> After this KIP, user can use client tools such as kafka-console-consumer.sh 
>> without specifying bootstrap-server.
>> 
>> Detailed information can be found here:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335433 
>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335433>
>> 
>> Any comments and feedback are welcome.
>> 
>> Thank you.
>> Deng Ziming.

[DISCUSS] KIP-786: Use localhost:9092 as default bootstrap-server/broker-list in client tools

[deng ziming]

Hey all,
I’d like to start the discussion for proposal, KIP-786: Use localhost:9092 as 
default bootstrap-server/broker-list in client tools.

After this KIP, user can use client tools such as kafka-console-consumer.sh 
without specifying bootstrap-server.

Detailed information can be found here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335433 
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335433>

Any comments and feedback are welcome.

Thank you.
Deng Ziming.

Re: [ANNOUNCE] New Kafka PMC Member: Chia-Ping Tsai

[deng ziming]

Congratulations Chia-Ping!

> On Mar 13, 2021, at 05:39, Sophie Blee-Goldman  
> wrote:
> 
> Congrats Chia-Ping! Thanks for all your contributions
> 
> On Fri, Mar 12, 2021 at 12:24 PM Mickael Maison 
> wrote:
> 
>> Congratulations Chia-Ping!
>> 
>> On Fri, Mar 12, 2021 at 7:54 PM Israel Ekpo  wrote:
>>> 
>>> Congrats @Chia-Ping!
>>> 
>>> On Fri, Mar 12, 2021 at 2:23 PM Guozhang Wang 
>> wrote:
>>> 
 Congratulations Chia-Ping! Really glad to have you on the PMC.
 
 
 Guozhang
 
 On Fri, Mar 12, 2021 at 11:14 AM Jun Rao 
>> wrote:
 
> Hi, Everyone,
> 
> Chia-Ping Tsai has been a Kafka committer since Oct. 15,  2020. He
>> has
 been
> very instrumental to the community since becoming a committer. It's
>> my
> pleasure to announce that Chia-Ping  is now a member of Kafka PMC.
> 
> Congratulations Chia-Ping!
> 
> Jun
> on behalf of Apache Kafka PMC
> 
 
 
 --
 -- Guozhang
 
>> 

Re: About Kafka 2.7.0 source code compilation error

[deng ziming]

Hello, please use gradlew, for example `./gradlew jar` `./gradlew idea`, or you 
can use the gradle plugin of IDEA. 
The `@ nowarn` warn seems to be related to different version of scala and jdk 
which you can just ignore.

> On Feb 18, 2021, at 16:38, 韩可 mailto:han...@cvicse.com>> 
> wrote:
> 
> Hello!
> 
> Recently, we need to build a compilation and development environment for 
> Kafka 2.7.0 source code. Now the source code can run successfully. However, 
> when we execute "gradle install" or "gradle build", we will report an error: 
> @ nowarn annotation does not suppress any warnings. The details are as 
> follows:
> 
> 
> 
> If you delete the @ nowarn annotation according to the location of the error, 
> you will still report other errors, so please find out if there is a good 
> solution, and hope to get your reply. Thank you!
> 
> The gradle version used is 6.6
> The scala version used is 2.13.4
> 

create KIP permission

[deng ziming]

Could you please grant me access to Create KIP's.  thank you, My ID is
 dengziming

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

[deng ziming]

Big +1 for your initiative and I have a question, we implement the
Raft protocol
just to be used in the management of metadata in Zookeeper or we will also
use it to replace the current logical of managing log-replica since the
algorithm we used to manage log-replica is analogous to Raft.


On Fri, Apr 17, 2020 at 7:45 AM Jason Gustafson  wrote:

> Hi All,
>
> I'd like to start a discussion on KIP-595:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum
> .
> This proposal specifies a Raft protocol to ultimately replace Zookeeper as
> documented in KIP-500. Please take a look and share your thoughts.
>
> A few minor notes to set the stage a little bit:
>
> - This KIP does not specify the structure of the messages used to represent
> metadata in Kafka, nor does it specify the internal API that will be used
> by the controller. Expect these to come in later proposals. Here we are
> primarily concerned with the replication protocol and basic operational
> mechanics.
> - We expect many details to change as we get closer to integration with
> the controller. Any changes we make will be made either as amendments to
> this KIP or, in the case of larger changes, as new proposals.
> - We have a prototype implementation which I will put online within the
> next week which may help in understanding some details. It has diverged a
> little bit from our proposal, so I am taking a little time to bring it in
> line. I'll post an update to this thread when it is available for review.
>
> Finally, I want to mention that this proposal was drafted by myself, Boyang
> Chen, and Guozhang Wang.
>
> Thanks,
> Jason
>

Re: Permission to create KIP

[deng ziming]

hi,Matthias
also add me,thank you, my wiki id is dengziming

On Tue, Dec 31, 2019 at 3:12 AM Matthias J. Sax 
wrote:

> Done.
>
> On 12/30/19 8:00 AM, Sagar wrote:
> > Hi,
> >
> > Its sagarmeansocean
> >
> > On Sat, Dec 28, 2019 at 12:29 AM Matthias J. Sax 
> > wrote:
> >
> >> What is your wiki account id?
> >>
> >>
> >> -Matthias
> >>
> >> On 12/27/19 7:12 AM, Sagar wrote:
> >>> Hi,
> >>>
> >>> I have done some work on adding prefix scan for state store and would
> >> like
> >>> to create a KIP for the same.
> >>>
> >>> Thanks!
> >>> Sagar.
> >>>
> >>
> >>
> >
>
>

Re: Group Metadata Manager on Broker 3]: Error in loading offsets from [__consumer_offsets,34] (kafka.coordinator.GroupMetadataManager) kafka.common.KafkaException: Unknown group metadata version 1

[deng ziming]

I once encountered the same problem when I installed 2 versions of Kafka
with the same config, and startup version A Kafka firstly and shutdown it,
and misoperated to start version B Kafka secondly and write some data to it
and shutdown it, and then startup version A Kafka.

I think it's OK to ignore it, maybe some multiple version altered the same
${log.dirs}

On Wed, Dec 4, 2019 at 11:31 PM Upendra Yadav  wrote:

> -- Forwarded message -
> From: Upendra Yadav 
> Date: Tue, Dec 3, 2019 at 5:28 PM
> Subject: Group Metadata Manager on Broker 3]: Error in loading offsets from
> [__consumer_offsets,34] (kafka.coordinator.GroupMetadataManager)
> kafka.common.KafkaException: Unknown group metadata version 1
> To: 
>
>
> Hi Team,
>
> I'm getting below exception in one of the kafka broker while restarting.
> But server is starting and serving properly.
>
> Why this error is coming?
> Is there any severe issue due to this?
> And how to resolve it?
>
> ERROR [Group Metadata Manager on Broker 3]: Error in loading offsets from
> [__consumer_offsets,34] (kafka.coordinator.GroupMetadataManager)
> kafka.common.KafkaException: Unknown group metadata version 1
> at
>
> kafka.coordinator.GroupMetadataManager$.schemaForGroup(GroupMetadataManager.scala:764)
> at
>
> kafka.coordinator.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:934)
> at
>
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:412)
> at
>
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:383)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at kafka.message.MessageSet.foreach(MessageSet.scala:71)
> at
>
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply$mcV$sp(GroupMetadataManager.scala:383)
> at
>
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
> at
>
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
> at
>
> kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1(GroupMetadataManager.scala:374)
> at
>
> kafka.coordinator.GroupMetadataManager$$anonfun$loadGroupsForPartition$1.apply$mcV$sp(GroupMetadataManager.scala:353)
> at
>
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>

[jenkins problem]

[deng ziming]

I always receive these kinds of messages about jenkins failed, and most prs
of Kafka build failed even just a trivial change，can someone inspects it.

[image: image.png]

[image: image.png]

Re: [issue assign]

[deng ziming]

my jira username is :
dengziming

thank you

On Fri, Nov 29, 2019 at 11:31 AM Gurudatt Kulkarni 
wrote:

> Share your jira username here.
>
> On Friday, November 29, 2019, deng ziming 
> wrote:
> > hello, I read the Contributing Code Changes page which says I can assign
> a ticket to myself but it turns out that I can't assign? how could I assign
> it to myself? thank you!
> >
> >
> >
>

[issue assign]

[deng ziming]

hello, I read the Contributing Code Changes

page
which says I can assign a ticket to myself but it turns out that I can't
assign? how could I assign it to myself? thank you!

[image: image.png]

Re: [jira] [Created] (KAFKA-9217) Partial partition's log-end-offset is zero

[deng ziming]

hello lisen,
>
> The amount of data my consumers consume is 400222
>
 do you consume all partitions from offset 0? if yes, this may probably be
a bug.

On Thu, Nov 21, 2019 at 3:22 PM lisen (Jira)  wrote:

> lisen created KAFKA-9217:
> 
>
>  Summary: Partial partition's log-end-offset is zero
>  Key: KAFKA-9217
>  URL: https://issues.apache.org/jira/browse/KAFKA-9217
>  Project: Kafka
>   Issue Type: Bug
>   Components: log
> Affects Versions: 0.10.0.1
>  Environment: kafka0.10.0.1
> Reporter: lisen
>  Fix For: 0.11.0.1
>
>
> The amount of data my consumers consume is 400222, But using the command
> to view the consumption results is only 279789, The command view results
> are as follows:
>
> !Snipaste_2019-11-21_15-00-09.png!
>
> The data result of partition 5 is
>
> !Snipaste_2019-11-21_14-53-06.png!
>
> I want to know if this is a kafka bug.Thanks.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

Re: [DISCUSS] KIP-547: Extend ConsumerInterceptor to allow modification of Consumer Commits

[deng ziming]

Hi, Eric
what's the use of this method? I reviewed the code and couldn't find much
of the metadata's usage, and I find its usage trivial.

On Tue, Nov 19, 2019 at 3:19 AM Eric Azama  wrote:

> Hi all,
>
> I'd like to open discussion on KIP-547: Extend ConsumerInterceptor to allow
> modification of Consumer Commits
>
> This KIP hopes to enable better access to the Metadata included while
> committing offsets.
>
> LINK:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-547%3A+Extend+ConsumerInterceptor+to+allow+modification+of+Consumer+Commits
>
>
> Thanks,
> Eric A.
>

Re: [DISCUSS] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[deng ziming]

I think it's ok, and you can add another issue about `asynchronous
metadata` if `topic expiry` is not enough.


On Thu, Nov 21, 2019 at 6:20 AM Brian Byrne  wrote:

> Hello all,
>
> I've refactored the KIP to remove implementing asynchronous metadata
> fetching in the producer during send(). It's now exclusively focused on
> reducing the topic metadata fetch payload and proposes adding a new
> configuration flag to control topic expiry behavior. Please take a look
> when possible.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-526
> %3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics
>
> Thanks,
> Brian
>
> On Fri, Oct 4, 2019 at 10:04 AM Brian Byrne  wrote:
>
> > Lucas, Guozhang,
> >
> > Thank you for the comments. Good point on METADATA_MAX_AGE_CONFIG - it
> > looks like the ProducerMetadata was differentiating between expiry and
> > refresh, but it should be unnecessary to do so once the cost of fetching
> a
> > single topic's metadata is reduced.
> >
> > I've updated the rejected alternatives and removed the config variables.
> >
> > Brian
> >
> > On Fri, Oct 4, 2019 at 9:20 AM Guozhang Wang  wrote:
> >
> >> Hello Brian,
> >>
> >> Thanks for the KIP.
> >>
> >> I think using asynchronous metadata update to address 1) metadata update
> >> blocking send, but for other issues, currently at producer we do have a
> >> configurable `METADATA_MAX_AGE_CONFIG` similar to consumer, by default
> is
> >> 5min. So maybe we do not need to introduce new configs here, but only
> >> change the semantics of that config from global expiry (today we just
> >> enforce a full metadata update for the whole cluster) to single-topic
> >> expiry, and we can also extend its expiry deadline whenever that
> metadata
> >> is successfully used to send a produce request.
> >>
> >>
> >> Guozhang
> >>
> >>
> >>
> >> On Thu, Oct 3, 2019 at 6:51 PM Lucas Bradstreet 
> >> wrote:
> >>
> >> > Hi Brian,
> >> >
> >> > This looks great, and should help reduce blocking and high metadata
> >> request
> >> > volumes when the producer is sending to large numbers of topics,
> >> especially
> >> > at low volumes. I think the approach to make metadata fetching
> >> asynchronous
> >> > and batch metadata requests together will help significantly.
> >> >
> >> > The only other approach I can think of is to allow users to supply the
> >> > producer with the expected topics upfront, allowing the producer to
> >> perform
> >> > a single initial metadata request before any sends occur. I see no
> real
> >> > advantages to this approach compared to the async method you’ve
> >> proposed,
> >> > but maybe we could add it to the rejected alternatives section?
> >> >
> >> > Thanks,
> >> >
> >> > Lucas
> >> >
> >> > On Fri, 20 Sep 2019 at 11:46, Brian Byrne 
> wrote:
> >> >
> >> > > I've updated the 'Proposed Changes' to include two new producer
> >> > > configuration variables: topic.expiry.ms and topic.refresh.ms.
> Please
> >> > take
> >> > > a look.
> >> > >
> >> > > Thanks,
> >> > > Brian
> >> > >
> >> > > On Tue, Sep 17, 2019 at 12:59 PM Brian Byrne 
> >> > wrote:
> >> > >
> >> > > > Dev team,
> >> > > >
> >> > > > Requesting discussion for improvement to the producer when dealing
> >> > with a
> >> > > > large number of topics.
> >> > > >
> >> > > > KIP:
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics
> >> > > >
> >> > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-8904
> >> > > >
> >> > > > Thoughts and feedback would be appreciated.
> >> > > >
> >> > > > Thanks,
> >> > > > Brian
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[deng ziming]

>
> For new (uncached) topics, one problem here is that we don't know which
> partition to map a record to in the event that it has a key or custom
> partitioner, so the RecordAccumulator wouldn't know which batch/broker it
> belongs. We'd need an intermediate record queue that subsequently moved the
> records into RecordAccumulators once metadata resolution was complete. For
> known topics, we don't currently block at all in waitOnMetadata.
>

You are right, I forget this fact, and the intermediate record queue will
help, but I have some questions

if we add an intermediate record queue in KafkaProducer, when should we
move the records into RecordAccumulators?
only NetworkClient is aware of the MetadataResponse, here is the
hierarchical structure of the related classes:
KafkaProducer
Accumulator
Sender
NetworkClient
metadataUpdater.handleCompletedMetadataResponse

so
1. we should also add a metadataUpdater to KafkaProducer?
2. if the topic really does not exists? the intermediate record queue will
become too large?
3. and should we `block` when the intermediate record queue is too large?
and this will again bring the blocking problem?



On Wed, Nov 20, 2019 at 12:40 AM Brian Byrne  wrote:

> Hi Deng,
>
> Thanks for the feedback.
>
> On Mon, Nov 18, 2019 at 6:56 PM deng ziming 
> wrote:
>
> > hi, I reviewed the current code, the ProduceMetadata maintains an expiry
> > threshold for every topic, every time when we write to a topic we will
> set
> > the expiry time to -1 to indicate it should be updated, this does work to
> > reduce the size of the topic working set, but the producer will continue
> > fetching metadata for these topics in every metadata request for the full
> > expiry duration.
> >
>
> Indeed, you are correct, I terribly misread the code here. Fortunately this
> was only a minor optimization in the KIP that's no longer necessary.
>
>
> and we can improve the situation by 2 means:
> > 1. we maintain a refresh threshold for every topic which is for
> example
> > 0.8 * expiry_threshold, and when we send `MetadataRequest` to brokers we
> > just request unknownLeaderTopics + unknownPartitionTopics + topics
> > reach refresh threshold.
> >
>
> Right, this is similar to what I suggested, with a larger window on the
> "staleness" that permits for batching to an appropriate size (except if
> there's any unknown topics, you'd want to issue the request immediately).
>
>
>
> > 2. we don't invoke KafkaProducer#waitOnMetadata when we call
> > KafkaProducer#send because of we just send data to RecordAccumulator, and
> > before we send data to brokers we will invoke RecordAccumulator#ready(),
> so
> > we can only invoke waitOnMetadata to block when (number topics
> > reach refresh threshold)>(number of all known topics)*0.2.
> >
>
> For new (uncached) topics, one problem here is that we don't know which
> partition to map a record to in the event that it has a key or custom
> partitioner, so the RecordAccumulator wouldn't know which batch/broker it
> belongs. We'd need an intermediate record queue that subsequently moved the
> records into RecordAccumulators once metadata resolution was complete. For
> known topics, we don't currently block at all in waitOnMetadata.
>
> The last major point of minimizing producer startup metadata RPCs may still
> need to be improved, but this would be a large improvement on the current
> situation.
>
> Thanks,
> Brian
>
>
>
> > I think the above 2 ways are enough to solve the current problem.
> >
> > On Tue, Nov 19, 2019 at 3:20 AM Colin McCabe  wrote:
> >
> > > On Mon, Nov 18, 2019, at 10:05, Brian Byrne wrote:
> > > > On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe 
> > wrote:
> > > >
> > > > > Two seconds doesn't seem like a reasonable amount of time to leave
> > for
> > > the
> > > > > metadata fetch.  Fetching halfway through the expiration period
> seems
> > > more
> > > > > reasonable.  It also doesn't require us to create a new
> configuration
> > > key,
> > > > > which is nice.
> > > > >
> > > > > Another option is to just do the metadata fetch every
> > > metadata.max.age.ms,
> > > > > but not expire the topic until we can't fetch the metadata for 2 *
> > > > > metadata.max.age.ms.
> > > > >
> > > >
> > > > I'd expect two seconds to be reasonable in the common case. Keep in
> > mind
> > > > that this doesn't affect correctness, and a control operation
> returning
> > >

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[deng ziming]

hi, I reviewed the current code, the ProduceMetadata maintains an expiry
threshold for every topic, every time when we write to a topic we will set
the expiry time to -1 to indicate it should be updated, this does work to
reduce the size of the topic working set, but the producer will continue
fetching metadata for these topics in every metadata request for the full
expiry duration.

and we can improve the situation by 2 means:
1. we maintain a refresh threshold for every topic which is for example
0.8 * expiry_threshold, and when we send `MetadataRequest` to brokers we
just request unknownLeaderTopics + unknownPartitionTopics + topics
reach refresh threshold.
2. we don't invoke KafkaProducer#waitOnMetadata when we call
KafkaProducer#send because of we just send data to RecordAccumulator, and
before we send data to brokers we will invoke RecordAccumulator#ready(), so
we can only invoke waitOnMetadata to block when (number topics
reach refresh threshold)>(number of all known topics)*0.2.

I think the above 2 ways are enough to solve the current problem.

On Tue, Nov 19, 2019 at 3:20 AM Colin McCabe  wrote:

> On Mon, Nov 18, 2019, at 10:05, Brian Byrne wrote:
> > On Fri, Nov 15, 2019 at 5:08 PM Colin McCabe  wrote:
> >
> > > Two seconds doesn't seem like a reasonable amount of time to leave for
> the
> > > metadata fetch.  Fetching halfway through the expiration period seems
> more
> > > reasonable.  It also doesn't require us to create a new configuration
> key,
> > > which is nice.
> > >
> > > Another option is to just do the metadata fetch every
> metadata.max.age.ms,
> > > but not expire the topic until we can't fetch the metadata for 2 *
> > > metadata.max.age.ms.
> > >
> >
> > I'd expect two seconds to be reasonable in the common case. Keep in mind
> > that this doesn't affect correctness, and a control operation returning
> > cached metadata should be on the order of milliseconds.
> >
>
> Hi Brian,
>
> Thanks again for the KIP.
>
> I think the issue here is not the common case, but the uncommon case where
> the metadata fetch takes longer than expected.  In that case, we don't want
> to be in the position of having our metadata expire because we waited too
> long to renew it.
>
> This is one reason why I think that the metadata expiration time should be
> longer than the metadata refresh time.  In fact, it might be worth having
> two separate configuration keys for these two values.  I could imagine a
> user who is having trouble with metadata expiration wanting to increase the
> metadata expiration time, but without increasing the metadata refresh
> period.  In a sense, the metadata expiration time is like the ZK session
> expiration time.  You might want to turn it up if the cluster is
> experiencing load spikes.
>
> >
> > But to the general
> > point, defining the algorithm would mean enforcing it to fair accuracy,
> > whereas if the suggestion is that it'll be performed at a reasonable
> time,
> > it allows for batching and other optimizations. Perhaps I shouldn't be
> > regarding what's defined in a KIP to be contractual in these cases, but
> you
> > could consider a first implementation to collect topics whose metadata
> has
> > exceeded (metadata.max.age.ms / 2), and sending the batch once a
> > constituent topic's metadata is near the expiry, or a sufficient number
> of
> > topics have been collected (10? 100? 1000?).
> >
>
> I'm concerned that if we change the metadata caching strategy without
> discussing it first, it may improve certain workloads but make others
> worse.  We need to be concrete about what the proposed strategy is so that
> we can really evaluate it.
>
> >
> >
> > > We should be specific about what happens if the first few metadata
> fetches
> > > fail.  Do we use exponential backoff to decide when to resend?  It
> seems
> > > like we really should, for all the usual reasons (reduce the load on
> > > brokers, ride out temporary service disruptions, etc.)  Maybe we could
> have
> > > an exponential retry backoff for each broker (in other words, we
> should try
> > > to contact a different broker before applying the backoff.)  I think
> this
> > > already sort of happens with the disconnect timeout, but we might need
> a
> > > more general solution.
> > >
> >
> > I don't plan to change this behavior. Currently it retries after a fixed
> > value of 'retry.backoff.ms' (defaults to 100 ms). It's possible that
> > different brokers are attempted, but I haven't dug into it.
> >
>
> I think it's critical to understand what the current behavior is before we
> try to change it.  The difference between retrying the same broker and
> trying a different one has a large impact it has on cluster load and
> latency.  For what it's worth, I believe the behavior is the second one,
> but it has been a while since I checked.  Let's figure this out.
>
> >
> > > Thanks for the clarification.  Fully asynchronous is the way to go, I
> > > agree.  I'm having trouble understanding how