[GitHub] [kafka] dajac commented on a diff in pull request #13666: KAFKA-14462; [13/N] CoordinatorEvent and CoordinatorEventProcessor

2023-05-30 Thread via GitHub
dajac commented on code in PR #13666: URL: https://github.com/apache/kafka/pull/13666#discussion_r1211172458 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/runtime/MultiThreadedEventProcessorTest.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apache So

[GitHub] [kafka] dajac commented on a diff in pull request #13666: KAFKA-14462; [13/N] CoordinatorEvent and CoordinatorEventProcessor

2023-05-30 Thread via GitHub
dajac commented on code in PR #13666: URL: https://github.com/apache/kafka/pull/13666#discussion_r1211171701 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/runtime/MultiThreadedEventProcessorTest.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apache So

[GitHub] [kafka] vamossagar12 commented on pull request #13504: KAFKA-14750: Check if topic exists in WorkerSinkTask when committing offsets.

2023-05-30 Thread via GitHub
vamossagar12 commented on PR #13504: URL: https://github.com/apache/kafka/pull/13504#issuecomment-1569597113 Thanks @dengziming for your confirmation. Regarding > one question is why only handle TimeoutException when getting offset The observation was that when a topic is delet

[GitHub] [kafka] dajac commented on a diff in pull request #13666: KAFKA-14462; [13/N] CoordinatorEvent and CoordinatorEventProcessor

2023-05-30 Thread via GitHub
dajac commented on code in PR #13666: URL: https://github.com/apache/kafka/pull/13666#discussion_r1211170527 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/runtime/MultiThreadedEventProcessorTest.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apache So

[GitHub] [kafka] dajac commented on a diff in pull request #13784: MINOR: Fix for MetadataQuorumCommandErrorTest.testRelativeTimeMs

2023-05-30 Thread via GitHub
dajac commented on code in PR #13784: URL: https://github.com/apache/kafka/pull/13784#discussion_r1211158842 ## tools/src/test/java/org/apache/kafka/tools/MetadataQuorumCommandErrorTest.java: ## @@ -53,12 +53,12 @@ public void testDescribeOptions() { @Test public voi

[GitHub] [kafka] vamossagar12 commented on a diff in pull request #13776: KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark

2023-05-30 Thread via GitHub
vamossagar12 commented on code in PR #13776: URL: https://github.com/apache/kafka/pull/13776#discussion_r1211158450 ## jmh-benchmarks/src/main/java/org/apache/kafka/jmh/connect/ReplaceFieldBenchmark.java: ## @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (AS

[GitHub] [kafka] dajac merged pull request #13639: KAFKA-14462; [12/N] Add GroupMetadataManager and ConsumerGroup

2023-05-30 Thread via GitHub
dajac merged PR #13639: URL: https://github.com/apache/kafka/pull/13639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

[GitHub] [kafka] fvaleri opened a new pull request, #13784: [MINOR] Fix for MetadataQuorumCommandErrorTest.testRelativeTimeMs

2023-05-30 Thread via GitHub
fvaleri opened a new pull request, #13784: URL: https://github.com/apache/kafka/pull/13784 This change should fix the following error on CI/CD. ```sh org.apache.kafka.common.KafkaException: Error while computing relative time, possible drift in system clock. Current timestamp is

[GitHub] [kafka] dajac commented on pull request #13738: KAFKA-14982: Improve the kafka-metadata-quorum output

2023-05-30 Thread via GitHub
dajac commented on PR #13738: URL: https://github.com/apache/kafka/pull/13738#issuecomment-1569434420 @fvaleri testRelativeTimeMs seems to fail regularly. There is an example here: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13782/1/tests. Could you t

[GitHub] [kafka] alok123t commented on a diff in pull request #13772: MINOR: Add helper util `Snapshots.lastContainedLogTimestamp`

2023-05-30 Thread via GitHub
alok123t commented on code in PR #13772: URL: https://github.com/apache/kafka/pull/13772#discussion_r1210872444 ## raft/src/main/java/org/apache/kafka/snapshot/Snapshots.java: ## @@ -151,4 +155,24 @@ public static Path markForDelete(Path logDir, OffsetAndEpoch snapshotId) {

[GitHub] [kafka] jeffkbkim commented on pull request #13267: KAFKA-14694: RPCProducerIdManager should not wait on new block

2023-05-30 Thread via GitHub
jeffkbkim commented on PR #13267: URL: https://github.com/apache/kafka/pull/13267#issuecomment-1569176269 test failure `testSendOffsetsToTransactionTimeout(String).quorum=zk – kafka.api.TransactionsTest` ``` org.apache.kafka.common.errors.TimeoutException: Timeout expired after

[GitHub] [kafka] cmccabe commented on a diff in pull request #13759: KAFKA-15019: Improve handling of broker heartbeat timeouts

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210858865 ## metadata/src/main/java/org/apache/kafka/controller/errors/ControllerExceptions.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [kafka] cmccabe commented on a diff in pull request #13759: KAFKA-15019: Improve handling of broker heartbeat timeouts

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210858865 ## metadata/src/main/java/org/apache/kafka/controller/errors/ControllerExceptions.java: ## @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] [kafka] jeffkbkim commented on a diff in pull request #13666: KAFKA-14462; [13/N] CoordinatorEvent and CoordinatorEventProcessor

2023-05-30 Thread via GitHub
jeffkbkim commented on code in PR #13666: URL: https://github.com/apache/kafka/pull/13666#discussion_r1210852722 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/runtime/MultiThreadedEventProcessorTest.java: ## @@ -0,0 +1,209 @@ +/* + * Licensed to the Apach

[GitHub] [kafka] rondagostino commented on a diff in pull request #13780: KAFKA-15039: Reduce logging level to trace in PartitionChangeBuilder.…

2023-05-30 Thread via GitHub
rondagostino commented on code in PR #13780: URL: https://github.com/apache/kafka/pull/13780#discussion_r1210847987 ## metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java: ## @@ -220,13 +220,15 @@ private boolean isValidNewLeader(int replica) { pr

[GitHub] [kafka] cmccabe commented on a diff in pull request #13759: KAFKA-15019: Improve handling of broker heartbeat timeouts

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210806096 ## core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala: ## @@ -1106,6 +1107,34 @@ class KRaftClusterTest { cluster.close() } } + + @Test

[GitHub] [kafka] cmccabe commented on a diff in pull request #13758: KAFKA-15010 ZK migration failover support

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13758: URL: https://github.com/apache/kafka/pull/13758#discussion_r1210817910 ## metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java: ## @@ -566,15 +578,46 @@ public void run() throws Exception {

[GitHub] [kafka] gharris1727 opened a new pull request, #13783: MINOR: Remove spurious warning about plugin.path config provider usage when null

2023-05-30 Thread via GitHub
gharris1727 opened a new pull request, #13783: URL: https://github.com/apache/kafka/pull/13783 If there is no plugin.path specified, this warning prints, and says `null` doesn't match `null`. This is misleading, and is caused by `rawPluginPath` being `null` and `transformedPluginPath` be

[GitHub] [kafka] cmccabe commented on a diff in pull request #13758: KAFKA-15010 ZK migration failover support

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13758: URL: https://github.com/apache/kafka/pull/13758#discussion_r1210814541 ## metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java: ## @@ -566,15 +578,46 @@ public void run() throws Exception {

[GitHub] [kafka] cmccabe commented on a diff in pull request #13758: KAFKA-15010 ZK migration failover support

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13758: URL: https://github.com/apache/kafka/pull/13758#discussion_r1210813001 ## metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationDriver.java: ## @@ -149,10 +152,9 @@ CompletableFuture migrationState() { } priv

[GitHub] [kafka] cmccabe commented on a diff in pull request #13758: KAFKA-15010 ZK migration failover support

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13758: URL: https://github.com/apache/kafka/pull/13758#discussion_r1210810667 ## core/src/main/scala/kafka/zk/ZkMigrationClient.scala: ## @@ -303,6 +304,14 @@ class ZkMigrationClient( new util.HashSet[Integer](zkClient.getSortedBrokerList.m

[GitHub] [kafka] jolshan merged pull request #13770: MINOR: Add transaction verification config to producerStateManager config

2023-05-30 Thread via GitHub
jolshan merged PR #13770: URL: https://github.com/apache/kafka/pull/13770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.

[GitHub] [kafka] cmccabe commented on a diff in pull request #13759: KAFKA-15019: Improve handling of broker heartbeat timeouts

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210806096 ## core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala: ## @@ -1106,6 +1107,34 @@ class KRaftClusterTest { cluster.close() } } + + @Test

[GitHub] [kafka] jolshan commented on pull request #13770: MINOR: Add transaction verification config to producerStateManager config

2023-05-30 Thread via GitHub
jolshan commented on PR #13770: URL: https://github.com/apache/kafka/pull/13770#issuecomment-1569075902 Checked the build -- streams upgrade tests had some issue that failed jdk 17 build. There were also two test failures -- [Build / JDK 8 and Scala 2.12 / org.apache.kafka.tools.

[GitHub] [kafka] rondagostino commented on a diff in pull request #13759: KAFKA-15019: Improve handling of broker heartbeat timeouts

2023-05-30 Thread via GitHub
rondagostino commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210720723 ## core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala: ## @@ -1106,6 +1107,34 @@ class KRaftClusterTest { cluster.close() } } + +

[GitHub] [kafka] Hangleton commented on pull request #13782: Suggest for performance fix: KAFKA-9693 Kafka latency spikes caused by log segment flush on roll - trunk version

2023-05-30 Thread via GitHub
Hangleton commented on PR #13782: URL: https://github.com/apache/kafka/pull/13782#issuecomment-1568955091 Many thanks for the patch and the collected data! Really interesting to see the impact of this change. A few questions: - What storage device and file system are used in the test

[jira] [Commented] (KAFKA-9693) Kafka latency spikes caused by log segment flush on roll

2023-05-30 Thread Ruslan Scherbakov (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727662#comment-17727662 ] Ruslan Scherbakov commented on KAFKA-9693: -- According to comments in PR in the G

[GitHub] [kafka] novosibman commented on pull request #13768: Suggest for performance fix: KAFKA-9693 Kafka latency spikes caused by log segment flush on roll

2023-05-30 Thread via GitHub
novosibman commented on PR #13768: URL: https://github.com/apache/kafka/pull/13768#issuecomment-1568910054 > We typically make changes to master first. Would you be willing to submit a PR for that instead? Prepared and tested trunk version: https://github.com/apache/kafka/pull/13782

[GitHub] [kafka] novosibman opened a new pull request, #13782: Suggest for performance fix: KAFKA-9693 Kafka latency spikes caused by log segment flush on roll - trunk version

2023-05-30 Thread via GitHub
novosibman opened a new pull request, #13782: URL: https://github.com/apache/kafka/pull/13782 Trunk version of initial change: https://github.com/apache/kafka/pull/13768 in branch "3.4" Key difference with branched change: Passed and used existing `scheduler` which already is being

[jira] [Comment Edited] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field

2023-05-30 Thread Yash Mayya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725223#comment-17725223 ] Yash Mayya edited comment on KAFKA-15012 at 5/30/23 6:34 PM: -

[jira] [Commented] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field

2023-05-30 Thread Yash Mayya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727653#comment-17727653 ] Yash Mayya commented on KAFKA-15012: Thanks [~ChrisEgerton], that's a fair point. I

[GitHub] [kafka] cmccabe commented on pull request #13759: KAFKA-15019: Improve handling of broker heartbeat timeouts

2023-05-30 Thread via GitHub
cmccabe commented on PR #13759: URL: https://github.com/apache/kafka/pull/13759#issuecomment-1568871804 Hi all, I've updated the approach in the PR. The new approach simply adds basic heartbeat handling in the timeout case. We can't do everything we normally can here. Most notably, a

[GitHub] [kafka] yashmayya commented on a diff in pull request #13776: KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark

2023-05-30 Thread via GitHub
yashmayya commented on code in PR #13776: URL: https://github.com/apache/kafka/pull/13776#discussion_r1210620016 ## connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ReplaceField.java: ## @@ -94,8 +96,8 @@ public void configure(Map configs) { {Con

[GitHub] [kafka] cadonna commented on a diff in pull request #13681: KAFKA-14133: Migrate ActiveTaskCreator mock in TaskManagerTest to Mockito

2023-05-30 Thread via GitHub
cadonna commented on code in PR #13681: URL: https://github.com/apache/kafka/pull/13681#discussion_r1210581130 ## streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java: ## @@ -3137,13 +3102,11 @@ public Set changelogPartitions() { }

[GitHub] [kafka] cmccabe commented on a diff in pull request #13759: KAFKA-15019: Improve handling of overload situations in the kcontroller

2023-05-30 Thread via GitHub
cmccabe commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210644585 ## metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java: ## @@ -223,6 +283,21 @@ public BrokerHeartbeatState next() { } } +

[GitHub] [kafka] yashmayya commented on a diff in pull request #13776: KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark

2023-05-30 Thread via GitHub
yashmayya commented on code in PR #13776: URL: https://github.com/apache/kafka/pull/13776#discussion_r1210628638 ## connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ReplaceField.java: ## @@ -94,8 +96,8 @@ public void configure(Map configs) { {Con

[GitHub] [kafka] jolshan commented on a diff in pull request #13666: KAFKA-14462; [13/N] CoordinatorEvent and CoordinatorEventProcessor

2023-05-30 Thread via GitHub
jolshan commented on code in PR #13666: URL: https://github.com/apache/kafka/pull/13666#discussion_r1210611398 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/runtime/CoordinatorEventProcessor.java: ## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Softwar

[GitHub] [kafka] yashmayya commented on a diff in pull request #13776: KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark

2023-05-30 Thread via GitHub
yashmayya commented on code in PR #13776: URL: https://github.com/apache/kafka/pull/13776#discussion_r1210628638 ## connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ReplaceField.java: ## @@ -94,8 +96,8 @@ public void configure(Map configs) { {Con

[GitHub] [kafka] yashmayya commented on a diff in pull request #13776: KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark

2023-05-30 Thread via GitHub
yashmayya commented on code in PR #13776: URL: https://github.com/apache/kafka/pull/13776#discussion_r1210622194 ## jmh-benchmarks/src/main/java/org/apache/kafka/jmh/connect/ReplaceFieldBenchmark.java: ## @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [kafka] novosibman commented on pull request #13768: Suggest for performance fix: KAFKA-9693 Kafka latency spikes caused by log segment flush on roll

2023-05-30 Thread via GitHub
novosibman commented on PR #13768: URL: https://github.com/apache/kafka/pull/13768#issuecomment-1568829724 > @novosibman What is the file system used in your test? OMB default should be `xfs` but wondering if it was changed Yes, used `xfs`. This fs type was found in OMB settings.

[GitHub] [kafka] hachikuji commented on a diff in pull request #13780: KAFKA-15039: Reduce logging level to trace in PartitionChangeBuilder.…

2023-05-30 Thread via GitHub
hachikuji commented on code in PR #13780: URL: https://github.com/apache/kafka/pull/13780#discussion_r1210611871 ## metadata/src/main/java/org/apache/kafka/controller/PartitionChangeBuilder.java: ## @@ -220,13 +220,15 @@ private boolean isValidNewLeader(int replica) { priva

[GitHub] [kafka] gharris1727 commented on pull request #13467: KAFKA-14863: Hide plugins with invalid constructors during plugin discovery

2023-05-30 Thread via GitHub
gharris1727 commented on PR #13467: URL: https://github.com/apache/kafka/pull/13467#issuecomment-1568821952 > Can you describe the impact on users with this change if a plugin with no valid constructor is present on the worker? Suppose you had a Connector without a valid constructor.

[jira] [Commented] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field

2023-05-30 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727631#comment-17727631 ] Chris Egerton commented on KAFKA-15012: --- [~yash.mayya] I'm wondering if compatibil

[GitHub] [kafka] dajac commented on pull request #13678: KAFKA-10337: await async commits in commitSync even if no offsets given

2023-05-30 Thread via GitHub
dajac commented on PR #13678: URL: https://github.com/apache/kafka/pull/13678#issuecomment-1568784867 @erikvanoosten I am sorry but I haven't had the time to get to it yet. Will do! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [kafka] dajac commented on a diff in pull request #13770: MINOR: Add config to producerStateManager config

2023-05-30 Thread via GitHub
dajac commented on code in PR #13770: URL: https://github.com/apache/kafka/pull/13770#discussion_r1210559849 ## storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManagerConfig.java: ## @@ -24,8 +24,11 @@ public class ProducerStateManagerConfig { publ

[GitHub] [kafka] zhangsm6 commented on pull request #13768: Suggest for performance fix: KAFKA-9693 Kafka latency spikes caused by log segment flush on roll

2023-05-30 Thread via GitHub
zhangsm6 commented on PR #13768: URL: https://github.com/apache/kafka/pull/13768#issuecomment-1568759449 @novosibman What is the file system used in your test? OMB default should be `xfs` but wondering if it was changed -- This is an automated message from the Apache Git Service. To respo

[GitHub] [kafka] jolshan commented on a diff in pull request #13770: MINOR: Add config to producerStateManager config

2023-05-30 Thread via GitHub
jolshan commented on code in PR #13770: URL: https://github.com/apache/kafka/pull/13770#discussion_r1210529153 ## storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManagerConfig.java: ## @@ -24,8 +24,11 @@ public class ProducerStateManagerConfig { pu

[GitHub] [kafka] C0urante commented on a diff in pull request #13776: KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark

2023-05-30 Thread via GitHub
C0urante commented on code in PR #13776: URL: https://github.com/apache/kafka/pull/13776#discussion_r1210481491 ## jmh-benchmarks/src/main/java/org/apache/kafka/jmh/connect/ReplaceFieldBenchmark.java: ## @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] [kafka] jeffkbkim commented on a diff in pull request #13267: KAFKA-14694: RPCProducerIdManager should not wait on new block

2023-05-30 Thread via GitHub
jeffkbkim commented on code in PR #13267: URL: https://github.com/apache/kafka/pull/13267#discussion_r1210489030 ## core/src/test/scala/unit/kafka/coordinator/transaction/ProducerIdManagerTest.scala: ## @@ -113,38 +142,113 @@ class ProducerIdManagerTest { } @Parameterize

[GitHub] [kafka] rondagostino commented on pull request #13759: KAFKA-15019: Improve handling of overload situations in the kcontroller

2023-05-30 Thread via GitHub
rondagostino commented on PR #13759: URL: https://github.com/apache/kafka/pull/13759#issuecomment-1568668239 > The 3 missed heartbeats in your example would result in 3 brokers being fenced by the time the overload state is entered? Or did I misunderstand? And what happens next? Usin

[jira] [Commented] (KAFKA-10897) kafka quota optimization

2023-05-30 Thread Afshin Moazami (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727591#comment-17727591 ] Afshin Moazami commented on KAFKA-10897: Hey team,  I am new to the community. 

[GitHub] [kafka] rondagostino commented on a diff in pull request #13759: KAFKA-15019: Improve handling of overload situations in the kcontroller

2023-05-30 Thread via GitHub
rondagostino commented on code in PR #13759: URL: https://github.com/apache/kafka/pull/13759#discussion_r1210468300 ## metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java: ## @@ -223,6 +283,21 @@ public BrokerHeartbeatState next() { } }

[GitHub] [kafka] clolov commented on pull request #13421: KAFKA-14824: ReplicaAlterLogDirsThread may cause serious disk growing in case of potential exception

2023-05-30 Thread via GitHub
clolov commented on PR #13421: URL: https://github.com/apache/kafka/pull/13421#issuecomment-1568651608 Okay, in principle I agree that cleaning should be restarted, but I do not understand why you want to mark the topicPartition as failed ``` override def markPartitionFailed(topicParti

[GitHub] [kafka] jsancio commented on a diff in pull request #13772: MINOR: Add helper util `Snapshots.lastContainedLogTimestamp`

2023-05-30 Thread via GitHub
jsancio commented on code in PR #13772: URL: https://github.com/apache/kafka/pull/13772#discussion_r1210417483 ## raft/src/main/java/org/apache/kafka/snapshot/Snapshots.java: ## @@ -151,4 +155,24 @@ public static Path markForDelete(Path logDir, OffsetAndEpoch snapshotId) {

[jira] [Commented] (KAFKA-15039) Reduce logging level to trace in PartitionChangeBuilder.tryElection()

2023-05-30 Thread Divij Vaidya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727555#comment-17727555 ] Divij Vaidya commented on KAFKA-15039: -- Thank you for the find. Could you please sh

[GitHub] [kafka] rondagostino opened a new pull request, #13780: KAFKA-15039: Reduce logging level to trace in PartitionChangeBuilder.…

2023-05-30 Thread via GitHub
rondagostino opened a new pull request, #13780: URL: https://github.com/apache/kafka/pull/13780 …tryElection() A CPU profile in a large cluster showed PartitionChangeBuilder.tryElection() taking significant CPU due to logging. Decrease the logging statements in that method from debug

[jira] [Created] (KAFKA-15039) Reduce logging level to trace in PartitionChangeBuilder.tryElection()

2023-05-30 Thread Ron Dagostino (Jira)
Ron Dagostino created KAFKA-15039: - Summary: Reduce logging level to trace in PartitionChangeBuilder.tryElection() Key: KAFKA-15039 URL: https://issues.apache.org/jira/browse/KAFKA-15039 Project: Kafk

[GitHub] [kafka] divijvaidya commented on pull request #13779: KAFKA-15037: pass remoteLogEnabled to unifiedLog

2023-05-30 Thread via GitHub
divijvaidya commented on PR #13779: URL: https://github.com/apache/kafka/pull/13779#issuecomment-1568472012 For point 2, I think @satishd has some refactoring in mind to add more logic to UnifiedLog as per https://github.com/apache/kafka/pull/13561#discussion_r1181520975 > all this

[jira] [Resolved] (KAFKA-14970) Dual write mode testing for SCRAM and Quota

2023-05-30 Thread Proven Provenzano (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Proven Provenzano resolved KAFKA-14970. --- Resolution: Fixed Committed and merged into 3.5. > Dual write mode testing for SCRA

[GitHub] [kafka] showuon commented on pull request #13779: KAFKA-15037: pass remoteLogEnabled to unifiedLog

2023-05-30 Thread via GitHub
showuon commented on PR #13779: URL: https://github.com/apache/kafka/pull/13779#issuecomment-1568352508 @satishd , please take a look. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Commented] (KAFKA-9693) Kafka latency spikes caused by log segment flush on roll

2023-05-30 Thread Divij Vaidya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727500#comment-17727500 ] Divij Vaidya commented on KAFKA-9693: - Thank you for the analysis. May I perhaps pro

[GitHub] [kafka] Hangleton commented on a diff in pull request #13493: KAFKA-14852: Propagate Topic Ids to the Group Coordinator for Offset Fetch

2023-05-30 Thread via GitHub
Hangleton commented on code in PR #13493: URL: https://github.com/apache/kafka/pull/13493#discussion_r1210141583 ## core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala: ## @@ -492,42 +492,59 @@ class GroupMetadataManager(brokerId: Int, * The most important

[jira] [Updated] (KAFKA-15038) Use topic id/name mapping from the Metadata cache in RLM

2023-05-30 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez updated KAFKA-15038: -- Component/s: core > Use topic id/name mapping from the Metadata cache in RLM > ---

[jira] [Updated] (KAFKA-15038) Use topic id/name mapping from the Metadata cache in the RemoteLogManager

2023-05-30 Thread Alexandre Dupriez (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez updated KAFKA-15038: -- Summary: Use topic id/name mapping from the Metadata cache in the RemoteLogManager (w

[jira] [Created] (KAFKA-15038) Use topic id/name mapping from the Metadata cache in RLM

2023-05-30 Thread Alexandre Dupriez (Jira)
Alexandre Dupriez created KAFKA-15038: - Summary: Use topic id/name mapping from the Metadata cache in RLM Key: KAFKA-15038 URL: https://issues.apache.org/jira/browse/KAFKA-15038 Project: Kafka

[GitHub] [kafka] showuon commented on a diff in pull request #13779: KAFKA-15037: pass remoteLogEnabled to unifiedLog

2023-05-30 Thread via GitHub
showuon commented on code in PR #13779: URL: https://github.com/apache/kafka/pull/13779#discussion_r1210048015 ## storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/ConsumerTask.java: ## @@ -159,7 +159,7 @@ public void run() { while (!closing)

[GitHub] [kafka] showuon opened a new pull request, #13779: KAFKA-15037: pass remoteLogEnabled to unifiedLog

2023-05-30 Thread via GitHub
showuon opened a new pull request, #13779: URL: https://github.com/apache/kafka/pull/13779 UnifiedLog relied on the `remoteStorageSystemEnable` to identify if the broker is enabling remote storage, but we never pass this value from the config into UnifiedLog. So it'll always be false.

[jira] [Created] (KAFKA-15037) initialize unifiedLog with remoteStorageSystemEnable correctly

2023-05-30 Thread Luke Chen (Jira)
Luke Chen created KAFKA-15037: - Summary: initialize unifiedLog with remoteStorageSystemEnable correctly Key: KAFKA-15037 URL: https://issues.apache.org/jira/browse/KAFKA-15037 Project: Kafka Iss

[jira] [Commented] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors

2023-05-30 Thread Sagar Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727444#comment-17727444 ] Sagar Rao commented on KAFKA-15018: --- Thanks [~yash.mayya] . I have assigned this to my

[GitHub] [kafka] clolov commented on pull request #13764: KAFKA-14691; [1/N] Add new fields to OffsetFetchRequest and OffsetFetchResponse

2023-05-30 Thread via GitHub
clolov commented on PR #13764: URL: https://github.com/apache/kafka/pull/13764#issuecomment-1568117181 > I have a few comments/questions: > > 1. I am not really comfortable with merging this without the server side implementation. @clolov Is there a strong reason to not do them to

[jira] [Assigned] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors

2023-05-30 Thread Sagar Rao (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Rao reassigned KAFKA-15018: - Assignee: Sagar Rao > Potential tombstone offsets corruption for exactly-once source connectors

[jira] [Commented] (KAFKA-14981) Set `group.instance.id` in streams consumer so that rebalance will not happen if a instance is restarted

2023-05-30 Thread Bruno Cadonna (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727427#comment-17727427 ] Bruno Cadonna commented on KAFKA-14981: --- I have the same feeling as [~ableegoldman

[jira] [Updated] (KAFKA-15010) KRaft Controller doesn't reconcile with Zookeeper metadata upon becoming new controller while in dual write mode.

2023-05-30 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison updated KAFKA-15010: --- Fix Version/s: 3.5.0 > KRaft Controller doesn't reconcile with Zookeeper metadata upon becom

[GitHub] [kafka] soarez commented on a diff in pull request #13777: KAFKA-15036: UnknownServerError on any leader failover

2023-05-30 Thread via GitHub
soarez commented on code in PR #13777: URL: https://github.com/apache/kafka/pull/13777#discussion_r1209972559 ## metadata/src/main/java/org/apache/kafka/controller/QuorumController.java: ## @@ -2028,8 +2028,11 @@ public CompletableFuture finalizedFeatures( if (lastComm

[GitHub] [kafka] akatona84 closed pull request #13733: KAFKA-13337: Plugin loader error handling is done separately, per plugin

2023-05-30 Thread via GitHub
akatona84 closed pull request #13733: KAFKA-13337: Plugin loader error handling is done separately, per plugin URL: https://github.com/apache/kafka/pull/13733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [kafka] akatona84 commented on pull request #13733: KAFKA-13337: Plugin loader error handling is done separately, per plugin

2023-05-30 Thread via GitHub
akatona84 commented on PR #13733: URL: https://github.com/apache/kafka/pull/13733#issuecomment-1568053208 PR #13334 addresses my concerns, if I'm not mistaking. It was refactored in a way that now it's loading other plugin dirs in case of errors popping up at one. Closing this PR. --

[GitHub] [kafka] soarez commented on a diff in pull request #13558: KAFKA-14845: Fix broker registration with Zookeeper when the previous ephemeral znode was not properly recorded by the broker

2023-05-30 Thread via GitHub
soarez commented on code in PR #13558: URL: https://github.com/apache/kafka/pull/13558#discussion_r1209954592 ## checkstyle/suppressions.xml: ## @@ -41,6 +41,8 @@ + Review Comment: Makes sense. Thanks! -- This is an automated message from the Apache Git S

[GitHub] [kafka] dajac commented on pull request #13765: KAFKA-15021; Skip leader epoch bump on ISR shrink

2023-05-30 Thread via GitHub
dajac commented on PR #13765: URL: https://github.com/apache/kafka/pull/13765#issuecomment-1568017910 > Yes, that is fair too. The definition of leadership epoch in which case changes to - it represents the version of the leader after a re-election. In this case, we should also remove the e

[GitHub] [kafka] dajac merged pull request #13754: MINOR: Hopefully fix flaky FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor

2023-05-30 Thread via GitHub
dajac merged PR #13754: URL: https://github.com/apache/kafka/pull/13754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

[GitHub] [kafka] dajac commented on pull request #13765: KAFKA-15021; Skip leader epoch bump on ISR shrink

2023-05-30 Thread via GitHub
dajac commented on PR #13765: URL: https://github.com/apache/kafka/pull/13765#issuecomment-1568009841 > This change also fixes a bug when computing the HWM. When computing the HWM, replicas that are not eligible to join the ISR but are caught up should not be included in the computation. Ot

[GitHub] [kafka] dajac commented on pull request #13765: KAFKA-15021; Skip leader epoch bump on ISR shrink

2023-05-30 Thread via GitHub
dajac commented on PR #13765: URL: https://github.com/apache/kafka/pull/13765#issuecomment-1568001597 > To achieve the objective that you desired, there is another way without changing the definition, i.e. change how the components reacts when the version/epoch is changes. We can choose to

[GitHub] [kafka] mimaison commented on pull request #13748: [BUGFIX] Bugfixed in KAFKA-8713, but it doesn't work properly.

2023-05-30 Thread via GitHub
mimaison commented on PR #13748: URL: https://github.com/apache/kafka/pull/13748#issuecomment-1567999823 @gharris1727 If @krespo is not able to update the PR, can you open one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [kafka] divijvaidya commented on pull request #13778: Feature/shuai add comment

2023-05-30 Thread via GitHub
divijvaidya commented on PR #13778: URL: https://github.com/apache/kafka/pull/13778#issuecomment-1567994161 Duplicate of https://github.com/apache/kafka/pull/13730 @DengShuaiSimon As I mentioned in the previous PR, unfortunately, we cannot merge this change. As per ASF code of conduc

[GitHub] [kafka] divijvaidya commented on pull request #13765: KAFKA-15021; Skip leader epoch bump on ISR shrink

2023-05-30 Thread via GitHub
divijvaidya commented on PR #13765: URL: https://github.com/apache/kafka/pull/13765#issuecomment-1567981427 Hey @jsancio With this change, we are changing the semantics of what a leadership epoch means. Prior to this change, leadership epoch is a version number representing membership o

[GitHub] [kafka] mimaison commented on a diff in pull request #13775: MINOR: update release.py

2023-05-30 Thread via GitHub
mimaison commented on code in PR #13775: URL: https://github.com/apache/kafka/pull/13775#discussion_r1209862688 ## release.py: ## @@ -698,7 +700,7 @@ def select_gpg_key(): email_contents = """ -To: d...@kafka.apache.org, us...@kafka.apache.org, kafka-clie...@googlegroups.c

[GitHub] [kafka] dajac commented on a diff in pull request #13770: MINOR: Add config to producerStateManager config

2023-05-30 Thread via GitHub
dajac commented on code in PR #13770: URL: https://github.com/apache/kafka/pull/13770#discussion_r1209855532 ## core/src/main/java/kafka/server/builders/LogManagerBuilder.java: ## @@ -111,7 +111,12 @@ public LogManagerBuilder setMaxTransactionTimeoutMs(int maxTransactionTimeout