Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-31 Thread via GitHub
lucasbru merged PR #15271: URL: https://github.com/apache/kafka/pull/15271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-31 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1472861675 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java: ## @@ -552,8 +555,63 @@ public void testDelayedReconciliationResul

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-31 Thread via GitHub
lucasbru commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1472843719 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java: ## @@ -552,8 +555,63 @@ public void testDelayedReconciliationResu

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-31 Thread via GitHub
dajac commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1472793128 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImplTest.java: ## @@ -552,8 +555,63 @@ public void testDelayedReconciliationResultA

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-31 Thread via GitHub
lucasbru commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1472523826 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1350,21 +1321,33 @@ public Map> currentAssignment() {

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-30 Thread via GitHub
dajac commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1471273124 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1350,21 +1321,33 @@ public Map> currentAssignment() { re

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-29 Thread via GitHub
dajac commented on PR #15271: URL: https://github.com/apache/kafka/pull/15271#issuecomment-1915528645 Sure. Will review it tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-29 Thread via GitHub
lucasbru commented on PR #15271: URL: https://github.com/apache/kafka/pull/15271#issuecomment-1915499373 @dajac Could you please take a look when you have time? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-29 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1470081608 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -997,11 +985,13 @@ void markReconciliationCompleted() { *

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-29 Thread via GitHub
lucasbru commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1469529850 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -208,22 +209,18 @@ public class MembershipManagerImpl implements

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on PR #15271: URL: https://github.com/apache/kafka/pull/15271#issuecomment-1912485870 Hey @lucasbru, thanks for the changes, not only fixing the bug but also moving in the direction we want of internally spreading the use of topicIds. Left a few comments. -- This is an

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on PR #15271: URL: https://github.com/apache/kafka/pull/15271#issuecomment-1912485446 Just to clarify, regarding the last note on the PR description, "if a new assignment or new metadata arrives during an ongoing reconciliation, it will never be applied". Maybe I'm missing

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467835947 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1340,18 +1336,15 @@ boolean reconciliationInProgress() { *

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467947528 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1311,21 +1294,34 @@ public Map> currentAssignment() {

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467947528 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1311,21 +1294,34 @@ public Map> currentAssignment() {

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467868971 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1010,17 +1000,24 @@ private void resolveMetadataForUnresolvedAss

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467868408 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -997,11 +985,13 @@ void markReconciliationCompleted() { *

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467835947 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -1340,18 +1336,15 @@ boolean reconciliationInProgress() { *

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467829868 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -766,23 +755,28 @@ public void transitionToStale() { }

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lianetm commented on code in PR #15271: URL: https://github.com/apache/kafka/pull/15271#discussion_r1467829521 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/MembershipManagerImpl.java: ## @@ -674,6 +662,7 @@ void transitionToSendingLeaveGroup() {

Re: [PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lucasbru commented on PR #15271: URL: https://github.com/apache/kafka/pull/15271#issuecomment-1912206859 @lianetm Could you have a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] KAFKA-16198: Reconciliation may lose partitions when topic metadata is delayed [kafka]

2024-01-26 Thread via GitHub
lucasbru opened a new pull request, #15271: URL: https://github.com/apache/kafka/pull/15271 The current reconciliation code in `AsyncKafkaConsumer`s `MembershipManager` may lose part of the server-provided assignment when metadata is delayed. The reason is incorrect handling of partially re