[GitHub] [kafka] mjsax commented on a diff in pull request #13925: KAFKA-10199: Consider tasks in state updater when computing offset sums
mjsax commented on code in PR #13925:
URL: https://github.com/apache/kafka/pull/13925#discussion_r1247273467
##
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##
@@ -1141,25 +1141,30 @@ public Map getTaskOffsetSums() {
// Not all tasks will create directories, and there may be directories
for tasks we don't currently own,
// so we consider all tasks that are either owned or on disk. This
includes stateless tasks, which should
// just have an empty changelogOffsets map.
-for (final TaskId id : union(HashSet::new, lockedTaskDirectories,
tasks.allTaskIds())) {
Review Comment:
It seems with the state updated enabled, `tasks` is actually only containing
"running tasks". It seems appropriate the rename this variable to
`runningTasks` (can also happen in a follow up PR).
I am actually also wondering if we still need this `Tasks` container any
longer to begin with? The purpose of the `Tasks` container was to simplify
`TaskManager` that manages both active and standby tasks. With the state
updated (from my understanding) the `TaskManager` only manages active tasks,
while standby tasks will be owned by the state-updated-thread (would it still
be useful for the state-updated-thread to use `Tasks` container, given that is
also own active tasks as long as they are restoring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jolshan closed pull request #13918: DO NOT MERGE -- testing
jolshan closed pull request #13918: DO NOT MERGE -- testing URL: https://github.com/apache/kafka/pull/13918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #13787: KAFKA-14884: Include check transaction is still ongoing right before append (take 2)
jolshan commented on code in PR #13787:
URL: https://github.com/apache/kafka/pull/13787#discussion_r1247245146
##
core/src/main/scala/kafka/network/RequestChannel.scala:
##
@@ -507,9 +507,14 @@ class RequestChannel(val queueSize: Int,
def sendShutdownRequest(): Unit = requestQueue.put(ShutdownRequest)
def sendCallbackRequest(request: CallbackRequest): Unit = {
-callbackQueue.put(request)
-if (!requestQueue.offer(RequestChannel.WakeupRequest))
- trace("Wakeup request could not be added to queue. This means queue is
full, so we will still process callback.")
+try {
+ callbackQueue.put(request)
+ if (!requestQueue.offer(RequestChannel.WakeupRequest))
+trace("Wakeup request could not be added to queue. This means queue is
full, so we will still process callback.")
+} catch {
+ case e: Throwable =>
+warn("There was an exception when attempting to send the callback
request", e)
Review Comment:
Hmmm -- I guess we would not send a response. Do we send a response normally
though if we shutdown during handling a request? Or do we just rely on it to
time out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] hachikuji commented on a diff in pull request #13787: KAFKA-14884: Include check transaction is still ongoing right before append (take 2)
hachikuji commented on code in PR #13787:
URL: https://github.com/apache/kafka/pull/13787#discussion_r1247229531
##
core/src/main/scala/kafka/network/RequestChannel.scala:
##
@@ -507,9 +507,14 @@ class RequestChannel(val queueSize: Int,
def sendShutdownRequest(): Unit = requestQueue.put(ShutdownRequest)
def sendCallbackRequest(request: CallbackRequest): Unit = {
-callbackQueue.put(request)
-if (!requestQueue.offer(RequestChannel.WakeupRequest))
- trace("Wakeup request could not be added to queue. This means queue is
full, so we will still process callback.")
+try {
+ callbackQueue.put(request)
+ if (!requestQueue.offer(RequestChannel.WakeupRequest))
+trace("Wakeup request could not be added to queue. This means queue is
full, so we will still process callback.")
+} catch {
+ case e: Throwable =>
+warn("There was an exception when attempting to send the callback
request", e)
Review Comment:
What happens to the request inside the `CallbackRequest` object? Do we need
to send a response?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] JimGalasyn commented on pull request #13937: Add Streams API broker compatibility table
JimGalasyn commented on PR #13937: URL: https://github.com/apache/kafka/pull/13937#issuecomment-1613837996 cc @guozhangwang @mjsax -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] JimGalasyn opened a new pull request, #13937: Add Streams API broker compatibility table
JimGalasyn opened a new pull request, #13937: URL: https://github.com/apache/kafka/pull/13937 Add a broker compatibility matrix to the Streams Developer Guide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] szalapski commented on pull request #5876: KAFKA-7509: Avoid passing most non-applicable properties to producer, consumer, and admin client
szalapski commented on PR #5876: URL: https://github.com/apache/kafka/pull/5876#issuecomment-1613828699 That sounds great. I'll try to upgrade and look into it soon. Feel free to close. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KAFKA-14246) Update threading model for Consumer (KIP-945)
[ https://issues.apache.org/jira/browse/KAFKA-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14246: -- Summary: Update threading model for Consumer (KIP-945) (was: Update threading model for Consumer) > Update threading model for Consumer (KIP-945) > - > > Key: KAFKA-14246 > URL: https://issues.apache.org/jira/browse/KAFKA-14246 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Hi community, > > We are refactoring the current KafkaConsumer and making it more asynchronous. > This is the master Jira to track the project's progress; subtasks will be > linked to this ticket. Please review the design document and feel free to > use this thread for discussion. > > The design document is here: > [https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor] > > The original email thread is here: > [https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l] > > I will continue to update the 1pager as reviews and comments come. > > Thanks, > P -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kafka] CalvinConfluent commented on a diff in pull request #13870: KAFKA-14500; [5/N] Implement JoinGroup protocol in new GroupCoordinator
CalvinConfluent commented on code in PR #13870:
URL: https://github.com/apache/kafka/pull/13870#discussion_r1246995181
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##
@@ -874,4 +1087,1362 @@ public void replay(
consumerGroup.updateMember(newMember);
}
}
+
+/**
+ * Replays GroupMetadataKey/Value to update the soft state of
+ * the generic group.
+ *
+ * @param key A GroupMetadataKey key.
+ * @param value A GroupMetadataValue record.
+ */
+public void replay(
+GroupMetadataKey key,
+GroupMetadataValue value,
+short version
+) {
+String groupId = key.group();
+
+if (value == null) {
+// Tombstone. Group should not be added.
+} else {
+List loadedMembers = new ArrayList<>();
+for (GroupMetadataValue.MemberMetadata member : value.members()) {
+int rebalanceTimeout = version == 0 ? member.sessionTimeout()
: member.rebalanceTimeout();
+
+JoinGroupRequestProtocolCollection supportedProtocols = new
JoinGroupRequestProtocolCollection();
+supportedProtocols.add(new JoinGroupRequestProtocol()
+.setName(value.protocol())
+.setMetadata(member.subscription()));
+
+GenericGroupMember loadedMember = new GenericGroupMember(
+member.memberId(),
+Optional.ofNullable(member.groupInstanceId()),
+member.clientId(),
+member.clientHost(),
+rebalanceTimeout,
+member.sessionTimeout(),
+value.protocolType(),
+supportedProtocols,
+member.assignment()
+);
+
+loadedMembers.add(loadedMember);
+}
+
+String protocolType = value.protocolType();
+
+GenericGroup genericGroup = new GenericGroup(
+this.logContext,
+groupId,
+loadedMembers.isEmpty() ? EMPTY : STABLE,
+time,
+value.generation(),
+protocolType == null || protocolType.isEmpty() ?
Optional.empty() : Optional.of(protocolType),
+Optional.ofNullable(value.protocol()),
+Optional.ofNullable(value.leader()),
+value.currentStateTimestamp() == -1 ? Optional.empty() :
Optional.of(value.currentStateTimestamp())
+);
+
+loadedMembers.forEach(member -> {
+genericGroup.add(member, null);
+log.info("Loaded member {} in group {} with generation {}.",
+member.memberId(), groupId, genericGroup.generationId());
+});
+
+genericGroup.setSubscribedTopics(
+genericGroup.computeSubscribedTopics()
+);
+}
+}
+
+/**
+ * Handle a JoinGroupRequest.
+ *
+ * @param context The request context.
+ * @param request The actual JoinGroup request.
+ *
+ * @return The result that contains records to append if the join group
phase completes.
+ */
+public CoordinatorResult genericGroupJoin(
+RequestContext context,
+JoinGroupRequestData request,
+CompletableFuture responseFuture
+) {
+CoordinatorResult result = EMPTY_RESULT;
+
+String groupId = request.groupId();
+String memberId = request.memberId();
+int sessionTimeoutMs = request.sessionTimeoutMs();
+
+if (sessionTimeoutMs < genericGroupMinSessionTimeoutMs ||
+sessionTimeoutMs > genericGroupMaxSessionTimeoutMs
+) {
+responseFuture.complete(new JoinGroupResponseData()
+.setMemberId(memberId)
+.setErrorCode(Errors.INVALID_SESSION_TIMEOUT.code())
+);
+} else {
+boolean isUnknownMember = memberId.equals(UNKNOWN_MEMBER_ID);
+// Group is created if it does not exist and the member id is
UNKNOWN. if member
+// is specified but group does not exist, request is rejected with
GROUP_ID_NOT_FOUND
+GenericGroup group;
+try {
+group = getOrMaybeCreateGenericGroup(groupId, isUnknownMember);
+} catch (Throwable t) {
+responseFuture.complete(new JoinGroupResponseData()
+.setMemberId(memberId)
+.setErrorCode(Errors.forException(t).code())
+);
+return EMPTY_RESULT;
+}
+
+if (group.isNew()) {
+// If a group was newly created, we need to append records to
the log
+// to commit the group to the timeline datastructure.
+
+CompletableFuture appendFuture = new
CompletableFuture<>();
+
[jira] [Updated] (KAFKA-14264) Refactor coordinator code
[ https://issues.apache.org/jira/browse/KAFKA-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14264: -- Component/s: clients > Refactor coordinator code > - > > Key: KAFKA-14264 > URL: https://issues.apache.org/jira/browse/KAFKA-14264 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > Fix For: 3.4.0 > > > To refactor the consumer, we changed how the coordinator is called. However, > there will be a time period where the old and new implementation need to > coexist, so we will need to override some of the methods and create a new > implementation of the coordinator. In particular: > # ensureCoordinatorReady needs to be non-blocking or we could just use the > sendFindCoordinatorRequest. > # joinGroupIfNeeded needs to be broken up into more find grain stages for > the new implementation to work. > We also need to create the coordinator state machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14875) Implement Wakeup()
[ https://issues.apache.org/jira/browse/KAFKA-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14875: -- Component/s: clients > Implement Wakeup() > -- > > Key: KAFKA-14875 > URL: https://issues.apache.org/jira/browse/KAFKA-14875 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement wakeup() and WakeupException. This would be different to the > current implementation because I think we just need to interrupt the blocking > futures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14761) Integration Tests for the New Consumer Implementation
[ https://issues.apache.org/jira/browse/KAFKA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14761: -- Component/s: clients > Integration Tests for the New Consumer Implementation > - > > Key: KAFKA-14761 > URL: https://issues.apache.org/jira/browse/KAFKA-14761 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > This Jira tracks the efforts of integratoin testing for the new consumer we > are implementing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14252) Create background thread skeleton for new Consumer threading model
[ https://issues.apache.org/jira/browse/KAFKA-14252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14252: -- Component/s: clients > Create background thread skeleton for new Consumer threading model > -- > > Key: KAFKA-14252 > URL: https://issues.apache.org/jira/browse/KAFKA-14252 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The event handler internally instantiates a background thread to consume > ApplicationEvents and produce BackgroundEvents. In this ticket, we will > create a skeleton of the background thread. We will incrementally add > implementation in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14468) Refactor Commit Logic
[ https://issues.apache.org/jira/browse/KAFKA-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14468: -- Component/s: clients > Refactor Commit Logic > - > > Key: KAFKA-14468 > URL: https://issues.apache.org/jira/browse/KAFKA-14468 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > Fix For: 3.5.0 > > > Refactor commit logic using the new multi-threaded coordinator construct. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14247) Implement EventHandler interface and DefaultEventHandler for Consumer
[ https://issues.apache.org/jira/browse/KAFKA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14247: -- Component/s: clients > Implement EventHandler interface and DefaultEventHandler for Consumer > - > > Key: KAFKA-14247 > URL: https://issues.apache.org/jira/browse/KAFKA-14247 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The background thread runs inside of the DefaultEventHandler to consume > events from the ApplicationEventQueue and produce events to the > BackgroundEventQueue. > The background thread runnable consist of a loop that tries to poll events > from the ApplicationQueue, processes the event if there are any, and poll > networkClient. > In this implementation, the DefaultEventHandler spawns a thread that runs the > BackgroundThreadRunnable. The runnable, as of the current PR, does the > following things: > # Initialize the networkClient > # Poll ApplicationEvent from the queue if there's any > # process the event > # poll the networkClient > PR: https://github.com/apache/kafka/pull/12672 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14960) Metadata Request Manager and listTopics/partitionsFor API
[ https://issues.apache.org/jira/browse/KAFKA-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14960: -- Component/s: clients > Metadata Request Manager and listTopics/partitionsFor API > - > > Key: KAFKA-14960 > URL: https://issues.apache.org/jira/browse/KAFKA-14960 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement listTopics and partitionsFor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14950) Implement assign() and assignment()
[ https://issues.apache.org/jira/browse/KAFKA-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14950: -- Component/s: clients > Implement assign() and assignment() > --- > > Key: KAFKA-14950 > URL: https://issues.apache.org/jira/browse/KAFKA-14950 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement assign() and assignment() -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14246) Update threading model for Consumer
[ https://issues.apache.org/jira/browse/KAFKA-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14246: -- Component/s: clients > Update threading model for Consumer > --- > > Key: KAFKA-14246 > URL: https://issues.apache.org/jira/browse/KAFKA-14246 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Hi community, > > We are refactoring the current KafkaConsumer and making it more asynchronous. > This is the master Jira to track the project's progress; subtasks will be > linked to this ticket. Please review the design document and feel free to > use this thread for discussion. > > The design document is here: > [https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor] > > The original email thread is here: > [https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l] > > I will continue to update the 1pager as reviews and comments come. > > Thanks, > P -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kafka] CalvinConfluent commented on a diff in pull request #13870: KAFKA-14500; [5/N] Implement JoinGroup protocol in new GroupCoordinator
CalvinConfluent commented on code in PR #13870:
URL: https://github.com/apache/kafka/pull/13870#discussion_r1246964799
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##
@@ -102,13 +108,34 @@ public Builder withLoader(CoordinatorLoader
loader) {
return this;
}
+public Builder withMetadataImage(MetadataImage metadataImage) {
+this.metadataImage = metadataImage;
+return this;
+}
+
+public Builder withTime(Time time) {
+this.time = time;
+return this;
+}
+
+public Builder withTimer(Timer timer) {
+this.timer = timer;
+return this;
+}
+
public GroupCoordinatorService build() {
if (config == null)
throw new IllegalArgumentException("Config must be set.");
if (writer == null)
throw new IllegalArgumentException("Writer must be set.");
if (loader == null)
throw new IllegalArgumentException("Loader must be set.");
+if (metadataImage == null)
+throw new IllegalArgumentException("MetadataImage must be
set.");
+if (time == null)
+throw new IllegalArgumentException("Time must be set.");
+if (timer == null)
Review Comment:
The timer is not used yet. Is it a place holder here?
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##
@@ -874,4 +1087,1362 @@ public void replay(
consumerGroup.updateMember(newMember);
}
}
+
+/**
+ * Replays GroupMetadataKey/Value to update the soft state of
+ * the generic group.
+ *
+ * @param key A GroupMetadataKey key.
+ * @param value A GroupMetadataValue record.
+ */
+public void replay(
+GroupMetadataKey key,
+GroupMetadataValue value,
+short version
+) {
+String groupId = key.group();
+
+if (value == null) {
+// Tombstone. Group should not be added.
+} else {
+List loadedMembers = new ArrayList<>();
+for (GroupMetadataValue.MemberMetadata member : value.members()) {
+int rebalanceTimeout = version == 0 ? member.sessionTimeout()
: member.rebalanceTimeout();
+
+JoinGroupRequestProtocolCollection supportedProtocols = new
JoinGroupRequestProtocolCollection();
+supportedProtocols.add(new JoinGroupRequestProtocol()
+.setName(value.protocol())
+.setMetadata(member.subscription()));
+
+GenericGroupMember loadedMember = new GenericGroupMember(
+member.memberId(),
+Optional.ofNullable(member.groupInstanceId()),
+member.clientId(),
+member.clientHost(),
+rebalanceTimeout,
+member.sessionTimeout(),
+value.protocolType(),
+supportedProtocols,
+member.assignment()
+);
+
+loadedMembers.add(loadedMember);
+}
+
+String protocolType = value.protocolType();
+
+GenericGroup genericGroup = new GenericGroup(
+this.logContext,
+groupId,
+loadedMembers.isEmpty() ? EMPTY : STABLE,
Review Comment:
I guess I missed some of the previous discussions, but why it is always
either stable or empty when we load a group? Does it mean the rebalancing
process will be reverted if the coordinator fails?
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##
@@ -874,4 +1087,1362 @@ public void replay(
consumerGroup.updateMember(newMember);
}
}
+
+/**
+ * Replays GroupMetadataKey/Value to update the soft state of
+ * the generic group.
+ *
+ * @param key A GroupMetadataKey key.
+ * @param value A GroupMetadataValue record.
+ */
+public void replay(
+GroupMetadataKey key,
+GroupMetadataValue value,
+short version
+) {
+String groupId = key.group();
+
+if (value == null) {
+// Tombstone. Group should not be added.
+} else {
+List loadedMembers = new ArrayList<>();
+for (GroupMetadataValue.MemberMetadata member : value.members()) {
+int rebalanceTimeout = version == 0 ? member.sessionTimeout()
: member.rebalanceTimeout();
+
+JoinGroupRequestProtocolCollection supportedProtocols = new
JoinGroupRequestProtocolCollection();
+supportedProtocols.add(new JoinGroupRequestProtocol()
+.setName(value.protocol())
+
[jira] [Updated] (KAFKA-14761) Integration Tests for the New Consumer Implementation
[ https://issues.apache.org/jira/browse/KAFKA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14761: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Integration Tests for the New Consumer Implementation > - > > Key: KAFKA-14761 > URL: https://issues.apache.org/jira/browse/KAFKA-14761 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > This Jira tracks the efforts of integratoin testing for the new consumer we > are implementing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14247) Implement EventHandler interface and DefaultEventHandler
[ https://issues.apache.org/jira/browse/KAFKA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14247: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Implement EventHandler interface and DefaultEventHandler > > > Key: KAFKA-14247 > URL: https://issues.apache.org/jira/browse/KAFKA-14247 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The background thread runs inside of the DefaultEventHandler to consume > events from the ApplicationEventQueue and produce events to the > BackgroundEventQueue. > The background thread runnable consist of a loop that tries to poll events > from the ApplicationQueue, processes the event if there are any, and poll > networkClient. > In this implementation, the DefaultEventHandler spawns a thread that runs the > BackgroundThreadRunnable. The runnable, as of the current PR, does the > following things: > # Initialize the networkClient > # Poll ApplicationEvent from the queue if there's any > # process the event > # poll the networkClient > PR: https://github.com/apache/kafka/pull/12672 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14247) Implement EventHandler interface and DefaultEventHandler for Consumer
[ https://issues.apache.org/jira/browse/KAFKA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14247: -- Summary: Implement EventHandler interface and DefaultEventHandler for Consumer (was: Implement EventHandler interface and DefaultEventHandler) > Implement EventHandler interface and DefaultEventHandler for Consumer > - > > Key: KAFKA-14247 > URL: https://issues.apache.org/jira/browse/KAFKA-14247 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The background thread runs inside of the DefaultEventHandler to consume > events from the ApplicationEventQueue and produce events to the > BackgroundEventQueue. > The background thread runnable consist of a loop that tries to poll events > from the ApplicationQueue, processes the event if there are any, and poll > networkClient. > In this implementation, the DefaultEventHandler spawns a thread that runs the > BackgroundThreadRunnable. The runnable, as of the current PR, does the > following things: > # Initialize the networkClient > # Poll ApplicationEvent from the queue if there's any > # process the event > # poll the networkClient > PR: https://github.com/apache/kafka/pull/12672 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14252) Create background thread skeleton
[ https://issues.apache.org/jira/browse/KAFKA-14252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14252: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Create background thread skeleton > - > > Key: KAFKA-14252 > URL: https://issues.apache.org/jira/browse/KAFKA-14252 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The event handler internally instantiates a background thread to consume > ApplicationEvents and produce BackgroundEvents. In this ticket, we will > create a skeleton of the background thread. We will incrementally add > implementation in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14252) Create background thread skeleton for new Consumer threading model
[ https://issues.apache.org/jira/browse/KAFKA-14252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14252: -- Summary: Create background thread skeleton for new Consumer threading model (was: Create background thread skeleton) > Create background thread skeleton for new Consumer threading model > -- > > Key: KAFKA-14252 > URL: https://issues.apache.org/jira/browse/KAFKA-14252 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The event handler internally instantiates a background thread to consume > ApplicationEvents and produce BackgroundEvents. In this ticket, we will > create a skeleton of the background thread. We will incrementally add > implementation in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14960) Metadata Request Manager and listTopics/partitionsFor API
[ https://issues.apache.org/jira/browse/KAFKA-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14960: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Metadata Request Manager and listTopics/partitionsFor API > - > > Key: KAFKA-14960 > URL: https://issues.apache.org/jira/browse/KAFKA-14960 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement listTopics and partitionsFor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14264) Refactor coordinator code
[ https://issues.apache.org/jira/browse/KAFKA-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14264: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Refactor coordinator code > - > > Key: KAFKA-14264 > URL: https://issues.apache.org/jira/browse/KAFKA-14264 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > Fix For: 3.4.0 > > > To refactor the consumer, we changed how the coordinator is called. However, > there will be a time period where the old and new implementation need to > coexist, so we will need to override some of the methods and create a new > implementation of the coordinator. In particular: > # ensureCoordinatorReady needs to be non-blocking or we could just use the > sendFindCoordinatorRequest. > # joinGroupIfNeeded needs to be broken up into more find grain stages for > the new implementation to work. > We also need to create the coordinator state machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14468) Refactor Commit Logic
[ https://issues.apache.org/jira/browse/KAFKA-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14468: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Refactor Commit Logic > - > > Key: KAFKA-14468 > URL: https://issues.apache.org/jira/browse/KAFKA-14468 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > Fix For: 3.5.0 > > > Refactor commit logic using the new multi-threaded coordinator construct. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14966) Extract reusable common logic from OffsetFetcher
[ https://issues.apache.org/jira/browse/KAFKA-14966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14966: -- Parent: (was: KAFKA-14965) Issue Type: Task (was: Sub-task) > Extract reusable common logic from OffsetFetcher > > > Key: KAFKA-14966 > URL: https://issues.apache.org/jira/browse/KAFKA-14966 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Major > Labels: kip-945 > Fix For: 3.6.0 > > > The OffsetFetcher is internally used by the KafkaConsumer to fetch offsets, > validate and reset positions. > For the new consumer based on a refactored threading model, similar > functionality will be needed by the ListOffsetsRequestManager component. > This task aims at identifying and extracting the OffsetFetcher functionality > that can be reused by the new consumer implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14950) Implement assign() and assignment()
[ https://issues.apache.org/jira/browse/KAFKA-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14950: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Implement assign() and assignment() > --- > > Key: KAFKA-14950 > URL: https://issues.apache.org/jira/browse/KAFKA-14950 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement assign() and assignment() -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14875) Implement Wakeup()
[ https://issues.apache.org/jira/browse/KAFKA-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14875: -- Parent: (was: KAFKA-14246) Issue Type: Task (was: Sub-task) > Implement Wakeup() > -- > > Key: KAFKA-14875 > URL: https://issues.apache.org/jira/browse/KAFKA-14875 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement wakeup() and WakeupException. This would be different to the > current implementation because I think we just need to interrupt the blocking > futures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14274) Introduce FetchRequestManager to integrate fetch into new consumer threading refactor
[
https://issues.apache.org/jira/browse/KAFKA-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14274:
--
Issue Type: Task (was: Improvement)
> Introduce FetchRequestManager to integrate fetch into new consumer threading
> refactor
> -
>
> Key: KAFKA-14274
> URL: https://issues.apache.org/jira/browse/KAFKA-14274
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task is to introduce a new class named {{FetchRequestManager}} that will
> be responsible for:
> # Formatting fetch requests to the background thread
> # Configuring the callback on fetch responses for the background thread
> The response handler will collect the fetch responses from the broker and
> create {{{}CompletedFetch{}}}, instances, much as is done in {{{}Fetcher{}}}.
> The newly introduced {{FetchUtils}} will be used for both
> {{FetchRequestManager}} and {{Fetcher}} to keep the logic as reusable as
> possible.
> The foreground logic will decompress the data into a {{{}Record{}}}, which
> will then be deserialized into a {{ConsumerRecord}} for returning to the user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14675) Extract metadata-related tasks from Fetcher into MetadataFetcher
[
https://issues.apache.org/jira/browse/KAFKA-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14675:
--
Issue Type: Task (was: Improvement)
> Extract metadata-related tasks from Fetcher into MetadataFetcher
>
>
> Key: KAFKA-14675
> URL: https://issues.apache.org/jira/browse/KAFKA-14675
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
>Affects Versions: 3.5.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task covers the work to extract from {{Fetcher}} the APIs that are
> related to metadata operations into a new class named
> {{{}MetadataFetcher{}}}. This will allow the refactoring of {{Fetcher}} and
> {{MetadataFetcher}} for the new consumer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14365) Extract common logic from Fetcher
[
https://issues.apache.org/jira/browse/KAFKA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14365:
--
Issue Type: Task (was: Improvement)
> Extract common logic from Fetcher
> -
>
> Key: KAFKA-14365
> URL: https://issues.apache.org/jira/browse/KAFKA-14365
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
> Fix For: 3.5.0
>
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task includes refactoring {{Fetcher}} by extracting out some common
> logic to allow forthcoming implementations to leverage it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14758) Extract inner classes from Fetcher for reuse in refactoring
[
https://issues.apache.org/jira/browse/KAFKA-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14758:
--
Issue Type: Task (was: Improvement)
> Extract inner classes from Fetcher for reuse in refactoring
> ---
>
> Key: KAFKA-14758
> URL: https://issues.apache.org/jira/browse/KAFKA-14758
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
> Fix For: 3.5.0
>
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task includes refactoring {{Fetcher}} by extracting out the inner
> classes into top-level (though still in {{{}internal{}}}) so that those
> classes can be referenced by forthcoming refactored fetch logic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14724) Port tests in FetcherTest to FetchRequestManagerTest
[
https://issues.apache.org/jira/browse/KAFKA-14724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14724:
--
Issue Type: Task (was: Improvement)
> Port tests in FetcherTest to FetchRequestManagerTest
>
>
> Key: KAFKA-14724
> URL: https://issues.apache.org/jira/browse/KAFKA-14724
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task involves copying the relevant tests from {{FetcherTest}} and
> modifying them to fit a new unit test named {{{}FetchRequestManagerTest{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14724) Port tests in FetcherTest to FetchRequestManagerTest
[
https://issues.apache.org/jira/browse/KAFKA-14724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14724:
--
Component/s: clients
consumer
> Port tests in FetcherTest to FetchRequestManagerTest
>
>
> Key: KAFKA-14724
> URL: https://issues.apache.org/jira/browse/KAFKA-14724
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task involves copying the relevant tests from {{FetcherTest}} and
> modifying them to fit a new unit test named {{{}FetchRequestManagerTest{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14848) KafkaConsumer incorrectly passes locally-scoped deserializers to FetchConfig
[
https://issues.apache.org/jira/browse/KAFKA-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14848:
--
Labels: kip-945 (was: )
> KafkaConsumer incorrectly passes locally-scoped deserializers to FetchConfig
>
>
> Key: KAFKA-14848
> URL: https://issues.apache.org/jira/browse/KAFKA-14848
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 3.5.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> [~rayokota] found some {{{}NullPointerException{}}}s that originate because
> of a recently introduced error in the {{KafkaConsumer}} constructor. The code
> was changed to pass the deserializer variables into the {{FetchConfig}}
> constructor. However, this code change incorrectly used the locally-scoped
> variables, not the instance-scoped variables. Since the locally-scoped
> variables could be {{{}null{}}}, this results in the {{FetchConfig}} storing
> {{null}} references, leading to downstream breakage.
> Suggested change:
> {noformat}
> - FetchConfig fetchConfig = new FetchConfig<>(config, keyDeserializer,
> valueDeserializer, isolationLevel);
> + FetchConfig fetchConfig = new FetchConfig<>(config,
> this.keyDeserializer, this.valueDeserializer, isolationLevel);
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13445) Add ECDSA test for JWT validation
[ https://issues.apache.org/jira/browse/KAFKA-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-13445: -- Labels: OAuth (was: ) > Add ECDSA test for JWT validation > - > > Key: KAFKA-13445 > URL: https://issues.apache.org/jira/browse/KAFKA-13445 > Project: Kafka > Issue Type: Improvement > Components: unit tests >Affects Versions: 3.1.0 >Reporter: Kirk True >Assignee: Kirk True >Priority: Minor > Labels: OAuth > Fix For: 3.1.0, 3.2.0 > > > The tests for OAuth JWT validation all assume usage of RSA, but we need to > have ECDSA support there too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13444) Fix OAuthCompatibilityTool help and add SSL options
[
https://issues.apache.org/jira/browse/KAFKA-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-13444:
--
Labels: OAuth (was: )
> Fix OAuthCompatibilityTool help and add SSL options
> ---
>
> Key: KAFKA-13444
> URL: https://issues.apache.org/jira/browse/KAFKA-13444
> Project: Kafka
> Issue Type: Improvement
> Components: tools
>Affects Versions: 3.1.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Minor
> Labels: OAuth
> Fix For: 3.1.0
>
>
> The {{OAuthCompatibilityTool}} is missing the SSL configuration options. In
> addition, the help text is incorrect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13443) Kafka broker exits when OAuth enabled and certain configuration not specified
[
https://issues.apache.org/jira/browse/KAFKA-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-13443:
--
Labels: OAuth beginner (was: beginner)
> Kafka broker exits when OAuth enabled and certain configuration not specified
> -
>
> Key: KAFKA-13443
> URL: https://issues.apache.org/jira/browse/KAFKA-13443
> Project: Kafka
> Issue Type: Bug
> Components: config, security
>Affects Versions: 3.1.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Blocker
> Labels: OAuth, beginner
> Fix For: 3.1.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The {{sasl.oauthbearer.jwks.endpoint.retry.backoff.ms}} and
> {{sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms}} configuration options
> were added to the {{SaslConfig}} class but their default values were not
> added to {{{}KafkaConfig{}}}. As a result, when the OAuth validation feature
> is enabled in the broker and those two configuration values aren't explicitly
> provided by the user, the broker exits.
> The fix is to define them in the {{KafkaConfig}} class.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14623) OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging
[ https://issues.apache.org/jira/browse/KAFKA-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14623: -- Labels: OAuth (was: ) > OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging > --- > > Key: KAFKA-14623 > URL: https://issues.apache.org/jira/browse/KAFKA-14623 > Project: Kafka > Issue Type: Bug > Components: clients, security >Affects Versions: 3.3.0, 3.3.1, 3.3.2 >Reporter: Kirk True >Assignee: Kirk True >Priority: Major > Labels: OAuth > Fix For: 3.4.0, 3.3.3 > > Original Estimate: 24h > Remaining Estimate: 24h > > The OAuth code that communicates via HTTP with the IdP > (HttpAccessTokenRetriever.java) includes logging that outputs the request and > response payloads. Among them are: > * > [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L265] > * > [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L274] > * > [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/secured/HttpAccessTokenRetriever.java#L320] > It should be determined if there are other places sensitive information might > be inadvertently exposed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13725) KIP-768 OAuth code mixes public and internal classes in same package
[
https://issues.apache.org/jira/browse/KAFKA-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-13725:
--
Labels: OAuth (was: )
> KIP-768 OAuth code mixes public and internal classes in same package
>
>
> Key: KAFKA-13725
> URL: https://issues.apache.org/jira/browse/KAFKA-13725
> Project: Kafka
> Issue Type: Bug
> Components: clients, security
>Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.3.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: OAuth
> Fix For: 3.4.0
>
>
> The {{org.apache.kafka.common.security.oauthbearer.secured}} package from
> KIP-768 incorrectly mixed all of the classes (public and internal) in the
> package together.
> This bug is to remove all but the public classes from that package and move
> the rest to a new
> {{{}org.apache.kafka.common.security.oauthbearer.internal.{}}}{{{}secured{}}}
> package. This should be back-ported to all versions in which the KIP-768
> OAuth work occurs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13202) KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC
[
https://issues.apache.org/jira/browse/KAFKA-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-13202:
--
Labels: OAuth (was: )
> KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC
> --
>
> Key: KAFKA-13202
> URL: https://issues.apache.org/jira/browse/KAFKA-13202
> Project: Kafka
> Issue Type: New Feature
> Components: clients, security
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: OAuth
> Fix For: 3.1.0
>
>
> This task is to provide a concrete implementation of the interfaces defined
> in
> [KIP-255|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75968876]
> to allow Kafka to connect to an [OAuth|https://en.wikipedia.org/wiki/OAuth]
> / [OIDC|https://en.wikipedia.org/wiki/OpenID#OpenID_Connect_(OIDC)] identity
> provider for authentication and token retrieval. While KIP-255 provides an
> unsecured JWT example for development, this will fill in the gap and provide
> a production-grade implementation.
> The OAuth/OIDC work will allow out-of-the-box configuration by any Apache
> Kafka users to connect to an external identity provider service (e.g. Okta,
> Auth0, Azure, etc.). The code will implement the standard OAuth
> {{clientcredentials}} grant type.
> The proposed change is largely composed of a pair of
> {{AuthenticateCallbackHandler}} implementations: one to login on the client
> and one to validate on the broker.
> See [KIP-768: Extend SASL/OAUTHBEARER with Support for
> OIDC|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575]
> for more detail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13446) Remove JWT access token from logs
[ https://issues.apache.org/jira/browse/KAFKA-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-13446: -- Labels: OAuth (was: ) > Remove JWT access token from logs > - > > Key: KAFKA-13446 > URL: https://issues.apache.org/jira/browse/KAFKA-13446 > Project: Kafka > Issue Type: Bug > Components: logging, security >Affects Versions: 3.1.0 >Reporter: Kirk True >Assignee: Kirk True >Priority: Major > Labels: OAuth > Fix For: 3.1.0 > > > The OAuth code logs the access token on both the client and the server, > potentially exposing service account details. Remove all logging entries to > prevent this from leaking. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14724) Port tests in FetcherTest to FetchRequestManagerTest
[
https://issues.apache.org/jira/browse/KAFKA-14724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14724:
--
Labels: kip-945 (was: )
> Port tests in FetcherTest to FetchRequestManagerTest
>
>
> Key: KAFKA-14724
> URL: https://issues.apache.org/jira/browse/KAFKA-14724
> Project: Kafka
> Issue Type: Improvement
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task involves copying the relevant tests from {{FetcherTest}} and
> modifying them to fit a new unit test named {{{}FetchRequestManagerTest{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14365) Extract common logic from Fetcher
[
https://issues.apache.org/jira/browse/KAFKA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14365:
--
Labels: kip-945 (was: )
> Extract common logic from Fetcher
> -
>
> Key: KAFKA-14365
> URL: https://issues.apache.org/jira/browse/KAFKA-14365
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
> Fix For: 3.5.0
>
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task includes refactoring {{Fetcher}} by extracting out some common
> logic to allow forthcoming implementations to leverage it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14675) Extract metadata-related tasks from Fetcher into MetadataFetcher
[
https://issues.apache.org/jira/browse/KAFKA-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14675:
--
Labels: kip-945 (was: )
> Extract metadata-related tasks from Fetcher into MetadataFetcher
>
>
> Key: KAFKA-14675
> URL: https://issues.apache.org/jira/browse/KAFKA-14675
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
>Affects Versions: 3.5.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task covers the work to extract from {{Fetcher}} the APIs that are
> related to metadata operations into a new class named
> {{{}MetadataFetcher{}}}. This will allow the refactoring of {{Fetcher}} and
> {{MetadataFetcher}} for the new consumer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14758) Extract inner classes from Fetcher for reuse in refactoring
[
https://issues.apache.org/jira/browse/KAFKA-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14758:
--
Labels: kip-945 (was: )
> Extract inner classes from Fetcher for reuse in refactoring
> ---
>
> Key: KAFKA-14758
> URL: https://issues.apache.org/jira/browse/KAFKA-14758
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
> Fix For: 3.5.0
>
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task includes refactoring {{Fetcher}} by extracting out the inner
> classes into top-level (though still in {{{}internal{}}}) so that those
> classes can be referenced by forthcoming refactored fetch logic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14937) Refactoring for client code to reduce boilerplate
[ https://issues.apache.org/jira/browse/KAFKA-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14937: -- Parent: (was: KAFKA-14274) Issue Type: Improvement (was: Sub-task) > Refactoring for client code to reduce boilerplate > - > > Key: KAFKA-14937 > URL: https://issues.apache.org/jira/browse/KAFKA-14937 > Project: Kafka > Issue Type: Improvement > Components: admin, clients, consumer, producer >Reporter: Kirk True >Assignee: Kirk True >Priority: Major > Labels: kip-945 > Fix For: 3.6.0 > > > There are a number of places in the client code where the same basic calls > are made by more than one client implementation. Minor refactoring will > reduce the amount of boilerplate code necessary for the client to construct > its internal state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14966) Extract reusable common logic from OffsetFetcher
[ https://issues.apache.org/jira/browse/KAFKA-14966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14966: -- Labels: kip-945 (was: ) > Extract reusable common logic from OffsetFetcher > > > Key: KAFKA-14966 > URL: https://issues.apache.org/jira/browse/KAFKA-14966 > Project: Kafka > Issue Type: Sub-task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Major > Labels: kip-945 > Fix For: 3.6.0 > > > The OffsetFetcher is internally used by the KafkaConsumer to fetch offsets, > validate and reset positions. > For the new consumer based on a refactored threading model, similar > functionality will be needed by the ListOffsetsRequestManager component. > This task aims at identifying and extracting the OffsetFetcher functionality > that can be reused by the new consumer implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14937) Refactoring for client code to reduce boilerplate
[ https://issues.apache.org/jira/browse/KAFKA-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14937: -- Labels: kip-945 (was: ) > Refactoring for client code to reduce boilerplate > - > > Key: KAFKA-14937 > URL: https://issues.apache.org/jira/browse/KAFKA-14937 > Project: Kafka > Issue Type: Sub-task > Components: admin, clients, consumer, producer >Reporter: Kirk True >Assignee: Kirk True >Priority: Major > Labels: kip-945 > Fix For: 3.6.0 > > > There are a number of places in the client code where the same basic calls > are made by more than one client implementation. Minor refactoring will > reduce the amount of boilerplate code necessary for the client to construct > its internal state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14965) Introduce ListOffsetsRequestManager to integrate ListOffsetsRequests into new consumer threading refactor
[
https://issues.apache.org/jira/browse/KAFKA-14965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14965:
--
Labels: kip-945 (was: )
> Introduce ListOffsetsRequestManager to integrate ListOffsetsRequests into new
> consumer threading refactor
> -
>
> Key: KAFKA-14965
> URL: https://issues.apache.org/jira/browse/KAFKA-14965
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
>Reporter: Lianet Magrans
>Assignee: Lianet Magrans
>Priority: Major
> Labels: kip-945
>
> This task introduces new functionality for handling ListOffsetsRequests for
> the new consumer implementation, as part for the ongoing work for the
> consumer threading model refactor.
> This task introduces a new class named {{ListOffsetsRequestManager,
> }}responsible of handling ListOffsets requests performed by the consumer to
> expose functionality like beginningOffsets, endOffsets and offsetsForTimes.
> The Offset{{{}Fetcher{}}} class is used internally by the {{KafkaConsumer}}
> to list offsets, so this task will be based on a refactored
> Offset{{{}Fetcher{}}}, reusing the fetching logic as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15081) Implement new consumer offsetsForTimes
[ https://issues.apache.org/jira/browse/KAFKA-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-15081: -- Labels: kip-945 (was: ) > Implement new consumer offsetsForTimes > -- > > Key: KAFKA-15081 > URL: https://issues.apache.org/jira/browse/KAFKA-15081 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Major > Labels: kip-945 > > Implement offsetForTimes for the kafka consumer based on the new threading > model, using the ListOffsetsRequestManager. No changes at the consumer API > level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15115) Implement resetPositions functionality in ListOffsetRequestManager
[ https://issues.apache.org/jira/browse/KAFKA-15115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-15115: -- Labels: kip-945 (was: ) > Implement resetPositions functionality in ListOffsetRequestManager > -- > > Key: KAFKA-15115 > URL: https://issues.apache.org/jira/browse/KAFKA-15115 > Project: Kafka > Issue Type: Task > Components: clients, consumer >Reporter: Lianet Magrans >Assignee: Lianet Magrans >Priority: Major > Labels: kip-945 > > Introduce support for resetting positions in the new > ListOffsetsRequestManager. This task will include a new event for the > resetPositions calls performed from the new consumer, and the logic for > handling such events in the ListOffsetRequestManager. > The reset positions implementation will keep the same behaviour as the one in > the old consumer, but adapted to the new threading model. So it is based in a > RESET_POSITIONS events that is submitted to the background thread, and the > processed by the ApplicationEventProcessor. The processing itself is done by > the ListOffsetRequestManager given that this will require a LIST_OFFSETS > request for the partitions awaiting reset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14252) Create background thread skeleton
[ https://issues.apache.org/jira/browse/KAFKA-14252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14252: -- Labels: kip-945 (was: ) > Create background thread skeleton > - > > Key: KAFKA-14252 > URL: https://issues.apache.org/jira/browse/KAFKA-14252 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The event handler internally instantiates a background thread to consume > ApplicationEvents and produce BackgroundEvents. In this ticket, we will > create a skeleton of the background thread. We will incrementally add > implementation in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14247) Implement EventHandler interface and DefaultEventHandler
[ https://issues.apache.org/jira/browse/KAFKA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14247: -- Labels: kip-945 (was: ) > Implement EventHandler interface and DefaultEventHandler > > > Key: KAFKA-14247 > URL: https://issues.apache.org/jira/browse/KAFKA-14247 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > The background thread runs inside of the DefaultEventHandler to consume > events from the ApplicationEventQueue and produce events to the > BackgroundEventQueue. > The background thread runnable consist of a loop that tries to poll events > from the ApplicationQueue, processes the event if there are any, and poll > networkClient. > In this implementation, the DefaultEventHandler spawns a thread that runs the > BackgroundThreadRunnable. The runnable, as of the current PR, does the > following things: > # Initialize the networkClient > # Poll ApplicationEvent from the queue if there's any > # process the event > # poll the networkClient > PR: https://github.com/apache/kafka/pull/12672 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14264) Refactor coordinator code
[ https://issues.apache.org/jira/browse/KAFKA-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14264: -- Labels: kip-945 (was: ) > Refactor coordinator code > - > > Key: KAFKA-14264 > URL: https://issues.apache.org/jira/browse/KAFKA-14264 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > Fix For: 3.4.0 > > > To refactor the consumer, we changed how the coordinator is called. However, > there will be a time period where the old and new implementation need to > coexist, so we will need to override some of the methods and create a new > implementation of the coordinator. In particular: > # ensureCoordinatorReady needs to be non-blocking or we could just use the > sendFindCoordinatorRequest. > # joinGroupIfNeeded needs to be broken up into more find grain stages for > the new implementation to work. > We also need to create the coordinator state machine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15116) Kafka Streams processing blocked during rebalance
[ https://issues.apache.org/jira/browse/KAFKA-15116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738782#comment-17738782 ] Matthias J. Sax commented on KAFKA-15116: - > Message A uses an internal store to store information about the entity. The > store knows that there is a pending event that is yet to be committed so it > blocks until it is committed. Are you saying that this happens in a background thread that you start yourself? If yes, it a non-supported pattern, and we cannot give any guarantee about the behavior of the system. If there is no background thread, that blocking would imply that `StreamThread` blocks (also something you should not do, as it would imply the that thread drops out of the consumer group after `max.poll.interval.ms` passed, and thus, how could message B get processed? Or is this internal store that you mentioned shared across `StreamThreads` (this would also be an anti-pattern, and we cannot give any guarantee how the system behalves if you do this). > The store knows that there is a pending event that is yet to be committed so > it blocks until it is committed. I am also wondering what you exactly mean by "committed" (it's a highly overloaded term, so it would be good to clarify). In Kafka itself, there could be two meanings: for at-least-once-processing "committing" means to commit the input topic offsets and mark the input records as processed. For exaclty-once-processing "committing" means to commit the Kafka TX, ie, committing the result record into the output topic plus committing the input topic offset to mark the input records as processed. Not sure which one you mean, or if you actually refer to some mechanism to commit into your internal store? I guess, I still don't understand your overall end-to-end workflow of your program. > Kafka Streams processing blocked during rebalance > - > > Key: KAFKA-15116 > URL: https://issues.apache.org/jira/browse/KAFKA-15116 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.5.0 >Reporter: David Gammon >Priority: Major > > We have a Kafka Streams application that simply takes a messages, processes > it and then produces an event out the other side. The complexity is that > there is a requirement that all events with the same partition key must be > committed before the next message is processed. > This works most of the time flawlessly but we have started to see problems > during deployments where the first message blocks the second message during a > rebalance because the first message isn’t committed before the second message > is processed. This ultimately results in transactions timing out and more > rebalancing. > We’ve tried lots of configuration to get the behaviour we require with no > luck. We’ve now put in a temporary fix so that Kafka Streams works with our > framework but it feels like this might be a missing feature or potentially a > bug. > +Example+ > Given: > * We have two messages (InA and InB). > * Both messages have the same partition key. > * A rebalance is in progress so streams is no longer able to commit. > When: > # Message InA -> processor -> OutA (not committed) > # Message InB -> processor -> blocked because #1 has not been committed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14274) Introduce FetchRequestManager to integrate fetch into new consumer threading refactor
[
https://issues.apache.org/jira/browse/KAFKA-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-14274:
--
Labels: kip-945 (was: )
> Introduce FetchRequestManager to integrate fetch into new consumer threading
> refactor
> -
>
> Key: KAFKA-14274
> URL: https://issues.apache.org/jira/browse/KAFKA-14274
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Major
> Labels: kip-945
>
> The {{Fetcher}} class is used internally by the {{KafkaConsumer}} to fetch
> records from the brokers. There is ongoing work to create a new consumer
> implementation with a significantly refactored threading model. The threading
> refactor work requires a similarly refactored {{{}Fetcher{}}}.
> This task is to introduce a new class named {{FetchRequestManager}} that will
> be responsible for:
> # Formatting fetch requests to the background thread
> # Configuring the callback on fetch responses for the background thread
> The response handler will collect the fetch responses from the broker and
> create {{{}CompletedFetch{}}}, instances, much as is done in {{{}Fetcher{}}}.
> The newly introduced {{FetchUtils}} will be used for both
> {{FetchRequestManager}} and {{Fetcher}} to keep the logic as reusable as
> possible.
> The foreground logic will decompress the data into a {{{}Record{}}}, which
> will then be deserialized into a {{ConsumerRecord}} for returning to the user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14960) Metadata Request Manager and listTopics/partitionsFor API
[ https://issues.apache.org/jira/browse/KAFKA-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14960: -- Labels: kip-945 (was: ) > Metadata Request Manager and listTopics/partitionsFor API > - > > Key: KAFKA-14960 > URL: https://issues.apache.org/jira/browse/KAFKA-14960 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement listTopics and partitionsFor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14950) Implement assign() and assignment()
[ https://issues.apache.org/jira/browse/KAFKA-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14950: -- Labels: kip-945 (was: ) > Implement assign() and assignment() > --- > > Key: KAFKA-14950 > URL: https://issues.apache.org/jira/browse/KAFKA-14950 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement assign() and assignment() -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14875) Implement Wakeup()
[ https://issues.apache.org/jira/browse/KAFKA-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14875: -- Labels: kip-945 (was: ) > Implement Wakeup() > -- > > Key: KAFKA-14875 > URL: https://issues.apache.org/jira/browse/KAFKA-14875 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Implement wakeup() and WakeupException. This would be different to the > current implementation because I think we just need to interrupt the blocking > futures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14468) Refactor Commit Logic
[ https://issues.apache.org/jira/browse/KAFKA-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14468: -- Labels: kip-945 (was: ) > Refactor Commit Logic > - > > Key: KAFKA-14468 > URL: https://issues.apache.org/jira/browse/KAFKA-14468 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > Fix For: 3.5.0 > > > Refactor commit logic using the new multi-threaded coordinator construct. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14246) Update threading model for Consumer
[ https://issues.apache.org/jira/browse/KAFKA-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14246: -- Labels: kip-945 (was: ) > Update threading model for Consumer > --- > > Key: KAFKA-14246 > URL: https://issues.apache.org/jira/browse/KAFKA-14246 > Project: Kafka > Issue Type: Improvement > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > Hi community, > > We are refactoring the current KafkaConsumer and making it more asynchronous. > This is the master Jira to track the project's progress; subtasks will be > linked to this ticket. Please review the design document and feel free to > use this thread for discussion. > > The design document is here: > [https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor] > > The original email thread is here: > [https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l] > > I will continue to update the 1pager as reviews and comments come. > > Thanks, > P -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14761) Integration Tests for the New Consumer Implementation
[ https://issues.apache.org/jira/browse/KAFKA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14761: -- Parent: KAFKA-14246 Issue Type: Sub-task (was: Task) > Integration Tests for the New Consumer Implementation > - > > Key: KAFKA-14761 > URL: https://issues.apache.org/jira/browse/KAFKA-14761 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > This Jira tracks the efforts of integratoin testing for the new consumer we > are implementing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14875) Implement Wakeup()
[ https://issues.apache.org/jira/browse/KAFKA-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14875: -- Parent: KAFKA-14246 Issue Type: Sub-task (was: Task) > Implement Wakeup() > -- > > Key: KAFKA-14875 > URL: https://issues.apache.org/jira/browse/KAFKA-14875 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > > Implement wakeup() and WakeupException. This would be different to the > current implementation because I think we just need to interrupt the blocking > futures. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14761) Integration Tests for the New Consumer Implementation
[ https://issues.apache.org/jira/browse/KAFKA-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14761: -- Labels: kip-945 (was: ) > Integration Tests for the New Consumer Implementation > - > > Key: KAFKA-14761 > URL: https://issues.apache.org/jira/browse/KAFKA-14761 > Project: Kafka > Issue Type: Task > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > Labels: kip-945 > > This Jira tracks the efforts of integratoin testing for the new consumer we > are implementing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kafka] jolshan commented on a diff in pull request #13901: KAFKA-14462; [20/N] Refresh subscription metadata on new metadata image
jolshan commented on code in PR #13901:
URL: https://github.com/apache/kafka/pull/13901#discussion_r1247163837
##
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java:
##
@@ -172,19 +186,21 @@ public List build(TopicsImage topicsImage) {
});
// Add subscription metadata.
-Map subscriptionMetadata = new HashMap<>();
-members.forEach((memberId, member) -> {
-member.subscribedTopicNames().forEach(topicName -> {
-TopicImage topicImage = topicsImage.getTopic(topicName);
-if (topicImage != null) {
-subscriptionMetadata.put(topicName, new TopicMetadata(
-topicImage.id(),
-topicImage.name(),
-topicImage.partitions().size()
-));
-}
+if (subscriptionMetadata == null) {
Review Comment:
If subscriptionMetadata is not null, that means we just want to take what we
are already given and not generate it based on the subscribedNames and the
image?
Is this done to optimize or are there cases where we want to have something
different than what we would have generated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (KAFKA-14246) Update threading model for Consumer
[ https://issues.apache.org/jira/browse/KAFKA-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-14246: -- Summary: Update threading model for Consumer (was: Refactor KafkaConsumer Threading Model) > Update threading model for Consumer > --- > > Key: KAFKA-14246 > URL: https://issues.apache.org/jira/browse/KAFKA-14246 > Project: Kafka > Issue Type: Improvement > Components: consumer >Reporter: Philip Nee >Assignee: Philip Nee >Priority: Major > > Hi community, > > We are refactoring the current KafkaConsumer and making it more asynchronous. > This is the master Jira to track the project's progress; subtasks will be > linked to this ticket. Please review the design document and feel free to > use this thread for discussion. > > The design document is here: > [https://cwiki.apache.org/confluence/display/KAFKA/Proposal%3A+Consumer+Threading+Model+Refactor] > > The original email thread is here: > [https://lists.apache.org/thread/13jvwzkzmb8c6t7drs4oj2kgkjzcn52l] > > I will continue to update the 1pager as reviews and comments come. > > Thanks, > P -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kafka] jolshan commented on a diff in pull request #13901: KAFKA-14462; [20/N] Refresh subscription metadata on new metadata image
jolshan commented on code in PR #13901:
URL: https://github.com/apache/kafka/pull/13901#discussion_r1247156849
##
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java:
##
@@ -728,6 +794,81 @@ public void replay(
}
consumerGroup.removeMember(memberId);
}
+
+updateGroupsByTopics(groupId, oldSubscribedTopicNames,
consumerGroup.subscribedTopicNames());
+}
+
+/**
+ * @return The set of groups subscribed to the topic.
+ */
+public Set groupsSubscribedToTopic(String topicName) {
+Set groups = groupsByTopics.get(topicName);
+return groups != null ? groups : Collections.emptySet();
+}
+
+/**
+ * Subscribes a group to a topic.
+ *
+ * @param groupId The group id.
+ * @param topicName The topic name.
+ */
+private void subscribeGroupToTopic(
+String groupId,
+String topicName
+) {
+groupsByTopics
+.computeIfAbsent(topicName, __ -> new
TimelineHashSet<>(snapshotRegistry, 1))
+.add(groupId);
+}
+
+/**
+ * Unsubscribes a group from a topic.
+ *
+ * @param groupId The group id.
+ * @param topicName The topic name.
+ */
+private void unsubscribeGroupFromTopic(
+String groupId,
+String topicName
+) {
+groupsByTopics.computeIfPresent(topicName, (__, groupIds) -> {
+groupIds.remove(groupId);
+return groupIds.isEmpty() ? null : groupIds;
+});
+}
+
+/**
+ * Updates the group by topics mapping.
+ *
+ * @param groupId The group id.
+ * @param oldSubscribedTopics The old group subscriptions.
+ * @param newSubscribedTopics The new group subscriptions.
+ */
+private void updateGroupsByTopics(
Review Comment:
do we have any test on this and the above code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #13901: KAFKA-14462; [20/N] Refresh subscription metadata on new metadata image
jolshan commented on code in PR #13901:
URL: https://github.com/apache/kafka/pull/13901#discussion_r1247155039
##
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/runtime/CoordinatorRuntimeTest.java:
##
@@ -825,4 +828,60 @@ public void testClose() throws Exception {
assertFutureThrows(write1, NotCoordinatorException.class);
assertFutureThrows(write2, NotCoordinatorException.class);
}
+
+@Test
+public void testOnNewMetadataImage() {
+TopicPartition tp0 = new TopicPartition("__consumer_offsets", 0);
+TopicPartition tp1 = new TopicPartition("__consumer_offsets", 1);
+
+MockCoordinatorLoader loader = mock(MockCoordinatorLoader.class);
+MockPartitionWriter writer = mock(MockPartitionWriter.class);
+MockCoordinatorBuilderSupplier supplier =
mock(MockCoordinatorBuilderSupplier.class);
+MockCoordinatorBuilder builder = mock(MockCoordinatorBuilder.class);
+
+CoordinatorRuntime runtime =
+new CoordinatorRuntime.Builder()
+.withLoader(loader)
+.withEventProcessor(new MockEventProcessor())
+.withPartitionWriter(writer)
+.withCoordinatorBuilderSupplier(supplier)
+.build();
+
+MockCoordinator coordinator0 = mock(MockCoordinator.class);
+MockCoordinator coordinator1 = mock(MockCoordinator.class);
+
+when(supplier.get()).thenReturn(builder);
+when(builder.withSnapshotRegistry(any())).thenReturn(builder);
+when(builder.withLogContext(any())).thenReturn(builder);
+when(builder.build())
+.thenReturn(coordinator0)
+.thenReturn(coordinator1);
+
+CompletableFuture future0 = new CompletableFuture<>();
+when(loader.load(tp0, coordinator0)).thenReturn(future0);
+
+CompletableFuture future1 = new CompletableFuture<>();
+when(loader.load(tp1, coordinator1)).thenReturn(future1);
+
+runtime.scheduleLoadOperation(tp0, 0);
+runtime.scheduleLoadOperation(tp1, 0);
+
+// Coordinator 0 is loaded. It should get the current image
+// that is the empty one.
+future0.complete(null);
+verify(coordinator0).onLoaded(MetadataImage.EMPTY);
+
+// Publish a new image.
+MetadataDelta delta = new MetadataDelta(MetadataImage.EMPTY);
+MetadataImage newImage = delta.apply(MetadataProvenance.EMPTY);
+runtime.onNewMetadataImage(newImage, delta);
+
+// Coordinator 0 should be notified about it.
+verify(coordinator0).onNewMetadataImage(newImage, delta);
Review Comment:
Do we also want to test the logic inside the onNewMetadataImage code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] jolshan commented on a diff in pull request #13901: KAFKA-14462; [20/N] Refresh subscription metadata on new metadata image
jolshan commented on code in PR #13901:
URL: https://github.com/apache/kafka/pull/13901#discussion_r1247154202
##
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/runtime/CoordinatorRuntimeTest.java:
##
@@ -825,4 +828,60 @@ public void testClose() throws Exception {
assertFutureThrows(write1, NotCoordinatorException.class);
assertFutureThrows(write2, NotCoordinatorException.class);
}
+
+@Test
+public void testOnNewMetadataImage() {
+TopicPartition tp0 = new TopicPartition("__consumer_offsets", 0);
+TopicPartition tp1 = new TopicPartition("__consumer_offsets", 1);
+
+MockCoordinatorLoader loader = mock(MockCoordinatorLoader.class);
+MockPartitionWriter writer = mock(MockPartitionWriter.class);
+MockCoordinatorBuilderSupplier supplier =
mock(MockCoordinatorBuilderSupplier.class);
+MockCoordinatorBuilder builder = mock(MockCoordinatorBuilder.class);
+
+CoordinatorRuntime runtime =
+new CoordinatorRuntime.Builder()
+.withLoader(loader)
+.withEventProcessor(new MockEventProcessor())
+.withPartitionWriter(writer)
+.withCoordinatorBuilderSupplier(supplier)
+.build();
+
+MockCoordinator coordinator0 = mock(MockCoordinator.class);
+MockCoordinator coordinator1 = mock(MockCoordinator.class);
+
+when(supplier.get()).thenReturn(builder);
+when(builder.withSnapshotRegistry(any())).thenReturn(builder);
+when(builder.withLogContext(any())).thenReturn(builder);
+when(builder.build())
+.thenReturn(coordinator0)
+.thenReturn(coordinator1);
+
+CompletableFuture future0 = new CompletableFuture<>();
+when(loader.load(tp0, coordinator0)).thenReturn(future0);
+
+CompletableFuture future1 = new CompletableFuture<>();
+when(loader.load(tp1, coordinator1)).thenReturn(future1);
+
+runtime.scheduleLoadOperation(tp0, 0);
+runtime.scheduleLoadOperation(tp1, 0);
+
+// Coordinator 0 is loaded. It should get the current image
+// that is the empty one.
+future0.complete(null);
+verify(coordinator0).onLoaded(MetadataImage.EMPTY);
+
+// Publish a new image.
+MetadataDelta delta = new MetadataDelta(MetadataImage.EMPTY);
+MetadataImage newImage = delta.apply(MetadataProvenance.EMPTY);
+runtime.onNewMetadataImage(newImage, delta);
+
+// Coordinator 0 should be notified about it.
+verify(coordinator0).onNewMetadataImage(newImage, delta);
Review Comment:
do we care to also show that we update the metadata image on the next
heartbeat? maybe we don't necessarily need to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] kirktrue commented on pull request #13591: KAFKA-14831: Illegal state errors should be fatal in transactional producer
kirktrue commented on PR #13591: URL: https://github.com/apache/kafka/pull/13591#issuecomment-1613748954 Thanks @jolshan, @urbandan, and @hachikuji for the reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:30 PM:
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option.{}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
users to provide a custom implementation that the override
`ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
was (Author: omnia_h_ibrahim):
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option.{}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
users to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] C0urante merged pull request #13934: MINOR: Replace synchronization with atomic update in Connect's StateTracker::changeState method
C0urante merged PR #13934: URL: https://github.com/apache/kafka/pull/13934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:01 PM:
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option.{}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
users to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
was (Author: omnia_h_ibrahim):
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option.{}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM:
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
was (Author: omnia_h_ibrahim):
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}{{{}We can also update the backward compatibility section in
KIP-690 to ask customers to override the
`ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM:
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option.{}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
was (Author: omnia_h_ibrahim):
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM:
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
was (Author: omnia_h_ibrahim):
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim edited comment on KAFKA-15102 at 6/29/23 7:00 PM:
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
was (Author: omnia_h_ibrahim):
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}
{{{}We can also update the backward compatibility section in KIP-690 to ask
customers to override the `ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738744#comment-17738744
]
Omnia Ibrahim commented on KAFKA-15102:
---
Thanks, [~ddufour1a] for raising this. The backward compatibility mentioned in
the KIP accounted only for using the default separator configuration and didn't
address the usage custom separator (my mistake here). [~ChrisEgerton] I think
having `{{{}replication.policy.internal.topic.separator.enabled` is a good
option. {}}}{{{}We can also update the backward compatibility section in
KIP-690 to ask customers to override the
`ReplicationPolicy.{}}}offsetSyncsTopic`and
`ReplicationPolicy.checkpointsTopic` methods to use old topics if they still
want to use the old internal topics or delete them so they are aware of this
issue.
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] jolshan commented on a diff in pull request #13901: KAFKA-14462; [20/N] Refresh subscription metadata on new metadata image
jolshan commented on code in PR #13901:
URL: https://github.com/apache/kafka/pull/13901#discussion_r1246997020
##
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java:
##
@@ -537,8 +538,35 @@ public void testUpdateSubscriptionMetadata() {
consumerGroup.computeSubscriptionMetadata(
null,
null,
-image
+image.topics()
)
);
}
+
+@Test
+public void testMetadataRefreshDeadline() {
Review Comment:
should this also test `requestMetadataRefresh`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KAFKA-15113) Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters
[
https://issues.apache.org/jira/browse/KAFKA-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738740#comment-17738740
]
Chris Egerton commented on KAFKA-15113:
---
[~yash.mayya] although it seems a little unlikely, I agree that it's worth
considering the use case of a separate Kafka cluster for topics consumed by a
sink connector and used for a DLQ.
I'm tempted to dismiss it as unrealistic but we'd definitely need a KIP before
deciding not to support use cases like this. If we wanted to support them, we
could potentially introduce different config namespaces for the DLQ admin
client and the offset management admin client, which can be used to override
the existing general admin overrides for a sink connector. This could also be
left as "future work" for any KIP that makes that kind of use case impossible.
RE compatibility and a restructured request format for connector
configurations: I don't know if it would be too difficult; we could either
differentiate by request header (connect-config-version or something like
that), or even try to deduce the format automatically based on the shape of the
JSON request. It's the kind of thing that may get a little hairy under the
hood, but could be made pretty smooth for users.
> Gracefully handle cases where a sink connector's admin and consumer client
> config overrides target different Kafka clusters
> ---
>
> Key: KAFKA-15113
> URL: https://issues.apache.org/jira/browse/KAFKA-15113
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Yash Mayya
>Priority: Minor
>
> Background reading -
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]
>
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
>
>
> From [https://github.com/apache/kafka/pull/13434#discussion_r1144415671] -
> {quote}Currently, admin clients are only instantiated for sink connectors to
> create the DLQ topic if required. So it seems like it could be technically
> possible for a sink connector's consumer client overrides to target a
> different Kafka cluster from its producer and admin client overrides. Such a
> setup won't work with this implementation of the get offsets API as it is
> using an admin client to get a sink connector's consumer group offsets.
> However, I'm not sure we want to use a consumer client to retrieve the
> offsets either as we shouldn't be disrupting the existing sink tasks'
> consumer group just to fetch offsets. Leveraging a sink task's consumer also
> isn't an option because fetching offsets for a stopped sink connector (where
> all the tasks will be stopped) should be allowed. I'm wondering if we should
> document that a connector's various client config override policies shouldn't
> target different Kafka clusters (side note - looks like we don't [currently
> document|https://kafka.apache.org/documentation/#connect] client config
> overrides for Connect beyond just the worker property
> {{{}connector.client.config.override.policy{}}}).
> {quote}
>
> {quote}I don't think we need to worry too much about this. I cannot imagine a
> sane use case that involves overriding a connector's Kafka clients with
> different Kafka clusters (not just bootstrap servers, but actually different
> clusters) for producer/consumer/admin. I'd be fine with adding a note to our
> docs that that kind of setup isn't supported but I really, really hope that
> it's not necessary and nobody's trying to do that in the first place. I also
> suspect that there are other places where this might cause issues, like with
> exactly-once source support or automatic topic creation for source connectors.
> That said, there is a different case we may want to consider: someone may
> have configured consumer overrides for a sink connector, but not admin
> overrides. This may happen if they don't use a DLQ topic. I don't know if we
> absolutely need to handle this now and we may consider filing a follow-up
> ticket to look into this, but one quick-and-dirty thought I've had is to
> configure the admin client used here with a combination of the configurations
> for the connector's admin client and its consumer, giving precedent to the
> latter.
> {quote}
>
> Also from [https://github.com/apache/kafka/pull/13818#discussion_r1224138055]
> -
> {quote}We will have undesirable behavior if the connector is targeting a
> Kafka cluster different from the Connect cluster's backing Kafka cluster and
> the user has configured the consumer overrides appropriately for their
> connector, but not the admin overrides (something we also discussed
> previously
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738733#comment-17738733
]
Chris Egerton edited comment on KAFKA-15102 at 6/29/23 6:43 PM:
[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve
backward compatibility perfectly, we would have had to ignore the custom
separator when creating topics affected by KIP-690, and possibly introduced
separate opt-in configuration logic to disable that behavior (i.e., resume
taking custom separators into account, even for topics that were not affected
prior to KIP-690).
However, since 3.1.0 came out a year and a half ago, things are less
cut-and-dry: there may be more negative fallout for users who are accustomed to
the current behavior than what's currently being experienced by those who are
used to the previous behavior.
One possibility could be to revert to older behavior for the remainder of our
3.x.y releases with a single configuration property such as
{{replication.policy.internal.topic.separator.enabled}} that defaults to
{{false}} for now and, come 4.0, defaults to {{{}true{}}}.
[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to
approach this accidental break in compatibility? It's probably worth bringing
this up in a discussion thread on the user/dev mailing lists, but I'd rather
get some feedback on a potential fix before bringing this before a larger
audience.
was (Author: chrisegerton):
[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve
backward compatibility perfectly, we would have had to ignore the custom
separator when creating topics affected by KIP-690, and possibly introduced
separate opt-in configuration logic to disable that behavior (i.e., resume
taking custom separators into account, even for topics that were not affected
prior to KIP-690).
However, since 3.1.0 came out a year and a half ago, things are less
cut-and-dry: there may be more negative fallout for users who are accustomed to
the current behavior than what's currently being experienced by those who are
used to the previous behavior.
One possibility could be to revert to older behavior for the remainder of our
3.x.y releases with a single configuration property such as
{{replication.policy.internal.topic.separator.enabled}} that defaults to
{{false}} for now and, come 4.0, defaults to {{{}true{}}}.
[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to
approach this accidental break in compatibility?
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738733#comment-17738733
]
Chris Egerton commented on KAFKA-15102:
---
[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve
backward compatibility perfectly, we would have had to ignore the custom
separator when creating topics affected by KIP-690, and possibly introduced
separate opt-in configuration logic to disable that behavior (i.e., resume
taking custom separators into account, even for topics that were not affected
prior to KIP-690).
However, since 3.1.0 came out a year and a half ago, things are less
cut-and-dry: there may be more negative fallout for users who are accustomed to
the current behavior than what's currently being experienced by those who are
used to the previous behavior.
One possibility could be to revert to older behavior for the remainder of our
3.x.y releases with a single configuration property such as
{{replication.policy.internal.topic.separator.enabled}} that defaults to
{{false}} for now and, come 4.0, defaults to {{{}true{}}}.
[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to
approach this accidental break in compatibility?
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] jolshan commented on pull request #13876: KAFKA-10733: Clean up producer exceptions
jolshan commented on PR #13876: URL: https://github.com/apache/kafka/pull/13876#issuecomment-1613602815 @lucasbru -- I'm also thinking about consistency from the sense of compatibility with previous state. Ideally our final state will balance consistency in how we handle errors (so new users can easily understand) while keeping things mostly consistent to how it was handled before (so previous users don't see their workflows broken) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] jolshan merged pull request #13591: KAFKA-14831: Illegal state errors should be fatal in transactional producer
jolshan merged PR #13591: URL: https://github.com/apache/kafka/pull/13591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] jolshan commented on pull request #13591: KAFKA-14831: Illegal state errors should be fatal in transactional producer
jolshan commented on PR #13591: URL: https://github.com/apache/kafka/pull/13591#issuecomment-1613597535 Ok -- test failures seem to be consistent with current trunk and not due to this change. Will go ahead and merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] C0urante merged pull request #13933: MINOR: Update anchor link for exactly-once source connectors
C0urante merged PR #13933: URL: https://github.com/apache/kafka/pull/13933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] cmccabe merged pull request #13904: KAFKA-15114: Minor: Update help in StorageTool for creating SCRAM credentials for KRaft bootstrap.
cmccabe merged PR #13904: URL: https://github.com/apache/kafka/pull/13904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KAFKA-15127) Allow offsets to be reset at the same time a connector is deleted.
[ https://issues.apache.org/jira/browse/KAFKA-15127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738704#comment-17738704 ] Sagar Rao commented on KAFKA-15127: --- I don't intend working on this immediately or in the near future. It's just created for future ref. > Allow offsets to be reset at the same time a connector is deleted. > -- > > Key: KAFKA-15127 > URL: https://issues.apache.org/jira/browse/KAFKA-15127 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sagar Rao >Assignee: Sagar Rao >Priority: Major > Labels: needs-kip > > This has been listed as [Future > Work|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Automaticallydeleteoffsetswithconnectors] > in KIP-875. Now that the delete offsets mechanism is also in place, we can > take this up which will allow connector names to be reused after connector > deletion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Divij Vaidya resolved KAFKA-15053.
--
Reviewer: Divij Vaidya
Resolution: Fixed
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Assignee: Bo Gao
>Priority: Major
> Labels: backport
> Fix For: 3.3.3, 3.6.0, 3.5.1, 3.4.2
>
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Divij Vaidya updated KAFKA-15053:
-
Labels: backport (was: )
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Assignee: Bo Gao
>Priority: Major
> Labels: backport
> Fix For: 3.3.3, 3.6.0, 3.5.1, 3.4.2
>
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Divij Vaidya updated KAFKA-15053:
-
Fix Version/s: 3.3.3
3.5.1
3.4.2
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Assignee: Bo Gao
>Priority: Major
> Fix For: 3.3.3, 3.6.0, 3.5.1, 3.4.2
>
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kafka] C0urante commented on pull request #13530: KAFKA-14858: Handle exceptions thrown from Connector::taskConfigs in Connect's standalone mode
C0urante commented on PR #13530: URL: https://github.com/apache/kafka/pull/13530#issuecomment-1613554645 Hmmm... is there an advantage to enabling users to pause/resume a failed connector? I feel like restarting it is still the natural corrective action in that case, which will also cause task config generation to be retried. Honestly, I think the biggest obstacle to preventing pause/resume for connectors that have failed to generate task configs is that it'll look uglier in the code base, especially if we take a naive approach and introduce some kind of public `WorkerConnector::setState` method (gross). Perhaps we could add a `WorkerConnector::taskConfigs` method that accepts a `boolean reportFailure` parameter (feel free to rename), which controls whether any exceptions encountered while invoking `Connector::taskConfigs` (or anywhere else in the method, really) should be reported to the connector's status listener and result in a state change? It's still a little unclean, but it does reflect the needs of our internal API: in one mode, we want these errors to be thrown to the caller without affecting any other state for the connector, and in another mode, we still want these errors to be thrown (so as to surface in REST responses as 500 errors), but also to result in a change in state for the connector. Also, one more advantage of keeping the state tracking (and reporting) inside the `WorkerConnector` class is that we currently use a [delegating status listener](https://github.com/apache/kafka/blob/30b087ead967b28d459945fe90c80545bf189d1f/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerConnector.java#L98-L99) to record metrics whenever the connector's status is updated. I believe with the current implementation in this PR, there would be a bug in metrics reporting for connectors (LMK if I'm mistaken, though). Finally, regarding augmenting error messages with a note that tasks may still be running--as long as we don't have to wrap any connector-thrown exceptions in order to introduce this additional wording, I'm in favor of it. I just don't want to add another set of stack frames and another link in the "caused by" chain without good reason since it makes things harder to read (I've seen people copy+paste stack traces that end with ["Tolerance exceeded in error handler"](https://github.com/apache/kafka/blob/30b087ead967b28d459945fe90c80545bf189d1f/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L591) and omit the actual root cause, which is... less than helpful). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] divijvaidya opened a new pull request, #13936: MINOR: Refactor & cleanup for RemoteIndexCache
divijvaidya opened a new pull request, #13936: URL: https://github.com/apache/kafka/pull/13936 **Changes** 1. Add new unit tests 2. Change the on-disk filename from `__.` to `_.` i.e. remove trailing suffix 3. Fix a small bug where we were parsing offset as Int when reading the file name from disk. Offset is long. 4. Perform input validation in RemoteLogSegmentMetadata. 5. Remove an extra loop in cleaner thread. Shutdownable thread already performs looping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13913: KAFKA-15119:Support incremental syncTopicAcls in MirrorSourceConnector
hudeqi commented on PR #13913: URL: https://github.com/apache/kafka/pull/13913#issuecomment-1613529431 Hi! please help to review this PR when you are free, thank you! @C0urante -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13924: KAFKA-15129;[1/N] Remove metrics in LogCleanerManager when LogCleaner shutdown
hudeqi commented on PR #13924: URL: https://github.com/apache/kafka/pull/13924#issuecomment-1613525851 I have submitted some PRs (two others: https://github.com/apache/kafka/pull/13926、https://github.com/apache/kafka/pull/13929) on the topic of "clean metric", please help to review, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] hudeqi commented on pull request #13930: KAFKA-15134:Enrich the prompt reason in CommitFailedException
hudeqi commented on PR #13930: URL: https://github.com/apache/kafka/pull/13930#issuecomment-1613522211 Hi! Could you help to review this PR? @hachikuji @guozhangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kafka] C0urante commented on pull request #13302: KAFKA-14759: Move Mock, Schema, and Verifiable connectors to new test-plugins module
C0urante commented on PR #13302: URL: https://github.com/apache/kafka/pull/13302#issuecomment-1613519953 Verified locally by running the `ConnectDistributedTest::test_file_source_and_sink`, `ConnectDistributedTest::test_bounce`, and `ConnectStandaloneFileTest::test_file_source_and_sink` test cases, which cover the addition of the file connectors to both standalone and distributed mode, and the addition of testing-only connectors to distributed mode. There don't appear to be any standalone mode tests that require the testing-only connectors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
