lianetm commented on code in PR #20521:
URL: https://github.com/apache/kafka/pull/20521#discussion_r2408267900
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThread.java:
##########
@@ -369,17 +373,37 @@ void cleanup() {
* If there is a metadata error, complete all uncompleted events that
require subscription metadata.
*/
private void maybeFailOnMetadataError(List<CompletableEvent<?>> events) {
- List<CompletableApplicationEvent<?>> subscriptionMetadataEvent = new
ArrayList<>();
+ List<MetadataErrorNotifiableEvent> notifiables = new ArrayList<>();
for (CompletableEvent<?> ce : events) {
- if (ce instanceof CompletableApplicationEvent &&
((CompletableApplicationEvent<?>) ce).requireSubscriptionMetadata())
- subscriptionMetadataEvent.add((CompletableApplicationEvent<?>)
ce);
+ if (ce instanceof MetadataErrorNotifiableEvent) {
+ notifiables.add((MetadataErrorNotifiableEvent) ce);
+ }
}
- if (subscriptionMetadataEvent.isEmpty())
+ if (notifiables.isEmpty())
return;
-
networkClientDelegate.getAndClearMetadataError().ifPresent(metadataError ->
- subscriptionMetadataEvent.forEach(event ->
event.future().completeExceptionally(metadataError))
- );
+
+ Optional<Exception> metadataErrorOpt =
networkClientDelegate.getAndClearMetadataError();
+
+ if (metadataErrorOpt.isEmpty())
+ return;
+
+ Exception metadataError = metadataErrorOpt.get();
+ notifiables.forEach(n ->
n.completeExceptionallyWithMetadataError(metadataError));
+ }
+
+ /**
+ * If there is a metadata error, complete this event exceptionally.
+ */
+ private boolean maybeFailOnMetadataError(MetadataErrorNotifiableEvent
notifiable) {
+ Optional<Exception> metadataErrorOpt =
networkClientDelegate.getAndClearMetadataError();
+
+ if (metadataErrorOpt.isEmpty())
+ return false;
+
+ Exception metadataError = metadataErrorOpt.get();
+ notifiable.completeExceptionallyWithMetadataError(metadataError);
+ return true;
Review Comment:
this seems a duplicate of what we have above on the other
`maybeFailOnMetadataError` .
Could we maybe keep the one above and change the calls that have a single
`MetadataErrorNotifiableEvent` to be something like
`maybeFailOnMetadataError(List.of(event))`?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThread.java:
##########
@@ -369,17 +373,37 @@ void cleanup() {
* If there is a metadata error, complete all uncompleted events that
require subscription metadata.
*/
private void maybeFailOnMetadataError(List<CompletableEvent<?>> events) {
- List<CompletableApplicationEvent<?>> subscriptionMetadataEvent = new
ArrayList<>();
+ List<MetadataErrorNotifiableEvent> notifiables = new ArrayList<>();
for (CompletableEvent<?> ce : events) {
- if (ce instanceof CompletableApplicationEvent &&
((CompletableApplicationEvent<?>) ce).requireSubscriptionMetadata())
- subscriptionMetadataEvent.add((CompletableApplicationEvent<?>)
ce);
+ if (ce instanceof MetadataErrorNotifiableEvent) {
+ notifiables.add((MetadataErrorNotifiableEvent) ce);
+ }
}
- if (subscriptionMetadataEvent.isEmpty())
+ if (notifiables.isEmpty())
return;
-
networkClientDelegate.getAndClearMetadataError().ifPresent(metadataError ->
- subscriptionMetadataEvent.forEach(event ->
event.future().completeExceptionally(metadataError))
- );
+
+ Optional<Exception> metadataErrorOpt =
networkClientDelegate.getAndClearMetadataError();
+
+ if (metadataErrorOpt.isEmpty())
+ return;
+
+ Exception metadataError = metadataErrorOpt.get();
Review Comment:
this section seems to be doing exactly what we had before in one line with
the `ifPresent`, just more verbose? (empty checks, new vars). Any reason why we
don't want to keep it how it was?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThread.java:
##########
@@ -369,17 +373,37 @@ void cleanup() {
* If there is a metadata error, complete all uncompleted events that
require subscription metadata.
*/
private void maybeFailOnMetadataError(List<CompletableEvent<?>> events) {
- List<CompletableApplicationEvent<?>> subscriptionMetadataEvent = new
ArrayList<>();
+ List<MetadataErrorNotifiableEvent> notifiables = new ArrayList<>();
for (CompletableEvent<?> ce : events) {
- if (ce instanceof CompletableApplicationEvent &&
((CompletableApplicationEvent<?>) ce).requireSubscriptionMetadata())
- subscriptionMetadataEvent.add((CompletableApplicationEvent<?>)
ce);
+ if (ce instanceof MetadataErrorNotifiableEvent) {
Review Comment:
nice simplification
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/MetadataErrorNotifiableEvent.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals.events;
+
+import org.apache.kafka.clients.Metadata;
+import org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread;
+import org.apache.kafka.clients.consumer.internals.NetworkClientDelegate;
+
+/**
+ * This interface is used for events that need to be notified when the
+ * {@link NetworkClientDelegate#getAndClearMetadataError()} has an error.
+ */
+public interface MetadataErrorNotifiableEvent {
+
+ /**
+ * The background thread detects metadata errors on every call to {@link
NetworkClientDelegate#poll(long, long)}.
+ * {@link NetworkClientDelegate} calls {@link
Metadata#maybeThrowAnyException()} and stores the result.
+ * The presence of a metadata error is checked in the {@link
ConsumerNetworkThread}'s loop by calling
+ * {@link NetworkClientDelegate#getAndClearMetadataError()}. There are two
places in the loop in which the
+ * metadata error is checked:
+ *
+ * <ul>
+ * <li>
+ * At the very top of the {@link ConsumerNetworkThread}'s loop,
the {@link ApplicationEventHandler}'s
+ * queue is drained. Before processing each event via
+ * {@link ApplicationEventProcessor#process(ApplicationEvent)}, if
a metadata error occurred, this method
+ * will be invoked on the event if it implements this interface.
+ * <p/>
+ * <em>Note</em>: for an event on which this method is invoked, it
will <em>not</em> be passed to the
+ * {@link ApplicationEventProcessor#process(ApplicationEvent)}
method.
+ * </li>
+ * <li>
+ * At the very bottom of the {@link ConsumerNetworkThread}'s loop,
the {@link CompletableEventReaper}
+ * is executed and any outstanding event is returned. If a
metadata error occurred, this method
+ * will be invoked on all unexpired events if it implements this
interface.
+ * </li>
+ * </ul>
+ *
+ * @param metadataException Error that originally came from {@link
Metadata#maybeThrowAnyException()}
+ */
+ void completeExceptionallyWithMetadataError(Exception metadataException);
Review Comment:
nit: would `onMetadataError` be more to the point?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThread.java:
##########
@@ -193,10 +194,13 @@ private void processApplicationEvents() {
try {
if (event instanceof CompletableEvent) {
applicationEventReaper.add((CompletableEvent<?>) event);
- // Check if there are any metadata errors and fail the
CompletableEvent if an error is present.
- // This call is meant to handle "immediately completed
events" which may not enter the awaiting state,
- // so metadata errors need to be checked and handled right
away.
- maybeFailOnMetadataError(List.of((CompletableEvent<?>)
event));
+ }
+ // Check if there are any metadata errors and fail the
CompletableEvent if an error is present.
+ // This call is meant to handle "immediately completed events"
which may not enter the awaiting state,
+ // so metadata errors need to be checked and handled right
away.
+ if (event instanceof MetadataErrorNotifiableEvent) {
+ if
(maybeFailOnMetadataError((MetadataErrorNotifiableEvent) event))
+ continue;
Review Comment:
related to the comment above, if we revert this back to how it was maybe we
don't need 2 overloads of the `maybeFailOnMetadataError`?
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/MetadataErrorNotifiableEvent.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals.events;
+
+import org.apache.kafka.clients.Metadata;
+import org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread;
+import org.apache.kafka.clients.consumer.internals.NetworkClientDelegate;
+
+/**
+ * This interface is used for events that need to be notified when the
+ * {@link NetworkClientDelegate#getAndClearMetadataError()} has an error.
+ */
+public interface MetadataErrorNotifiableEvent {
Review Comment:
this interface could serve to any component wanting to register to metadata
errors, so better to leave the "event" term out of it?
Maybe `MetadataErrorListener` ? (I'm probably influenced by the existing
`ClusterResourceListener`, would be consistent)
##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/events/MetadataErrorNotifiableEvent.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.clients.consumer.internals.events;
+
+import org.apache.kafka.clients.Metadata;
+import org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread;
+import org.apache.kafka.clients.consumer.internals.NetworkClientDelegate;
+
+/**
+ * This interface is used for events that need to be notified when the
+ * {@link NetworkClientDelegate#getAndClearMetadataError()} has an error.
+ */
+public interface MetadataErrorNotifiableEvent {
+
+ /**
+ * The background thread detects metadata errors on every call to {@link
NetworkClientDelegate#poll(long, long)}.
+ * {@link NetworkClientDelegate} calls {@link
Metadata#maybeThrowAnyException()} and stores the result.
+ * The presence of a metadata error is checked in the {@link
ConsumerNetworkThread}'s loop by calling
+ * {@link NetworkClientDelegate#getAndClearMetadataError()}. There are two
places in the loop in which the
+ * metadata error is checked:
+ *
+ * <ul>
+ * <li>
+ * At the very top of the {@link ConsumerNetworkThread}'s loop,
the {@link ApplicationEventHandler}'s
+ * queue is drained. Before processing each event via
+ * {@link ApplicationEventProcessor#process(ApplicationEvent)}, if
a metadata error occurred, this method
+ * will be invoked on the event if it implements this interface.
+ * <p/>
+ * <em>Note</em>: for an event on which this method is invoked, it
will <em>not</em> be passed to the
+ * {@link ApplicationEventProcessor#process(ApplicationEvent)}
method.
+ * </li>
+ * <li>
+ * At the very bottom of the {@link ConsumerNetworkThread}'s loop,
the {@link CompletableEventReaper}
+ * is executed and any outstanding event is returned. If a
metadata error occurred, this method
+ * will be invoked on all unexpired events if it implements this
interface.
+ * </li>
+ * </ul>
+ *
+ * @param metadataException Error that originally came from {@link
Metadata#maybeThrowAnyException()}
+ */
Review Comment:
Agree, but this is all implemented within the `ConsumerNetworkThread.poll`
really, so I wonder if it makes sense to have it as java doc for this
interface? (any changes there we would surely update the java doc for poll but
most probably not here). What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
