[
https://issues.apache.org/jira/browse/KAFKA-17696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886770#comment-17886770
]
Lianet Magrans commented on KAFKA-17696:
----------------------------------------
Sure, thanks for your help! I haven't thought much about it yet but left an
initial idea that comes to mind. Take a closer look and let me know if you have
questions or think of other alternatives. Here to help! Once we sort this one
out we should be closer to unblocking
https://issues.apache.org/jira/browse/KAFKA-17337
> New consumer background operations unaware of metadata errors
> -------------------------------------------------------------
>
> Key: KAFKA-17696
> URL: https://issues.apache.org/jira/browse/KAFKA-17696
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Lianet Magrans
> Assignee: 黃竣陽
> Priority: Blocker
> Labels: consumer-threading-refactor
> Fix For: 4.0.0
>
>
> When a metadata error happens (ie. Unauthorized topic), the network layer is
> the one to detect it and it just propagates it to the app thread via en
> ErrorEvent.
> [https://github.com/apache/kafka/blob/0edf5dbd204df9eb62bfea1b56993e95737df5a3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/NetworkClientDelegate.java#L153]
> That allows api calls that processBackgroundEvents to throw the error in the
> app thread (ex. poll, unsubscribe and close, which are the only api calls
> that currently processBackgroundEvents).
> This means that all other api calls that do not processBackgroundEvent will
> never know about errors like Unauthorized topics. Moreover, it really means
> that the background operations are not notified/aborted when a metadata error
> happens (auth error). Ex. call to position block waiting for the
> updateFetchPositions
> ([here|https://github.com/apache/kafka/blob/0edf5dbd204df9eb62bfea1b56993e95737df5a3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1586]),
> will leave a
> pendingOffsetFetchEvent waiting to complete, even when the background already
> got an Unauthorized exception (but it only passed it to the app thread via
> ErrorEvent)
> I wonder if we should ensure that metadata errors are not only propagated to
> the app thread via ErrorEvents, but also ensure that we notify all request
> managers in the background (so that they can decide if completeExceptionally
> their outstanding events). Ex. OffsetsRequestManager.onMetadataError should
> completeExceptionally the pendingOffsetFetchEvent (just first thought, there
> could be other approaches, but note that calling processBackgroundEvent in
> api calls like positions will not do because we would block first on the
> CheckAndUpdatePositions, then processBackgroundEvents that would only happen
> after the CheckAndUpdate)
>
> This behaviour can be repro with the integration test
> AuthorizerIntegrationTest.testOffsetFetchWithNoAccess with the new consumer
> enabled (discovered with [https://github.com/apache/kafka/pull/17107] )
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
