[ 
https://issues.apache.org/jira/browse/KAFKA-21144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Lau reassigned KAFKA-21144:
--------------------------------

    Assignee: Alan Lau  (was: Lianet Magrans)

> Producer auth errors may become stale but still be thrown later on cold/new 
> topics
> ----------------------------------------------------------------------------------
>
>                 Key: KAFKA-21144
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21144
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients, producer 
>            Reporter: Lianet Magrans
>            Assignee: Alan Lau
>            Priority: Major
>
> Whenever the producer encounters an auth failure, it's recorded as a fatal 
> error in the producer's metadata, and it's thrown only when the producer has 
> to wait for metadata for a topic (this would be when using a new topic or 
> "idle" topics, with no produce for longer than the metadata.max.idle).
> These errors are only cleared when thrown, which means that it could remain 
> in the internal cache for a long time if the producer lets some topics go 
> idle. So a producer that keeps producing to its hot topics keeps the error 
> indefinitely and fails the first time it produces to an idle topic or to a 
> new topic (throwing the original fatal auth expcetion it kept in cache, it 
> clears it at that point, and a retry would succeed). But this could be 
> possibly hours or days after the actual auth issue was resolved. 
> This initially made sense I expect given that auth errors were expected to be 
> strictly non-transient, "bad credentials" that would require app action 
> (change creds, restart). But the auth landscape has evolved, now we have 
> oauth layers in between, and we've seen how a producer could see a 
> token-related transient auth failure, that apps would still throw long after 
> it got solved (e.g., if letting some topics go idle, then produce again)
> This task is to review if we could improve how the producer recovers from 
> these kind of scenarios where auth failures could recover. We need to 
> carefully consider that at the moment, fatal errors are an umbrella for 
> several (Bootstrap, UnsupportedVersion, Auth).
> Even though the fatal error recording is at the network layer, I expect this 
> "stale + throw late" issue can happen only in the producer (consumers do the 
> maybeThrow on every poll) 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to