Lianet Magrans created KAFKA-21144:
--------------------------------------
Summary: Producer auth errors may become stale but still be thrown
later on cold/new topics
Key: KAFKA-21144
URL: https://issues.apache.org/jira/browse/KAFKA-21144
Project: Kafka
Issue Type: Bug
Components: clients, producer
Reporter: Lianet Magrans
Assignee: Lianet Magrans
Whenever the producer encounters an auth failure, it's recorded as a fatal
error in the producer's metadata, and it's thrown only when the producer has to
wait for metadata for a topic (this would be when using a new topic or "idle"
topics, with no produce for longer than the metadata.max.idle).
These errors are only cleared when thrown, which means that it could remain in
the internal cache for a long time if the producer lets some topics go idle. So
a producer that keeps producing to its hot topics keeps the error indefinitely
and fails the first time it produces to an idle topic or to a new topic
(throwing the original fatal auth expcetion it kept in cache, it clears it at
that point, and a retry would succeed). But this could be possibly hours or
days after the actual auth issue was resolved.
This initially made sense I expect given that auth errors were expected to be
strictly non-transient, "bad credentials" that would require app action (change
creds, restart). But the auth landscape has evolved, now we have oauth layers
in between, and we've seen how a producer could see a token-related transient
auth failure, that apps would still throw long after it got solved (e.g., if
letting some topics go idle, then produce again)
This task is to review if we could improve how the producer recovers from these
kind of scenarios where auth failures could recover. We need to carefully
consider that at the moment, fatal errors are an umbrella for several
(Bootstrap, UnsupportedVersion, Auth).
Even though the fatal error recording is at the network layer, I expect this
"stale + throw late" issue can happen only in the producer (consumers do the
maybeThrow on every poll)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
