Hangleton commented on PR #13240: URL: https://github.com/apache/kafka/pull/13240#issuecomment-1454676488
Hello David, found a case where the use of OffsetCommit requests version 9 from the consumer resulted in an error in the ACL authorization tests. The reason is that when the commit offsets are performed by the tests to exercise ACLs, the topic IDs are not yet present in the consumer metadata cache, in this case because the consumer hasn't subscribed to the topics or be assigned any of their partitions. As a result, the OffsetCommitRequest version 9 was sent with zero topic ids. In order to avoid this, we could: 1. Enforce a metadata update for any topic in the offsets to commit, using the transient topics list exposed by the `ConsumerMetadata` and used for instance by the `OffsetFetcher` to fetch offsets for out-of-band topic-partitions which aren't part of any subscription. This approach however adds complexity to the offset commit implementation in the consumer and only address this specific use case. There could be other cases where metadata hasn't converged yet and the topic id would not be available to the consumer. 2. Adds a condition on the support of topic ids when constructing the `OffsetCommitRequest`. This is the approach used when constructing the `Fetch` request. The advantage of the approach is that it provides the invariant that any OffsetCommitRequest version >= 9 have valid (non-zero) topic ids in it. One downside is that if a bug in the consumer makes a topic id unavailable, the consumer will keep using version <= 8 permanently and silently, while we would want to know about it as implementors to address any potential gap. The second approach is currently implemented in the PR. Happy to discuss more about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
