clolov commented on PR #14136: URL: https://github.com/apache/kafka/pull/14136#issuecomment-1693019922
Sorry, I do not understand the concern. Let me explain my reasoning. Based on https://github.com/apache/kafka/pull/14136#issuecomment-1691749594 the controller sends blocking requests in the order LISR, UpdateMetadata, StopReplica (which eventually trickle down to `stopPartitions`). Let's consider the usage of the pseudo metadata cache (from now on referred to as the concurrent map) in the RemoteLogManager before the change proposed in this pull request. `onLeadershipChange` does not lookup topic identifiers in the concurrent map, it only populates it with entries. The places where topic identifiers are looked up are in `stopPartitions`, `fetchRemoteLogSegmentMetadata`, and `findOffsetByTimestamp`. `stopPartitions` is only called last in the list of requests sent by the controller. This cannot arrive before the previous update metadata has propagated due to the blocking behaviour of the controller sending requests. Both `fetchRemoteLogSegmentMetadata` and `findOffsetByTimestamp` are on the read path of Tiered Storage i.e. they are triggered by consumers. In order for consumers to continue consuming after a leadership change they need to get the latest metadata, as such update metadata requests need to have propagated through the cluster before brokers start returning the new leaders. Now let's change the concurrent map with the metadata cache. I believe the same logic still holds true i.e. an update metadata request containing the new topic identifiers needs to have propagated through the cluster before any requests using said topic identifiers can start to be serviced by the broker. Am I wrong somewhere in my reasoning? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
