lianetm commented on PR #20324: URL: https://github.com/apache/kafka/pull/20324#issuecomment-3559906123
Another gap I noticed, not when collecting from the buffer, but when sending pre-fetches (so it's not directly with the changes of this PR, but related). Here we're ensuring we collect data from the buffer only when there is no need to validate positions (or when positions where validated). Following a similar principle, I expect we should ensure pre-fetches are only built when we have update the partition state (metadata changed? needs validation?). The classic consumer ensures that here : https://github.com/apache/kafka/blob/dc9ae8bf805a93896104a51417318729e032fc3e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ClassicKafkaConsumer.java#L674-L675 But I don't see how we are ensuring that on the async? we send pre-fetches by directly triggering a `CreateFetchRequestsEvent`, which will generate fetch requests without `validatePositionsOnMetadataChange`. The gap seems clear when we look at the usages of the `fetchRequestManager.createFetchRequests()` - from AsyncPollEvent -> here it's ok, we only createFetchRequests after having updated positions - from process(CreateFetchRequestsEvent) -> this is the gap I'm seeing, triggered for pre-fetching Am I missing something or is it indeed a gap? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
