kevin-wu24 commented on PR #23737:
URL: https://github.com/apache/kafka/pull/23737#issuecomment-6177682493

   > @kevin-wu24 Thanks for the PR. It's really a great find! The fix LGTM.
   > 
   > I think this bug also reveals we're actually missing a check that 
`MetadataLoader` only gets committed records Could we add a cheap check in 
`MetadataLoader#handleCommit` using the existing `highWaterMarkAccessor`, e.g:
   > 
   > ```
   > OptionalLong hwm = highWaterMarkAccessor.get();
   > if (hwm.isPresent() && batch.lastOffset() >= hwm.getAsLong()) {
   >     throw new IllegalStateException(...);
   > }
   > ```
   > 
   > With this fix in theory it'd never fire anyway, just a tripwire for future 
regressions. Or do you think it's overkill?
   
   Hi @0xffff-zhiyan,
   
   Specifically for the issue we observed, the `SnapshotEmitter` is the one who 
logged a metadata error that it was trying to write a snapshot past the HWM, so 
we did become aware of it via an exception being thrown. For brokers, metadata 
errors like that are non-fatal, and do not crash the process. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to