lucasbru commented on PR #15000:
URL: https://github.com/apache/kafka/pull/15000#issuecomment-1878769287
@philipnee some considerations:
- Does the old consumer guarantee that the interceptor is called when
`poll` or `close` are not called by the consumer anymore after an autocommit? I
don't think so, because it's running as part of the application thread, so how
would it provide that guarantee without even being called?
- I think the interceptor is meant for monitoring / metrics -- it doesn't
have the strict requirements as, for example, an ongoing commit shouldn't be
dropped during close because it will result in data loss or duplicate
processing.
"Make the consumer interceptor thread-safe" -- How would we even do that?
Consumer intereceptors are implemented by the user, we cannot make them thread
safe by just putting a lock on it. The only way would be to write a KIP, have a
breaking change and a migration guide, and ask all users to upgrade their code.
Not sure that is worth it, given the original consumer does not seem to provide
the guarantee that you are asking for.
I think the best we can do is to insert the interceptor callback into the
invoker queue upon commit completion, and make sure during close to execute all
commit callbacks that are enqueued, no? If the user fails to call close, we are
out of luck anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
