Guozhang Wang created KAFKA-14641:
-------------------------------------
Summary: Cleanup CommitNeeded after EOS-V1 is removed
Key: KAFKA-14641
URL: https://issues.apache.org/jira/browse/KAFKA-14641
Project: Kafka
Issue Type: Improvement
Reporter: Guozhang Wang
This is a follow-up of KAFKA-14294.
Today we have several flags to determine if KS need to execute a commit: 1)
task-level "commitNeeded" which is set whenever process() or punctuator() is
called, 2) if there are input topic offsets to commit, retrieved from the
"task.prepareCommit()", 3) the "transactionInFlight" flag from producer as a
fix of KAFKA-14294 (this subsumes the first "commitNeeded" functionality).
Given that we are still having EOS-v1, cleanup this would be a bit complex. But
after the deprecated EOS-V1 is removed, we can cleanup those controls since for
any commit cases, we would need to commit all tasks anyways whereas in EOS-v1,
we would commit probably a subset of tasks since they are done by different
producers and hence different txns.
A quick thought is the following:
1) We would not need the per-task "commitNeeded" anymore.
2) We would maintain a single "commitNeeded" flag on the task-executor, hence
on the thread level. It is set whenever `process()` or `punctuator` is called.
3) Whenever we need to commit, either a) periodically, b) upon revocation, c)
upon user request, we simply check that flag, and if necessary commit all tasks
and reset the flag.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
