[
https://issues.apache.org/jira/browse/KAFKA-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979819#comment-16979819
]
ASF GitHub Bot commented on KAFKA-9224:
---------------------------------------
abbccdda commented on pull request #7737: KAFKA-9224 (EOS improvement): Flush
state store after transaction commit
URL: https://github.com/apache/kafka/pull/7737
This patch attempts to enforce a strict order to flush the state store only
after ongoing transaction gets committed under EOS. Major changes include:
- Rewrite `StreamTask.commit` to separate EOS and non EOS commit scenario
- Under EOS, we make the atomic operation in the following order: commit
ongoing transaction, flush the store, begin another transaction. This means
there is no intermediate data left within the cache and the ongoing
modification to the state store only gets visible after the transaction gets
committed.
- Add a `bounded` config on the stream thread cache to reject further
writes. This feature is important as under EOS the thread cache is the only
container we are going to leverage. When the cache is almost full, it will
begin throwing a `CacheFullException` to the caller.
- In ProcessorContext, handle `CacheFullException` by requesting a task
commit in next round. This should only be needed on ReadWrite access to state
stores for general put operation, and avoiding further interruption to the
application.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> State store should not see uncommitted transaction result
> ---------------------------------------------------------
>
> Key: KAFKA-9224
> URL: https://issues.apache.org/jira/browse/KAFKA-9224
> Project: Kafka
> Issue Type: Improvement
> Reporter: Boyang Chen
> Assignee: Boyang Chen
> Priority: Major
>
> Currently under EOS, the write to state store is not guaranteed to happen
> after the ongoing transaction is finished. This means interactive query could
> see uncommitted data within state store which is not ideal for users relying
> on state stores for strong consistency. Ideally, we should have an option to
> include state store commit as part of ongoing transaction, however an
> immediate step towards a better reasoned system is to `write after
> transaction commit`, which means we always buffer data within stream cache
> for EOS until the ongoing transaction is committed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
