[
https://issues.apache.org/jira/browse/KAFKA-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sophie Blee-Goldman updated KAFKA-9450:
---------------------------------------
Description:
When EOS is turned on, the commit interval is set quite low (100ms) and all the
store layers are flushed during a commit. This is necessary for forwarding
records in the cache to the changelog, but unfortunately also forces rocksdb to
flush the current memtable before it's full. The result is a large number of
small writes to disk, losing the benefits of batching, and a large number of
very small L0 files that are likely to slow compaction.
Since we have to delete the stores to recreate from scratch anyways during an
unclean shutdown with EOS, we may as well skip flushing the innermost
StateStore during a commit and only do so during a graceful shutdown, before a
rebalance, etc. This is currently blocked on a refactoring of the state store
layers to allow decoupling the flush of the caching layer from the actual state
store.
Note that this is especially problematic with EOS due to the necessarily-low
commit interval, but still hurts even with at-least-once and a much larger
commit interval.
was:
When EOS is turned on, the commit interval is set quite low (100ms) and all the
store layers are flushed during a commit. This is necessary for forwarding
records in the cache to the changelog, but unfortunately also forces rocksdb to
flush the current memtable before it's full. The result is a large number of
small writes to disk, losing the benefits of batching, and a large number of
very small L0 files that are likely to slow compaction.
Since we have to delete the stores to recreate from scratch anyways during an
unclean shutdown with EOS, we may as well skip flushing the innermost
StateStore during a commit and only do so during a graceful shutdown, before a
rebalance, etc. This is currently blocked on a refactoring of the state store
layers to allow decoupling the flush of the caching layer from the actual state
store.
> Decouple inner state flushing from committing
> ---------------------------------------------
>
> Key: KAFKA-9450
> URL: https://issues.apache.org/jira/browse/KAFKA-9450
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Sophie Blee-Goldman
> Priority: Major
>
> When EOS is turned on, the commit interval is set quite low (100ms) and all
> the store layers are flushed during a commit. This is necessary for
> forwarding records in the cache to the changelog, but unfortunately also
> forces rocksdb to flush the current memtable before it's full. The result is
> a large number of small writes to disk, losing the benefits of batching, and
> a large number of very small L0 files that are likely to slow compaction.
> Since we have to delete the stores to recreate from scratch anyways during an
> unclean shutdown with EOS, we may as well skip flushing the innermost
> StateStore during a commit and only do so during a graceful shutdown, before
> a rebalance, etc. This is currently blocked on a refactoring of the state
> store layers to allow decoupling the flush of the caching layer from the
> actual state store.
> Note that this is especially problematic with EOS due to the necessarily-low
> commit interval, but still hurts even with at-least-once and a much larger
> commit interval.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
