[
https://issues.apache.org/jira/browse/KAFKA-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reopened KAFKA-4281:
------------------------------------
> Should be able to forward aggregation values immediately
> --------------------------------------------------------
>
> Key: KAFKA-4281
> URL: https://issues.apache.org/jira/browse/KAFKA-4281
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.1.0
> Reporter: Greg Fodor
> Assignee: Greg Fodor
> Priority: Major
>
> KIP-63 introduced changes to the behavior of aggregations such that the
> result of aggregations will not appear to subsequent processors until a state
> store flush occurs. This is problematic for latency sensitive aggregations
> since flushes occur generally at commit.interval.ms, which is usually a few
> seconds. Combined with several aggregations, this can result in several
> seconds of latency through a topology for steps dependent upon aggregations.
> Two potential solutions:
> - Allow finer control over the state store flushing intervals
> - Allow users to change the behavior so that certain aggregations will
> immediately forward records to the next step (as was the case pre-KIP-63)
> A PR is attached that takes the second approach. To add this unfortunately a
> large number of files needed to be touched, and this effectively doubles the
> number of method signatures around grouping on KTable and KStream. I tried an
> alternative approach that let the user opt-in to immediate forwarding via an
> additional builder method on KGroupedStream/Table but this didn't work as
> expected because in order for the latency to go away, the KTableImpl itself
> must also mark its source as forward immediate (otherwise we will still see
> latency due to the materialization of the KTableSource still relying upon
> state store flushes to propagate.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
