Sophie Blee-Goldman commented on KAFKA-7591:

{quote}Shouldn't you need to reset the app if the window size changes?
What I mean is, isn't it semantically incorrect to suddenly change the window 
size (without resetting)? For example the current WindowStore implementations 
store the window start time but not the end time, and don't persist the window 
size in any way. If the window size is suddenly changed from 1 minute to 5 min, 
all the currently stored results will be inaccurate as they only contain data 
within 1 min, but Streams will now interpret it containing all the data within 
that 5 minute period. We could in theory recompute all the windows, but only if 
we knew the old window size (and even then that would be quite painful). If it 
was originally a 1 min tumbling window, and it changes to a 5 min tumbling 
window, we will lose everything in every 4/5 windows, because Streams will only 
lookup the windows starting at 0min, 5min, 10min, etc (whereas before we had 
windows starting at each 1min interval)

> Changelog retention period doesn't synchronise with window-store size
> ---------------------------------------------------------------------
>                 Key: KAFKA-7591
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7591
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Jon Bates
>            Priority: Major
> When a new windowed state store is created, the associated changelog topic's 
> `retention.ms` value is set to `window-size + 
> h3. Expected Behaviour
> If the window-size is updated, the changelog topic's `retention.ms` config 
> should be updated to reflect the new size
> h3. Actual Behaviour
> The changelog-topic's `retention.ms` setting is not amended, resulting in 
> possible loss of data upon application restart
> n.b. Although it is easy to update changelog topic config, I logged this as 
> `major` due to the potential for data-loss for any user of Kafka-Streams who 
> may not be intimately aware of the relationship between a windowed store and 
> the changelog config

This message was sent by Atlassian Jira

Reply via email to