Re: State Store Data Retention

2021-05-10 Thread Bruno Cadonna
Hi Navneeth, I wrote that the *local state stores* are not affected when the topic configs cleanup.policy and retention.ms are passed to the state store. The *changelog topics* will consider the configs and they will remove data as specified in the configs. In the case of a state migration

Re: State Store Data Retention

2021-05-07 Thread Navneeth Krishnan
Hi Bruno/All, I have a follow up question regarding the same topic. As per you had mentioned there will be no impact to key value stores even when retention.ms and clean up policy is provided. Does that mean the change log topic will not clear the data in the broker even after the retention

Re: State Store Data Retention

2021-04-19 Thread Bruno Cadonna
Hi Upesh, The answers to your questions are: 1. The configs cleanup.policy and retention.ms are topic configs. Hence, they only affect the changelog of a state store, not the local state store in a Kafka Streams client. Locally, window and session stores remove data they do not need

State Store Data Retention

2021-04-18 Thread Upesh Desai
Hello, I have not been able to find a concrete answer on if/how state stores on a running kafka streams instance remove data when it has passed the configured retention.ms config. So a couple clarification questions: 1. If the stores are configured with: cleanup.policy=compact,delete AND