Re: Streams changelog topic retention is high

[Sameer Kumar]

Thanks Damian, I worked. I changed the  StreamsConfig.WINDOW_STORE_
CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG  and was able to reduce the
retention time of the changelog topic.

-Sameer.

On Mon, Oct 30, 2017 at 9:38 PM, Damian Guy  wrote:

> The retention for the joins is as specified above. With until set to 240
> and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG
> set to 1 day (the default) this would be 10080
>
> For plain key value stores, there should be no retention period as the
> topics are compacted only.
>
> On Mon, 30 Oct 2017 at 15:48 Sameer Kumar  wrote:
>
> > Actually I am using Key Value store, I do use join as part of my
> DAG(until
> > for the same has been set at 240 mins). The sink processor is key-value,
> is
> > there any option to control it.
> >
> > -Sameer.
> >
> > On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy 
> wrote:
> >
> > > The topics in question are both changelogs for window stores. The
> > retention
> > > period for them is calculated as the Window retention period, which is
> > the
> > > value that is passed to `JoinWindows.until(...)` (default is 1 day)
> plus
> > > the value of the config
> > > StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> > > default is 1 day)
> > >
> > >
> > >
> > > On Mon, 30 Oct 2017 at 12:49 Sameer Kumar 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have configured my settings to be the following:-
> > > >
> > > > log.retention.hours=3
> > > > delete.topic.enable=true
> > > > delete.retention.ms=1080
> > > > min.cleanable.dirty.ratio=0.20
> > > > segment.ms=18
> > > >
> > > > Howsoever, the changelog topic created as part of stream has the
> > > > rentention.ms to be 10080, the source topic has it to be 3
> hours.
> > > >
> > > > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > > > --zookeeper 172.29.65.91:2181 --topic
> > > > c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> > > > Topic:c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> > > > PartitionCount:60   ReplicationFactor:1 Configs:retention.ms
> =
> > > > 10080,cleanup.policy=delete,compact
> > > >
> > > > Can someone please explain this behavior.
> > > >
> > > > -Sameer.
> > > >
> > >
> >
>

Re: Streams changelog topic retention is high

[Damian Guy]

The retention for the joins is as specified above. With until set to 240
and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG
set to 1 day (the default) this would be 10080

For plain key value stores, there should be no retention period as the
topics are compacted only.

On Mon, 30 Oct 2017 at 15:48 Sameer Kumar  wrote:

> Actually I am using Key Value store, I do use join as part of my DAG(until
> for the same has been set at 240 mins). The sink processor is key-value, is
> there any option to control it.
>
> -Sameer.
>
> On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy  wrote:
>
> > The topics in question are both changelogs for window stores. The
> retention
> > period for them is calculated as the Window retention period, which is
> the
> > value that is passed to `JoinWindows.until(...)` (default is 1 day) plus
> > the value of the config
> > StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> > default is 1 day)
> >
> >
> >
> > On Mon, 30 Oct 2017 at 12:49 Sameer Kumar 
> wrote:
> >
> > > Hi,
> > >
> > > I have configured my settings to be the following:-
> > >
> > > log.retention.hours=3
> > > delete.topic.enable=true
> > > delete.retention.ms=1080
> > > min.cleanable.dirty.ratio=0.20
> > > segment.ms=18
> > >
> > > Howsoever, the changelog topic created as part of stream has the
> > > rentention.ms to be 10080, the source topic has it to be 3 hours.
> > >
> > > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > > --zookeeper 172.29.65.91:2181 --topic
> > > c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> > > Topic:c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> > > PartitionCount:60   ReplicationFactor:1 Configs:retention.ms=
> > > 10080,cleanup.policy=delete,compact
> > >
> > > Can someone please explain this behavior.
> > >
> > > -Sameer.
> > >
> >
>

Re: Streams changelog topic retention is high

[Sameer Kumar]

Actually I am using Key Value store, I do use join as part of my DAG(until
for the same has been set at 240 mins). The sink processor is key-value, is
there any option to control it.

-Sameer.

On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy  wrote:

> The topics in question are both changelogs for window stores. The retention
> period for them is calculated as the Window retention period, which is the
> value that is passed to `JoinWindows.until(...)` (default is 1 day) plus
> the value of the config
> StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
> default is 1 day)
>
>
>
> On Mon, 30 Oct 2017 at 12:49 Sameer Kumar  wrote:
>
> > Hi,
> >
> > I have configured my settings to be the following:-
> >
> > log.retention.hours=3
> > delete.topic.enable=true
> > delete.retention.ms=1080
> > min.cleanable.dirty.ratio=0.20
> > segment.ms=18
> >
> > Howsoever, the changelog topic created as part of stream has the
> > rentention.ms to be 10080, the source topic has it to be 3 hours.
> >
> > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> > --zookeeper 172.29.65.91:2181 --topic
> > c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> > Topic:c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> > PartitionCount:60   ReplicationFactor:1 Configs:retention.ms=
> > 10080,cleanup.policy=delete,compact
> >
> > Can someone please explain this behavior.
> >
> > -Sameer.
> >
>

Re: Streams changelog topic retention is high

[Damian Guy]

The topics in question are both changelogs for window stores. The retention
period for them is calculated as the Window retention period, which is the
value that is passed to `JoinWindows.until(...)` (default is 1 day) plus
the value of the config
StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG (
default is 1 day)



On Mon, 30 Oct 2017 at 12:49 Sameer Kumar  wrote:

> Hi,
>
> I have configured my settings to be the following:-
>
> log.retention.hours=3
> delete.topic.enable=true
> delete.retention.ms=1080
> min.cleanable.dirty.ratio=0.20
> segment.ms=18
>
> Howsoever, the changelog topic created as part of stream has the
> rentention.ms to be 10080, the source topic has it to be 3 hours.
>
> [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
> --zookeeper 172.29.65.91:2181 --topic
> c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> Topic:c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
> PartitionCount:60   ReplicationFactor:1 Configs:retention.ms=
> 10080,cleanup.policy=delete,compact
>
> Can someone please explain this behavior.
>
> -Sameer.
>

Streams changelog topic retention is high

[Sameer Kumar]

Hi,

I have configured my settings to be the following:-

log.retention.hours=3
delete.topic.enable=true
delete.retention.ms=1080
min.cleanable.dirty.ratio=0.20
segment.ms=18

Howsoever, the changelog topic created as part of stream has the
rentention.ms to be 10080, the source topic has it to be 3 hours.

[root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe
--zookeeper 172.29.65.91:2181 --topic
c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
Topic:c-7-aq7-KSTREAM-JOINOTHER-19-store-changelog
PartitionCount:60   ReplicationFactor:1 Configs:retention.ms=
10080,cleanup.policy=delete,compact

Can someone please explain this behavior.

-Sameer.