Hello Guozhang thanks.
So after reading much more docs I still do not have the complete picture. These are our relevant settings from kafka broker configuration: log.cleanup.policy=delete # set log.retention.bytes to 15 gb log.retention.bytes=16106127360 # set log.retention.hours to 30 days log.retention.hours=720 Though one of internal kafka stream topics (with cleanup.policy = compact) grew to ~40gb today. What am I missing? I thought these settings should take care that a given topic is never >15 gb, right? Thanks Björn > On 29. Mar 2018, at 00:05, Guozhang Wang <wangg...@gmail.com> wrote: > > Hello, > > You can set the topic-level configs via the > StreamsConfig#topicPrefix(String), please find the following web docs > (search for KIP-173): > > https://kafka.apache.org/documentation/streams/upgrade-guide#streams_api_changes_100 > > > Guozhang > > > > On Wed, Mar 28, 2018 at 3:23 AM, Björn Häuser <bjoernhaeu...@gmail.com> > wrote: > >> Hello Everyone, >> >> we are running a Kafka Streams Application with does time window >> aggregates (using kafka 1.0.0). >> >> Unfortunately one of the changelog topics is now growing quite a bit in >> size maxing out the brokers. I did not find any settings in the kafka >> stream properties to configure retention and went ahead and set it >> retention.bytes to 15gb. Unfortunately this does not seem to apply and the >> topic size is still around 140gb. >> >> Is this intended? I could not find any documentation about setting the >> retention size for the internal topics. >> >> Thanks >> Björn >> >> >> > > > -- > -- Guozhang
