Re: [VOTE] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

Hi Everyone,

This KIP passes with three binding +1 votes from Matthias, Bill, and Guozhang 
and two non-binding +1 from votes from John and Bruno.
I would like to thank everyone for the feedback and votes, with special thanks 
to John for guiding me through my first KIP!

A very early stage PR is available at https://github.com/apache/kafka/pull/6593.

Cheers,
Maarten

On Mon, 22 Apr 2019 07:55 AM John Roesler 
wrote:

> Hi Maarten,
>
> Congratulations, your KIP has 3 binding votes, no vetoes, and has been open
> for over 72 hours, so you can close the vote and submit a PR for review!
>
> The way to close the vote is you just reply to this thread and tally up the
> votes, and say that the KIP vote has passed (see the other vote threads for
> examples). Then, you can update:
> * your KIP wiki to change the status to "Adopted"
> * the main KIP page to move your KIP to the "Adopted" section (It should be
> in the "under discussion" section
> * the Kafka Streams page, to move your KIP to the "Adopted" section (
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams)
>
> Looking forward to the PR!
>
> -John
>
> On Thu, Apr 18, 2019 at 7:58 PM Matthias J. Sax 
> wrote:
>
>>Thanks a lot for the KIP!
>>
>>
>> +1 (binding)
>>
>>
>> -Matthias
>>
>> On 4/18/19 12:31 PM, Guozhang Wang wrote:
>> > +1 (binding).
>> >
>> > Thanks.
>> >
>> > On Thu, Apr 18, 2019 at 10:48 AM John Roesler  wrote:
>> >
>> >> Thanks, Maarten!
>> >>
>> >> +1 (non-binding)
>> >>
>> >> -John
>> >>
>> >> On Wed, Apr 17, 2019 at 1:45 PM Bill Bejeck  wrote:
>> >>
>> >>> Thanks for the KIP.
>> >>>
>> >>> +1(binding)
>> >>>
>> >>> -Bill
>> >>>
>> >>> On Wed, Apr 17, 2019 at 12:58 PM Bruno Cadonna 
>> >> wrote:
>> >>>
>> >>>> Hi Maarten Duijn,
>> >>>>
>> >>>> Thank you for driving this.
>> >>>>
>> >>>> +1 (non-binding)
>> >>>>
>> >>>> Best,
>> >>>> Bruno
>> >>>>
>> >>>> On Wed, Apr 17, 2019 at 8:21 AM Maarten Duijn 
>> >>>> wrote:
>> >>>>
>> >>>>> Hello all,
>> >>>>>
>> >>>>> There has been informal agreement so I would like to call for a vote
>> >> on
>> >>>>> KIP-446: Add changelog topic configuration to KTable suppress. This
>> >>> will
>> >>>>> allow users to configure internal topics created by the suppress
>> >>> operator
>> >>>>> via the streams DSL.
>> >>>>>
>> >>>>> KIP:
>> >>>>>
>> >>>>
>> >>>
>> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress
>> >>>>> JIRA: https://issues.apache.org/jira/browse/KAFKA-8147<
>> >>>>> https://issues.apache.org/jira/browse/KAFKA-8029>
>> >>>>> PR: will follow shortly
>> >>>>>
>> >>>>> Cheers,
>> >>>>> Maarten
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>> >
>>
>>

Re: [VOTE] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

For those interested, a WIP PR is now available at: 
https://github.com/apache/kafka/pull/6593

Maarten

[VOTE] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

Hello all,

There has been informal agreement so I would like to call for a vote on 
KIP-446: Add changelog topic configuration to KTable suppress. This will allow 
users to configure internal topics created by the suppress operator via the 
streams DSL.

KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress
JIRA: 
https://issues.apache.org/jira/browse/KAFKA-8147
PR: will follow shortly

Cheers,
Maarten

Re: [DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

Thank you for the explanation regarding the internals, I have edited the KIP 
accordingly and updated the Javadoc. About the possible data loss when altering 
changelog config, I think we can improve by doing (one of) the following.

1) Add a warning in the comments that clearly states what might happen when 
change logging is disabled and adjust it when persistent stores are added.

2) Change `withLoggingDisabled` to `minimizeLogging`. Instead of disabling 
logging, a call to this method minimizes the topic size by aggressively 
removing the records emitted downstream by the suppress operator. I believe 
this can be achieved by setting `delete.retention.ms=0` in the topic config.

3) Remove `withLoggingDisabled` from the proposal.

4) Leave both methods as-proposed, as you indicated, this is in line with the 
other parts of the Streams API

A user might want to disable logging when downstream is not a Kafka topic but 
some other service that does not benefit from atleast-once-delivery of the 
suppressed records in case of failover or rebalance.
Seeing as it might cause data loss, the methods should not be used lightly and 
I think some comments are warranted. Personally, I rely purely on Kafka to 
prevent data loss even when a store persisted locally, so when support is added 
for persistent suppression, I feel the comments may stay.

Maarten

[DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

Kafka Streams currently does not allow configuring the internal changelog
topic created by KTable.suppress. This KIP introduces a design for adding
topic configurations to the suppress API.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress

[DISCUSS] KIP-446: Add changelog topic configuration to KTable suppress

[Maarten Duijn]

Kafka Streams currently does not allow configuring the internal changelog
topic created by KTable.suppress. This KIP introduces a design for adding
topic configurations to the suppress API.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-446%3A+Add+changelog+topic+configuration+to+KTable+suppress