RE: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Eric Wasserman]

During the vote it was suggested the name of the property be changed from:

log.cleaner.compaction.lag.ms

to

log.cleaner.compaction.delay.ms

Note that this feature concerns controlling the portion of the head of the
log (new messages go on the head) that will be left un-compacted (i.e. in
full detail) in the event that the log undergoes a compaction. This
property is not saying *when* to do a compaction rather it is how far a
compaction may go into the head. A compaction can still happen to the log
so we are not "delaying" the compaction itself, rather we are preventing
the compaction from proceeding all the way to the head of the log.

One other consideration
The original name ("lag") was proposed when non-temporal constraints were
being considered (size, # of messages). Those were deferred in the KIP
process. However, the new name ("delay") is (to my ear) purely temporal. In
the event use cases for other forms of compaction point configuration do
arise naming them consistently may be trickier.

To James Cheng's concern:

I just thought of something: what happens to the value of "
> log.cleaner.delete.retention.ms"?
> Does it still have the same meaning as before? Does the timer start when
> log compaction happens
> (as it currently does), so in reality, tombstones will only be removed
> from the log some time
> after (log.cleaner.min.compaction.lag.ms + log.cleaner.delete.retention.ms
> )?


Nothing about tombstones handling changes. The tombstone timer starts when
the tombstone is created (i.e. during compaction). As long as one
interprets "log.cleaner.delete.retention.ms" as the minimum time-to-live of
the tombstones then the meaning of that property doesn't change. The new
feature only indirectly makes the tombstone removal later by virtue of
delaying their creation.

RE: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Eric Wasserman]

Thanks everyone for voting.
The vote passed with 5 binding +1's (Gwen, Jay, Ewen, Ismael, Jun)
and with 9 nb's (Becket, James, Tom, Mnikumar, Ben, Grant, Joel, Eric)

The discussion should move to KAFKA-1981

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Gwen Shapira]

Hey, its been over 3 days since the vote started and we clearly have over 3
+1s.

Eric,

Feel free to close the vote and we can move to code reviews :)

Gwen

On Fri, May 27, 2016 at 8:08 AM, Jun Rao  wrote:

> Thanks for the proposal. +1
>
> Jun
>
> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman 
> wrote:
>
> > Hi,
> >
> > I would like to begin voting on KIP-58 - Make Log Compaction Point
> > Configurable
> >
> > KIP-58 is here:  <
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > >
> >
> > The Jira ticket KAFKA-1981 Make log compaction point configurable
> > is here: 
> >
> > The original pull request is here: <
> > https://github.com/apache/kafka/pull/1168>
> > (this includes configurations for size and message count lags that will
> be
> > removed per discussion of KIP-58).
> >
> > The vote will run for 72 hours.
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Jun Rao]

Thanks for the proposal. +1

Jun

On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman 
wrote:

> Hi,
>
> I would like to begin voting on KIP-58 - Make Log Compaction Point
> Configurable
>
> KIP-58 is here:  <
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> >
>
> The Jira ticket KAFKA-1981 Make log compaction point configurable
> is here: 
>
> The original pull request is here: <
> https://github.com/apache/kafka/pull/1168>
> (this includes configurations for size and message count lags that will be
> removed per discussion of KIP-58).
>
> The vote will run for 72 hours.
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Ismael Juma]

Sounds good.
On 25 May 2016 17:26, "Gwen Shapira"  wrote:

> All topic level names are inconsistent. We can have a separate discussion /
> KIP on getting out of that mess.
>
> Gwen
>
> On Wed, May 25, 2016 at 6:07 AM, Ismael Juma  wrote:
>
> > +1 (binding)
> >
> > I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside,
> I
> > did notice that the topic level config for `log.segment.delete.delay.ms`
> > (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> > inconsistent.
> >
> > Ismael
> >
> > On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> > name,
> > > and consistent with log.segment.delete.delay.ms. Checked configs for
> > other
> > > suffixes that seemed reasonable and despite only appearing in that one
> > > broker config, it seems the best match.
> > >
> > > -Ewen
> > >
> > > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
> > >
> > > > I'm +1 on the concept.
> > > >
> > > > As with others I think the core challenge is to express this in an
> > > > intuitive way, and carry the same terminology across the docs, the
> > > configs,
> > > > and docstrings for the configs. Pictures would help.
> > > >
> > > > -Jay
> > > >
> > > > On Tue, May 24, 2016 at 6:54 PM, James Cheng 
> > > wrote:
> > > >
> > > > > I'm not sure what are the rules for who is allowed to vote, but
> I'm:
> > > > >
> > > > > +1 (non-binding) on the proposal
> > > > >
> > > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> > little
> > > > > confusing.
> > > > >
> > > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > > similar.
> > > > >
> > > > > The KIP describes it as the portion of the topic "that will remain
> > > > > uncompacted", so if you're open to alternate names:
> > > > >
> > > > > "log.cleaner.uncompacted.range.ms"
> > > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> > tail"
> > > > > and "log head" mixed up...)
> > > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to
> have
> > > the
> > > > > word "retention" in non-time-based topics?)
> > > > >
> > > > > I just thought of something: what happens to the value of "
> > > > > log.cleaner.delete.retention.ms"? Does it still have the same
> > meaning
> > > as
> > > > > before? Does the timer start when log compaction happens (as it
> > > currently
> > > > > does), so in reality, tombstones will only be removed from the log
> > some
> > > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > > log.cleaner.delete.retention.ms)?
> > > > >
> > > > > -James
> > > > >
> > > > > > On May 24, 2016, at 5:46 PM, Becket Qin 
> > > wrote:
> > > > > >
> > > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > > >
> > > > > > I am wondering should we change the config name to "
> > > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > > configuration
> > > > > > name is a little confusing. I was thinking do we have a "max"
> lag?
> > > And
> > > > is
> > > > > > this "lag" a bad thing?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira  >
> > > > wrote:
> > > > > >
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Thanks for responding to all my original concerns in the
> > discussion
> > > > > thread.
> > > > > >>
> > > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > > eric.wasser...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> > Point
> > > > > >>> Configurable
> > > > > >>>
> > > > > >>> KIP-58 is here:  <
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > > 
> > > > > >>>
> > > > > >>> The Jira ticket KAFKA-1981 Make log compaction point
> configurable
> > > > > >>> is here: 
> > > > > >>>
> > > > > >>> The original pull request is here: <
> > > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > > >>> (this includes configurations for size and message count lags
> > that
> > > > will
> > > > > >> be
> > > > > >>> removed per discussion of KIP-58).
> > > > > >>>
> > > > > >>> The vote will run for 72 hours.
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Joel Koshy]

+1 on the proposal

Re: inconsistent names: KAFKA-3234
 has a patch and
discussion in the PR that should help address the inconsistencies and
various other issues but we decided it would need a small KIP. (If someone
else wishes to take over that feel free to)

On Wed, May 25, 2016 at 9:26 AM, Gwen Shapira  wrote:

> All topic level names are inconsistent. We can have a separate discussion /
> KIP on getting out of that mess.
>
> Gwen
>
> On Wed, May 25, 2016 at 6:07 AM, Ismael Juma  wrote:
>
> > +1 (binding)
> >
> > I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside,
> I
> > did notice that the topic level config for `log.segment.delete.delay.ms`
> > (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> > inconsistent.
> >
> > Ismael
> >
> > On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> > name,
> > > and consistent with log.segment.delete.delay.ms. Checked configs for
> > other
> > > suffixes that seemed reasonable and despite only appearing in that one
> > > broker config, it seems the best match.
> > >
> > > -Ewen
> > >
> > > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
> > >
> > > > I'm +1 on the concept.
> > > >
> > > > As with others I think the core challenge is to express this in an
> > > > intuitive way, and carry the same terminology across the docs, the
> > > configs,
> > > > and docstrings for the configs. Pictures would help.
> > > >
> > > > -Jay
> > > >
> > > > On Tue, May 24, 2016 at 6:54 PM, James Cheng 
> > > wrote:
> > > >
> > > > > I'm not sure what are the rules for who is allowed to vote, but
> I'm:
> > > > >
> > > > > +1 (non-binding) on the proposal
> > > > >
> > > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> > little
> > > > > confusing.
> > > > >
> > > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > > similar.
> > > > >
> > > > > The KIP describes it as the portion of the topic "that will remain
> > > > > uncompacted", so if you're open to alternate names:
> > > > >
> > > > > "log.cleaner.uncompacted.range.ms"
> > > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> > tail"
> > > > > and "log head" mixed up...)
> > > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to
> have
> > > the
> > > > > word "retention" in non-time-based topics?)
> > > > >
> > > > > I just thought of something: what happens to the value of "
> > > > > log.cleaner.delete.retention.ms"? Does it still have the same
> > meaning
> > > as
> > > > > before? Does the timer start when log compaction happens (as it
> > > currently
> > > > > does), so in reality, tombstones will only be removed from the log
> > some
> > > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > > log.cleaner.delete.retention.ms)?
> > > > >
> > > > > -James
> > > > >
> > > > > > On May 24, 2016, at 5:46 PM, Becket Qin 
> > > wrote:
> > > > > >
> > > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > > >
> > > > > > I am wondering should we change the config name to "
> > > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > > configuration
> > > > > > name is a little confusing. I was thinking do we have a "max"
> lag?
> > > And
> > > > is
> > > > > > this "lag" a bad thing?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira  >
> > > > wrote:
> > > > > >
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> Thanks for responding to all my original concerns in the
> > discussion
> > > > > thread.
> > > > > >>
> > > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > > eric.wasser...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> > Point
> > > > > >>> Configurable
> > > > > >>>
> > > > > >>> KIP-58 is here:  <
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > > 
> > > > > >>>
> > > > > >>> The Jira ticket KAFKA-1981 Make log compaction point
> configurable
> > > > > >>> is here: 
> > > > > >>>
> > > > > >>> The original pull request is here: <
> > > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > > >>> (this includes configurations for size and message count lags
> > that
> > > > will
> > > > > >> be
> > > > > >>> removed per discussion of KIP-58).
> > > > > >>>
> > > > > >>> The vote will run for 72 hours.
> > > > > >>>
> > > > > >>
> > > > >
> > > 

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Gwen Shapira]

All topic level names are inconsistent. We can have a separate discussion /
KIP on getting out of that mess.

Gwen

On Wed, May 25, 2016 at 6:07 AM, Ismael Juma  wrote:

> +1 (binding)
>
> I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside, I
> did notice that the topic level config for `log.segment.delete.delay.ms`
> (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> inconsistent.
>
> Ismael
>
> On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava 
> wrote:
>
> > +1 (binding)
> >
> > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> name,
> > and consistent with log.segment.delete.delay.ms. Checked configs for
> other
> > suffixes that seemed reasonable and despite only appearing in that one
> > broker config, it seems the best match.
> >
> > -Ewen
> >
> > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
> >
> > > I'm +1 on the concept.
> > >
> > > As with others I think the core challenge is to express this in an
> > > intuitive way, and carry the same terminology across the docs, the
> > configs,
> > > and docstrings for the configs. Pictures would help.
> > >
> > > -Jay
> > >
> > > On Tue, May 24, 2016 at 6:54 PM, James Cheng 
> > wrote:
> > >
> > > > I'm not sure what are the rules for who is allowed to vote, but I'm:
> > > >
> > > > +1 (non-binding) on the proposal
> > > >
> > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> little
> > > > confusing.
> > > >
> > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > similar.
> > > >
> > > > The KIP describes it as the portion of the topic "that will remain
> > > > uncompacted", so if you're open to alternate names:
> > > >
> > > > "log.cleaner.uncompacted.range.ms"
> > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> tail"
> > > > and "log head" mixed up...)
> > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> > the
> > > > word "retention" in non-time-based topics?)
> > > >
> > > > I just thought of something: what happens to the value of "
> > > > log.cleaner.delete.retention.ms"? Does it still have the same
> meaning
> > as
> > > > before? Does the timer start when log compaction happens (as it
> > currently
> > > > does), so in reality, tombstones will only be removed from the log
> some
> > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > log.cleaner.delete.retention.ms)?
> > > >
> > > > -James
> > > >
> > > > > On May 24, 2016, at 5:46 PM, Becket Qin 
> > wrote:
> > > > >
> > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > >
> > > > > I am wondering should we change the config name to "
> > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > configuration
> > > > > name is a little confusing. I was thinking do we have a "max" lag?
> > And
> > > is
> > > > > this "lag" a bad thing?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira 
> > > wrote:
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> Thanks for responding to all my original concerns in the
> discussion
> > > > thread.
> > > > >>
> > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > eric.wasser...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> Point
> > > > >>> Configurable
> > > > >>>
> > > > >>> KIP-58 is here:  <
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > 
> > > > >>>
> > > > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > > > >>> is here: 
> > > > >>>
> > > > >>> The original pull request is here: <
> > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > >>> (this includes configurations for size and message count lags
> that
> > > will
> > > > >> be
> > > > >>> removed per discussion of KIP-58).
> > > > >>>
> > > > >>> The vote will run for 72 hours.
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Grant Henke]

+1 (non-binding)

On Wed, May 25, 2016 at 8:20 AM, Ben Stopford  wrote:

> +1 (non-binding)
>
> > On 25 May 2016, at 14:07, Ismael Juma  wrote:
> >
> > +1 (binding)
> >
> > I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside,
> I
> > did notice that the topic level config for `log.segment.delete.delay.ms`
> > (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> > inconsistent.
> >
> > Ismael
> >
> > On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> >> +1 (binding)
> >>
> >> Agreed that the log.cleaner.compaction.delay.ms is probably a better
> name,
> >> and consistent with log.segment.delete.delay.ms. Checked configs for
> other
> >> suffixes that seemed reasonable and despite only appearing in that one
> >> broker config, it seems the best match.
> >>
> >> -Ewen
> >>
> >> On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
> >>
> >>> I'm +1 on the concept.
> >>>
> >>> As with others I think the core challenge is to express this in an
> >>> intuitive way, and carry the same terminology across the docs, the
> >> configs,
> >>> and docstrings for the configs. Pictures would help.
> >>>
> >>> -Jay
> >>>
> >>> On Tue, May 24, 2016 at 6:54 PM, James Cheng 
> >> wrote:
> >>>
>  I'm not sure what are the rules for who is allowed to vote, but I'm:
> 
>  +1 (non-binding) on the proposal
> 
>  I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
>  confusing.
> 
>  I like Becket's "log.cleaner.compaction.delay.ms", or something
> >> similar.
> 
>  The KIP describes it as the portion of the topic "that will remain
>  uncompacted", so if you're open to alternate names:
> 
>  "log.cleaner.uncompacted.range.ms"
>  "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> tail"
>  and "log head" mixed up...)
>  "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> >> the
>  word "retention" in non-time-based topics?)
> 
>  I just thought of something: what happens to the value of "
>  log.cleaner.delete.retention.ms"? Does it still have the same meaning
> >> as
>  before? Does the timer start when log compaction happens (as it
> >> currently
>  does), so in reality, tombstones will only be removed from the log
> some
>  time after (log.cleaner.min.compaction.lag.ms +
>  log.cleaner.delete.retention.ms)?
> 
>  -James
> 
> > On May 24, 2016, at 5:46 PM, Becket Qin 
> >> wrote:
> >
> > +1 (non-binding) on the proposal. Just a minor suggestion.
> >
> > I am wondering should we change the config name to "
> > log.cleaner.compaction.delay.ms"? The first glance at the
> >>> configuration
> > name is a little confusing. I was thinking do we have a "max" lag?
> >> And
> >>> is
> > this "lag" a bad thing?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira 
> >>> wrote:
> >
> >> +1 (binding)
> >>
> >> Thanks for responding to all my original concerns in the discussion
>  thread.
> >>
> >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
>  eric.wasser...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I would like to begin voting on KIP-58 - Make Log Compaction Point
> >>> Configurable
> >>>
> >>> KIP-58 is here:  <
> >>>
> >>>
> >>
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> 
> >>>
> >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> >>> is here: 
> >>>
> >>> The original pull request is here: <
> >>> https://github.com/apache/kafka/pull/1168>
> >>> (this includes configurations for size and message count lags that
> >>> will
> >> be
> >>> removed per discussion of KIP-58).
> >>>
> >>> The vote will run for 72 hours.
> >>>
> >>
> 
> 
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Ewen
> >>
>
>


-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Ben Stopford]

+1 (non-binding)

> On 25 May 2016, at 14:07, Ismael Juma  wrote:
> 
> +1 (binding)
> 
> I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside, I
> did notice that the topic level config for `log.segment.delete.delay.ms`
> (mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
> inconsistent.
> 
> Ismael
> 
> On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava 
> wrote:
> 
>> +1 (binding)
>> 
>> Agreed that the log.cleaner.compaction.delay.ms is probably a better name,
>> and consistent with log.segment.delete.delay.ms. Checked configs for other
>> suffixes that seemed reasonable and despite only appearing in that one
>> broker config, it seems the best match.
>> 
>> -Ewen
>> 
>> On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
>> 
>>> I'm +1 on the concept.
>>> 
>>> As with others I think the core challenge is to express this in an
>>> intuitive way, and carry the same terminology across the docs, the
>> configs,
>>> and docstrings for the configs. Pictures would help.
>>> 
>>> -Jay
>>> 
>>> On Tue, May 24, 2016 at 6:54 PM, James Cheng 
>> wrote:
>>> 
 I'm not sure what are the rules for who is allowed to vote, but I'm:
 
 +1 (non-binding) on the proposal
 
 I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
 confusing.
 
 I like Becket's "log.cleaner.compaction.delay.ms", or something
>> similar.
 
 The KIP describes it as the portion of the topic "that will remain
 uncompacted", so if you're open to alternate names:
 
 "log.cleaner.uncompacted.range.ms"
 "log.cleaner.uncompacted.head.ms" (Except that I always get "log tail"
 and "log head" mixed up...)
 "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
>> the
 word "retention" in non-time-based topics?)
 
 I just thought of something: what happens to the value of "
 log.cleaner.delete.retention.ms"? Does it still have the same meaning
>> as
 before? Does the timer start when log compaction happens (as it
>> currently
 does), so in reality, tombstones will only be removed from the log some
 time after (log.cleaner.min.compaction.lag.ms +
 log.cleaner.delete.retention.ms)?
 
 -James
 
> On May 24, 2016, at 5:46 PM, Becket Qin 
>> wrote:
> 
> +1 (non-binding) on the proposal. Just a minor suggestion.
> 
> I am wondering should we change the config name to "
> log.cleaner.compaction.delay.ms"? The first glance at the
>>> configuration
> name is a little confusing. I was thinking do we have a "max" lag?
>> And
>>> is
> this "lag" a bad thing?
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> 
> On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira 
>>> wrote:
> 
>> +1 (binding)
>> 
>> Thanks for responding to all my original concerns in the discussion
 thread.
>> 
>> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
 eric.wasser...@gmail.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> I would like to begin voting on KIP-58 - Make Log Compaction Point
>>> Configurable
>>> 
>>> KIP-58 is here:  <
>>> 
>>> 
>> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
 
>>> 
>>> The Jira ticket KAFKA-1981 Make log compaction point configurable
>>> is here: 
>>> 
>>> The original pull request is here: <
>>> https://github.com/apache/kafka/pull/1168>
>>> (this includes configurations for size and message count lags that
>>> will
>> be
>>> removed per discussion of KIP-58).
>>> 
>>> The vote will run for 72 hours.
>>> 
>> 
 
 
>>> 
>> 
>> 
>> 
>> --
>> Thanks,
>> Ewen
>> 

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Ismael Juma]

+1 (binding)

I also think `log.cleaner.compaction.delay.ms` is clearer. As an aside, I
did notice that the topic level config for `log.segment.delete.delay.ms`
(mentioned by Ewen) is `file.delete.delay.ms`, which seems a bit
inconsistent.

Ismael

On Wed, May 25, 2016 at 4:43 AM, Ewen Cheslack-Postava 
wrote:

> +1 (binding)
>
> Agreed that the log.cleaner.compaction.delay.ms is probably a better name,
> and consistent with log.segment.delete.delay.ms. Checked configs for other
> suffixes that seemed reasonable and despite only appearing in that one
> broker config, it seems the best match.
>
> -Ewen
>
> On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
>
> > I'm +1 on the concept.
> >
> > As with others I think the core challenge is to express this in an
> > intuitive way, and carry the same terminology across the docs, the
> configs,
> > and docstrings for the configs. Pictures would help.
> >
> > -Jay
> >
> > On Tue, May 24, 2016 at 6:54 PM, James Cheng 
> wrote:
> >
> > > I'm not sure what are the rules for who is allowed to vote, but I'm:
> > >
> > > +1 (non-binding) on the proposal
> > >
> > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
> > > confusing.
> > >
> > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> similar.
> > >
> > > The KIP describes it as the portion of the topic "that will remain
> > > uncompacted", so if you're open to alternate names:
> > >
> > > "log.cleaner.uncompacted.range.ms"
> > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log tail"
> > > and "log head" mixed up...)
> > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> the
> > > word "retention" in non-time-based topics?)
> > >
> > > I just thought of something: what happens to the value of "
> > > log.cleaner.delete.retention.ms"? Does it still have the same meaning
> as
> > > before? Does the timer start when log compaction happens (as it
> currently
> > > does), so in reality, tombstones will only be removed from the log some
> > > time after (log.cleaner.min.compaction.lag.ms +
> > > log.cleaner.delete.retention.ms)?
> > >
> > > -James
> > >
> > > > On May 24, 2016, at 5:46 PM, Becket Qin 
> wrote:
> > > >
> > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > >
> > > > I am wondering should we change the config name to "
> > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > configuration
> > > > name is a little confusing. I was thinking do we have a "max" lag?
> And
> > is
> > > > this "lag" a bad thing?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira 
> > wrote:
> > > >
> > > >> +1 (binding)
> > > >>
> > > >> Thanks for responding to all my original concerns in the discussion
> > > thread.
> > > >>
> > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > eric.wasser...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> I would like to begin voting on KIP-58 - Make Log Compaction Point
> > > >>> Configurable
> > > >>>
> > > >>> KIP-58 is here:  <
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > 
> > > >>>
> > > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > > >>> is here: 
> > > >>>
> > > >>> The original pull request is here: <
> > > >>> https://github.com/apache/kafka/pull/1168>
> > > >>> (this includes configurations for size and message count lags that
> > will
> > > >> be
> > > >>> removed per discussion of KIP-58).
> > > >>>
> > > >>> The vote will run for 72 hours.
> > > >>>
> > > >>
> > >
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Manikumar Reddy]

+1 (non binding)

On Wed, May 25, 2016 at 4:03 PM, Tom Crayford  wrote:

> +1 (non binding)
>
> Agree on log.cleaner.compaction.delay.ms being the better name.
>
> I think this setting is going to be extremely hard to tune for users, and
> worry about adding yet more configuration - Kafka already has a huge number
> of tunables though, so we're in well trod ground with "just add more
> tuning". I can't however, come up with any better mechanism (that doesn't
> require tuning at all) without gross interactions with consumer offset
> storage, so I remain a +1 here.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 25 May 2016, Ewen Cheslack-Postava  > wrote:
>
> > +1 (binding)
> >
> > Agreed that the log.cleaner.compaction.delay.ms is probably a better
> name,
> > and consistent with log.segment.delete.delay.ms. Checked configs for
> other
> > suffixes that seemed reasonable and despite only appearing in that one
> > broker config, it seems the best match.
> >
> > -Ewen
> >
> > On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:
> >
> > > I'm +1 on the concept.
> > >
> > > As with others I think the core challenge is to express this in an
> > > intuitive way, and carry the same terminology across the docs, the
> > configs,
> > > and docstrings for the configs. Pictures would help.
> > >
> > > -Jay
> > >
> > > On Tue, May 24, 2016 at 6:54 PM, James Cheng 
> > wrote:
> > >
> > > > I'm not sure what are the rules for who is allowed to vote, but I'm:
> > > >
> > > > +1 (non-binding) on the proposal
> > > >
> > > > I agree that the "log.cleaner.min.compaction.lag.ms" name is a
> little
> > > > confusing.
> > > >
> > > > I like Becket's "log.cleaner.compaction.delay.ms", or something
> > similar.
> > > >
> > > > The KIP describes it as the portion of the topic "that will remain
> > > > uncompacted", so if you're open to alternate names:
> > > >
> > > > "log.cleaner.uncompacted.range.ms"
> > > > "log.cleaner.uncompacted.head.ms" (Except that I always get "log
> tail"
> > > > and "log head" mixed up...)
> > > > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have
> > the
> > > > word "retention" in non-time-based topics?)
> > > >
> > > > I just thought of something: what happens to the value of "
> > > > log.cleaner.delete.retention.ms"? Does it still have the same
> meaning
> > as
> > > > before? Does the timer start when log compaction happens (as it
> > currently
> > > > does), so in reality, tombstones will only be removed from the log
> some
> > > > time after (log.cleaner.min.compaction.lag.ms +
> > > > log.cleaner.delete.retention.ms)?
> > > >
> > > > -James
> > > >
> > > > > On May 24, 2016, at 5:46 PM, Becket Qin 
> > wrote:
> > > > >
> > > > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > > > >
> > > > > I am wondering should we change the config name to "
> > > > > log.cleaner.compaction.delay.ms"? The first glance at the
> > > configuration
> > > > > name is a little confusing. I was thinking do we have a "max" lag?
> > And
> > > is
> > > > > this "lag" a bad thing?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira 
> > > wrote:
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> Thanks for responding to all my original concerns in the
> discussion
> > > > thread.
> > > > >>
> > > > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > > > eric.wasser...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> I would like to begin voting on KIP-58 - Make Log Compaction
> Point
> > > > >>> Configurable
> > > > >>>
> > > > >>> KIP-58 is here:  <
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > > > 
> > > > >>>
> > > > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > > > >>> is here: 
> > > > >>>
> > > > >>> The original pull request is here: <
> > > > >>> https://github.com/apache/kafka/pull/1168>
> > > > >>> (this includes configurations for size and message count lags
> that
> > > will
> > > > >> be
> > > > >>> removed per discussion of KIP-58).
> > > > >>>
> > > > >>> The vote will run for 72 hours.
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Ewen Cheslack-Postava]

+1 (binding)

Agreed that the log.cleaner.compaction.delay.ms is probably a better name,
and consistent with log.segment.delete.delay.ms. Checked configs for other
suffixes that seemed reasonable and despite only appearing in that one
broker config, it seems the best match.

-Ewen

On Tue, May 24, 2016 at 8:16 PM, Jay Kreps  wrote:

> I'm +1 on the concept.
>
> As with others I think the core challenge is to express this in an
> intuitive way, and carry the same terminology across the docs, the configs,
> and docstrings for the configs. Pictures would help.
>
> -Jay
>
> On Tue, May 24, 2016 at 6:54 PM, James Cheng  wrote:
>
> > I'm not sure what are the rules for who is allowed to vote, but I'm:
> >
> > +1 (non-binding) on the proposal
> >
> > I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
> > confusing.
> >
> > I like Becket's "log.cleaner.compaction.delay.ms", or something similar.
> >
> > The KIP describes it as the portion of the topic "that will remain
> > uncompacted", so if you're open to alternate names:
> >
> > "log.cleaner.uncompacted.range.ms"
> > "log.cleaner.uncompacted.head.ms" (Except that I always get "log tail"
> > and "log head" mixed up...)
> > "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have the
> > word "retention" in non-time-based topics?)
> >
> > I just thought of something: what happens to the value of "
> > log.cleaner.delete.retention.ms"? Does it still have the same meaning as
> > before? Does the timer start when log compaction happens (as it currently
> > does), so in reality, tombstones will only be removed from the log some
> > time after (log.cleaner.min.compaction.lag.ms +
> > log.cleaner.delete.retention.ms)?
> >
> > -James
> >
> > > On May 24, 2016, at 5:46 PM, Becket Qin  wrote:
> > >
> > > +1 (non-binding) on the proposal. Just a minor suggestion.
> > >
> > > I am wondering should we change the config name to "
> > > log.cleaner.compaction.delay.ms"? The first glance at the
> configuration
> > > name is a little confusing. I was thinking do we have a "max" lag? And
> is
> > > this "lag" a bad thing?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira 
> wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks for responding to all my original concerns in the discussion
> > thread.
> > >>
> > >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> > eric.wasser...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I would like to begin voting on KIP-58 - Make Log Compaction Point
> > >>> Configurable
> > >>>
> > >>> KIP-58 is here:  <
> > >>>
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > 
> > >>>
> > >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> > >>> is here: 
> > >>>
> > >>> The original pull request is here: <
> > >>> https://github.com/apache/kafka/pull/1168>
> > >>> (this includes configurations for size and message count lags that
> will
> > >> be
> > >>> removed per discussion of KIP-58).
> > >>>
> > >>> The vote will run for 72 hours.
> > >>>
> > >>
> >
> >
>



-- 
Thanks,
Ewen

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Jay Kreps]

I'm +1 on the concept.

As with others I think the core challenge is to express this in an
intuitive way, and carry the same terminology across the docs, the configs,
and docstrings for the configs. Pictures would help.

-Jay

On Tue, May 24, 2016 at 6:54 PM, James Cheng  wrote:

> I'm not sure what are the rules for who is allowed to vote, but I'm:
>
> +1 (non-binding) on the proposal
>
> I agree that the "log.cleaner.min.compaction.lag.ms" name is a little
> confusing.
>
> I like Becket's "log.cleaner.compaction.delay.ms", or something similar.
>
> The KIP describes it as the portion of the topic "that will remain
> uncompacted", so if you're open to alternate names:
>
> "log.cleaner.uncompacted.range.ms"
> "log.cleaner.uncompacted.head.ms" (Except that I always get "log tail"
> and "log head" mixed up...)
> "log.cleaner.uncompacted.retention.ms" (Will it be confusing to have the
> word "retention" in non-time-based topics?)
>
> I just thought of something: what happens to the value of "
> log.cleaner.delete.retention.ms"? Does it still have the same meaning as
> before? Does the timer start when log compaction happens (as it currently
> does), so in reality, tombstones will only be removed from the log some
> time after (log.cleaner.min.compaction.lag.ms +
> log.cleaner.delete.retention.ms)?
>
> -James
>
> > On May 24, 2016, at 5:46 PM, Becket Qin  wrote:
> >
> > +1 (non-binding) on the proposal. Just a minor suggestion.
> >
> > I am wondering should we change the config name to "
> > log.cleaner.compaction.delay.ms"? The first glance at the configuration
> > name is a little confusing. I was thinking do we have a "max" lag? And is
> > this "lag" a bad thing?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira  wrote:
> >
> >> +1 (binding)
> >>
> >> Thanks for responding to all my original concerns in the discussion
> thread.
> >>
> >> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman <
> eric.wasser...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I would like to begin voting on KIP-58 - Make Log Compaction Point
> >>> Configurable
> >>>
> >>> KIP-58 is here:  <
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> 
> >>>
> >>> The Jira ticket KAFKA-1981 Make log compaction point configurable
> >>> is here: 
> >>>
> >>> The original pull request is here: <
> >>> https://github.com/apache/kafka/pull/1168>
> >>> (this includes configurations for size and message count lags that will
> >> be
> >>> removed per discussion of KIP-58).
> >>>
> >>> The vote will run for 72 hours.
> >>>
> >>
>
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[James Cheng]

I'm not sure what are the rules for who is allowed to vote, but I'm:

+1 (non-binding) on the proposal

I agree that the "log.cleaner.min.compaction.lag.ms" name is a little confusing.

I like Becket's "log.cleaner.compaction.delay.ms", or something similar.

The KIP describes it as the portion of the topic "that will remain 
uncompacted", so if you're open to alternate names:

"log.cleaner.uncompacted.range.ms"
"log.cleaner.uncompacted.head.ms" (Except that I always get "log tail" and "log 
head" mixed up...)
"log.cleaner.uncompacted.retention.ms" (Will it be confusing to have the word 
"retention" in non-time-based topics?)

I just thought of something: what happens to the value of 
"log.cleaner.delete.retention.ms"? Does it still have the same meaning as 
before? Does the timer start when log compaction happens (as it currently 
does), so in reality, tombstones will only be removed from the log some time 
after (log.cleaner.min.compaction.lag.ms + log.cleaner.delete.retention.ms)?

-James

> On May 24, 2016, at 5:46 PM, Becket Qin  wrote:
> 
> +1 (non-binding) on the proposal. Just a minor suggestion.
> 
> I am wondering should we change the config name to "
> log.cleaner.compaction.delay.ms"? The first glance at the configuration
> name is a little confusing. I was thinking do we have a "max" lag? And is
> this "lag" a bad thing?
> 
> Thanks,
> 
> Jiangjie (Becket) Qin
> 
> 
> On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira  wrote:
> 
>> +1 (binding)
>> 
>> Thanks for responding to all my original concerns in the discussion thread.
>> 
>> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman 
>> wrote:
>> 
>>> Hi,
>>> 
>>> I would like to begin voting on KIP-58 - Make Log Compaction Point
>>> Configurable
>>> 
>>> KIP-58 is here:  <
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
 
>>> 
>>> The Jira ticket KAFKA-1981 Make log compaction point configurable
>>> is here: 
>>> 
>>> The original pull request is here: <
>>> https://github.com/apache/kafka/pull/1168>
>>> (this includes configurations for size and message count lags that will
>> be
>>> removed per discussion of KIP-58).
>>> 
>>> The vote will run for 72 hours.
>>> 
>> 

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Becket Qin]

+1 (non-binding) on the proposal. Just a minor suggestion.

I am wondering should we change the config name to "
log.cleaner.compaction.delay.ms"? The first glance at the configuration
name is a little confusing. I was thinking do we have a "max" lag? And is
this "lag" a bad thing?

Thanks,

Jiangjie (Becket) Qin


On Tue, May 24, 2016 at 4:21 PM, Gwen Shapira  wrote:

> +1 (binding)
>
> Thanks for responding to all my original concerns in the discussion thread.
>
> On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman 
> wrote:
>
> > Hi,
> >
> > I would like to begin voting on KIP-58 - Make Log Compaction Point
> > Configurable
> >
> > KIP-58 is here:  <
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> > >
> >
> > The Jira ticket KAFKA-1981 Make log compaction point configurable
> > is here: 
> >
> > The original pull request is here: <
> > https://github.com/apache/kafka/pull/1168>
> > (this includes configurations for size and message count lags that will
> be
> > removed per discussion of KIP-58).
> >
> > The vote will run for 72 hours.
> >
>

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

[Gwen Shapira]

+1 (binding)

Thanks for responding to all my original concerns in the discussion thread.

On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman 
wrote:

> Hi,
>
> I would like to begin voting on KIP-58 - Make Log Compaction Point
> Configurable
>
> KIP-58 is here:  <
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable
> >
>
> The Jira ticket KAFKA-1981 Make log compaction point configurable
> is here: 
>
> The original pull request is here: <
> https://github.com/apache/kafka/pull/1168>
> (this includes configurations for size and message count lags that will be
> removed per discussion of KIP-58).
>
> The vote will run for 72 hours.
>