Sure Guozhang! Working on it and will post it when it is ready...
Thanks,
Senthil
-Original Message-
From: Guozhang Wang
Sent: Sunday, December 8, 2019 6:47 PM
To: dev
Subject: [EXTERNAL] Re: [VOTE] KIP-280: Enhanced log compaction
Thanks for the updated KIP, recasting my vote +1
> Thanks Matthias!
> >
> > Received 2 +1 binding... looking for one more +1 binding !
> >
> > Regards,
> > Senthil
> >
> > -Original Message-
> > From: Matthias J. Sax
> > Sent: Wednesday, November 6, 2019 12:10 AM
> > To: dev
RNAL] Re: [VOTE] KIP-280: Enhanced log compaction
Hi, Senthil,
Thanks for the updated KIP. +1 from me.
Could you also add in the recommendation section what the users should do when
consuming the compacted topic with the new strategies (e.g. have to be more
careful with what records to keep)?
Senthilnathan Muthusamy
wrote:
> Jun,
>
> If the updated KIP looks good, can you please vote for it.
>
> Thanks,
> Senthil
>
> -Original Message-
> From: Jun Rao
> Sent: Thursday, November 7, 2019 4:33 PM
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log co
Jun,
If the updated KIP looks good, can you please vote for it.
Thanks,
Senthil
-Original Message-
From: Jun Rao
Sent: Thursday, November 7, 2019 4:33 PM
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi, Senthil,
Thanks for the KIP. Added a few more comments
inal Message-
> From: Matthias J. Sax
> Sent: Wednesday, November 6, 2019 12:10 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> +1 (binding)
>
> On 11/5/19 11:44 AM, Senthilnathan Muthusamy wrote:
> > Thanks Gouzhang and I
Thanks Matthias!
Received 2 +1 binding... looking for one more +1 binding !
Regards,
Senthil
-Original Message-
From: Matthias J. Sax
Sent: Wednesday, November 6, 2019 12:10 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
+1 (binding)
On 11/5/19 11
--
> From: Guozhang Wang
> Sent: Monday, November 4, 2019 11:01 AM
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> I only have one minor comment on the DISCUSS thread, otherwise I'm +1
> (binding).
>
> On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Mu
Thanks Gouzhang and I have made a note in the JIRA item to update the wiki.
Till now got 1 +1 binding... waiting for 2 more +1 binding... thnx!
Regards,
Senthil
-Original Message-
From: Guozhang Wang
Sent: Monday, November 4, 2019 11:01 AM
To: dev
Subject: Re: [VOTE] KIP-280
I only have one minor comment on the DISCUSS thread, otherwise I'm +1
(binding).
On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Muthusamy
wrote:
> Hi all,
>
> I would like to start the vote on the updated KIP-280: Enhanced log
> compaction. Thanks to Guozhang, Matthias & Tom for the valuable
Hi all,
I would like to start the vote on the updated KIP-280: Enhanced log compaction.
Thanks to Guozhang, Matthias & Tom for the valuable feedback on the discussion
thread...
KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
JIRA:
an absolute sequence.
-Original Message-
From: Luís Cabral
Sent: Thursday, August 16, 2018 10:32 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-280: Enhanced log compaction
Hi,
@Guozhang & @Jun:
There were some left over comments from when the topic was still fresh (you may
tion.
As for the suggestion, that is indeed a wonderful idea. I suggest that you
create a KIP and address the topic with the rest of the community.
Kind Regards,
Luis
From: Jason Gustafson
Sent: 14 August 2018 01:25
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hey Luis,
Th
ed to the KIP as the result of the
> discussions, where it was seen as a potential benefit for other clients who
> may prefer that.
>
> Cheers,
> Luís
>
> From: Jason Gustafson
> Sent: 10 August 2018 22:38
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
2018 22:38
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi Luis,
It's still not very clear to me why we need the header-based strategy. Can
you elaborate why having the timestamp-based approach alone is not
sufficient? The use case in the motivation just describes a "most r
> > > about each one), so that influences me to avoid writing about every
> > > > > micro-impact that may exist, and simply leave it inferred (as you
> > have
> > > > just
> > > > > done).
> > > > > But I also don’t feel strong
st message in a partition (which is possible now). We
> > need
> > > > to preserve the offset of the last message so that we don't reuse the
> > > > offset for a different message. One way to simply never remove the
> last
> > > > message. Another way (suggest
e want to be a bit careful
> with
> > > > removing the last message in a partition (which is possible now). We
> > need
> > > > to preserve the offset of the last message so that we don't reuse the
> > > > offset for a different message. One way t
an empty message
> > > batch.
> > >
> > > That is a good point, but isn’t the last message always kept
> regardless?
> > > In all of my tests with this approach, never have I seen it being
> > removed.
> > > This is not because I made it so while chan
Thanks for the KIP, Luis. A brief comment below.
On Wed, Jul 4, 2018 at 11:11 AM Luís Cabral
wrote:
> As a reader, I tend to prefer brief documentation on new features (they
> tend to be too many for me to find the willpower to read a 200-page essay
> about each one), so that influences me to
e it so while changing the code, it was simply
> > like this before, which made me smile!
> > Given these results, I just *assumed* (oops) that these scenarios
> > represented the reality, so the compaction would only affected the
> “tail”,
> > while the “head” remained untouch
nly affected the “tail”,
> while the “head” remained untouched. Now that you say its possible that the
> last message actually gets overwritten somehow, I guess a new bullet point
> will have to be added to the KIP for this (after I’ve found the time to
> review the portion of the code th
point will have to be
added to the KIP for this (after I’ve found the time to review the portion of
the code that enacts this behaviour).
Kind Regards,
Luís Cabral
From: Jun Rao
Sent: 03 July 2018 23:58
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi, Luis,
Thanks for th
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Sorry to join the discussion late. Can you you add to the motivation the
use cases for header-based compaction. This seems not very clear to me.
Thanks,
Jason
On Mon, Jul 2, 2018 at 11:00 AM, Guozhang Wang wrote:
> Hi Luis,
&g
Hi, Luis,
Thanks for the KIP. Overall, this seems a useful KIP. A few comments below.
1. I guess both new configurations will be at the topic level?
2. Since the log cleaner now needs to keep both the offset and another long
(say timestamp) in the de-dup map, it reduces the number of keys that
Sorry to join the discussion late. Can you you add to the motivation the
use cases for header-based compaction. This seems not very clear to me.
Thanks,
Jason
On Mon, Jul 2, 2018 at 11:00 AM, Guozhang Wang wrote:
> Hi Luis,
>
> I believe that compaction property is indeed overridable at
Hi Luis,
I believe that compaction property is indeed overridable at per-topic
level, as in
https://github.com/apache/kafka/blob/0cacbcf30e0a90ab9fad7bc310e5477cf959f1fd/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java#L116
And also documented in
Hi Guozhang,
You are right that it is not straightforward to add a dependent property
validation.
Though it is possible to re-design it to allow for this, that effort would be
better placed under its own KIP, if it really becomes useful for other
properties as well.
Given this, the
Hi Guozhang,
At the moment the KIP has your vote, Matthias' and Ted's.
Should I ask someone else to have a look?
Cheers,
Luis
On Monday, July 2, 2018, 12:16:48 PM GMT+2, Mickael Maison
wrote:
+1 (non binding). Thanks for the KIP!
On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang
+1 (non binding). Thanks for the KIP!
On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang wrote:
> Hi Luis,
>
> Regarding the minor suggest, I agree it would be better to make it as
> mandatory, but it might be a bit tricky because it is a conditional
> mandatory one depending on the other config's
Hi Luis,
Regarding the minor suggest, I agree it would be better to make it as
mandatory, but it might be a bit tricky because it is a conditional
mandatory one depending on the other config's value. Would like to see your
updated PR.
Regarding the KIP itself, both Matthias and myself can recast
Hi,
Thank you all for having a look!
The KIP is now updated with the result of these late discussions, though I did
take some liberty with this part:
- If the "compaction.strategy.header" configuration is not set (or is
blank), then the compaction strategy will fallback to "offset";
+1
On Thu, Jun 28, 2018 at 4:56 AM, Luís Cabral
wrote:
> Hi Ted,
> Can I also get your input on this?
>
> bq. +1 from my side for using `compaction.strategy` with values
> "offset","timestamp" and "header" and `compaction.strategy.header`
> -Matthias
>
> bq. +1 from me as well.
> -Guozhang
>
>
Hi Ted,
Can I also get your input on this?
bq. +1 from my side for using `compaction.strategy` with values
"offset","timestamp" and "header" and `compaction.strategy.header`
-Matthias
bq. +1 from me as well.
-Guozhang
Cheers,
Luis
new system
> >> supported compaction strategy in the future without potentially breaking
> >> something, as we basically assign the whole space of Strings (minus
> >> `offset`, `timestamp`) as valid configs to enable header based
> compaction.
> >>
> >>
either adding a config or going with
>> `header=`. Using `_timestamp_`, `_offset_`, and `` might be
>> good enough (even if this is the solution I like least)---for this case,
>> we should state explicitly, that the whole space of `_*_` is reserved
>> and users are not allowed to s
> and `_timestamp_` and throws an exception for all other `_*_` configs.
>
>
> -Matthias
>
>
> On 6/18/18 2:03 PM, Luís Cabral wrote:
> > I’m ok with that...
> >
> > Ted / Matthias?
> >
> >
> > From: Guozhang Wang
> > Sent: 18 June 2018 22:49
&g
owed to set those for header compaction. In fact, I
> would also add a check for the config that only allows for `_offset_`
> and `_timestamp_` and throws an exception for all other `_*_` configs.
>
>
> -Matthias
>
>
> On 6/18/18 2:03 PM, Luís Cabral wrote:
> > I’m ok wi
s an exception for all other `_*_` configs.
-Matthias
On 6/18/18 2:03 PM, Luís Cabral wrote:
> I’m ok with that...
>
> Ted / Matthias?
>
>
> From: Guozhang Wang
> Sent: 18 June 2018 22:49
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-280: Enhanced log com
I’m ok with that...
Ted / Matthias?
From: Guozhang Wang
Sent: 18 June 2018 22:49
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
How about make the preserved values to be "_offset_" and "_timestamp_"
then? Currently in the KIP they are
Luís
>
>
> From: Guozhang Wang
> Sent: 18 June 2018 19:35
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> Hello Luís,
>
> I agree that having an expression evaluation as a config value is not the
> best approach; if there are better
,
Luís
From: Guozhang Wang
Sent: 18 June 2018 19:35
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hello Luís,
I agree that having an expression evaluation as a config value is not the
best approach; if there are better ideas to allow users to specify the
header key
n't the selection in header
> >>>> have higher precedence over the configuration"? What selection do you
> >>>> mean? And want configuration?
> >>>>
> >>>>
> >>>> About the first point, I think this is actually a valid conce
> mean? And want configuration?
>>>>
>>>>
>>>> About the first point, I think this is actually a valid concern: To
>>>> address this issue, it seems that we would need to change the accepted
>>>> format of the config. Inst
ss this issue, it seems that we would need to change the accepted
>>>> format of the config. Instead of "offset", "timestamp", "",
>>>> we could replace the last one with "header=".
>>>>
>>>> WDYT?
>>>
irst point, I think this is actually a valid concern: To
> > >> address this issue, it seems that we would need to change the accepted
> > >> format of the config. Instead of "offset", "timestamp",
> "",
> > >> we could replace the last one
first point, I think this is actually a valid concern: To
> >> address this issue, it seems that we would need to change the accepted
> >> format of the config. Instead of "offset", "timestamp", "",
> >> we could replace the last one with &quo
:06 AM, Ted Yu wrote:
>>> If selection exists in header, the selection should override the config
>> value.
>>> Cheers
>>> Original message From: Luis Cabral
>> Date: 6/15/18 1:40 AM (GMT-08:00) To:
>> dev@kafka.apache.org Subject: Re:
rride the config
> value.
> > Cheers
> > ---- Original message From: Luis Cabral
> Date: 6/15/18 1:40 AM (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [VOTE] KIP-280: Enhanced log compaction
> > Hi,
> >
> > bq. Can the value be determined no
he config value.
> Cheers
> Original message From: Luis Cabral
> Date: 6/15/18 1:40 AM (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [VOTE] KIP-280: Enhanced log compaction
> Hi,
>
> bq. Can the value be determined now ? My thinking is that what if
If selection exists in header, the selection should override the config value.
Cheers
Original message From: Luis Cabral
Date: 6/15/18 1:40 AM (GMT-08:00) To:
dev@kafka.apache.org Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi,
bq. Can the value be determined now
Hi,
bq. Can the value be determined now ? My thinking is that what if there is a
third compaction strategy proposed in the future ? We should guard against user
unknowingly choosing the 'future' strategy.
The idea is that the header name to use is flexible, which protects current
clients that
bq. When this configuration is set to anything other than "*offset*" or "
*timestamp*", then the record headers are scanned for a key matching this
value.
Can the value be determined now ? My thinking is that what if there is a
third compaction strategy proposed in the future ? We should guard
+1 (binding)
-Matthias
On 6/9/18 12:39 AM, Luís Cabral wrote:
> Hi all,
>
> Any takers on having a look at this KIP and voting on it?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
>
> Cheers,
> Luis
>
signature.asc
Description: OpenPGP digital
Hi all,
Any takers on having a look at this KIP and voting on it?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
Cheers,
Luis
> After a thorough discussion, this KIP is now ready to go into a vote:
>
> KIP-280: Enhanced log compaction - Apache Kafka - Apache Software
> Foundation
>
>
> |
> |
> | |
> KIP-280: Enhanced log compaction - Apache Kafka - Apache Software Founda...
>
>
>
Hi all,
After a thorough discussion, this KIP is now ready to go into a vote:
KIP-280: Enhanced log compaction - Apache Kafka - Apache Software Foundation
|
|
| |
KIP-280: Enhanced log compaction - Apache Kafka - Apache Software Founda...
|
|
|
Kind Regards,
Luís Cabral
57 matches
Mail list logo
