Hi,
This is Xiongqi (Wesley) Wu from Linkedin Kafka dev team.
I want to request permission to Create KIP for the log compaction project we
are currently working on here at Linkedin.
My wiki id is : xiongqiwu
--Xiongqi (Wesley) Wu
Hi Kafka,
Just updated the confluence page to include the link to this KIP.
Any comment will be appreciated:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy
Thank you.
Xiongqi (Wesley) Wu
On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu wrote:
>
will also enforce "max.compaction.lag.ms" is not
less than "min.compaction.lag.ms".
https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy
On Tue, Aug 14, 2018 at 5:01 PM, xiongqi wu wrote:
>
> Per discussion with Dong,
think about a way to trigger compaction as well
> through an API call, which would need to be flagged somewhere (ZK admin/
> space?) but we're struggling to think how that would be coordinated across
> brokers and partitions. Have you given any thought to that?
>
>
>
>
>
ay/KAFKA/KIP-
> 58+-+Make+Log+Compaction+Point+Configurable),
> just that KIP-58 is a "upper-bound" on what messages can be compacted, and
> this is for a "lower-bound" on what messages NEED to be compacted?
>
>
> Guozhang
>
> On Mon, Aug 13, 2018 at 2:3
requirement in some use cases (e.g., GDPR)"
>
>
> Is there any guarantee that after this KIP the GDPR problem is solved or do
> we need to do something else as well, e.g., more KIPs?
>
>
> Thanks
>
> Eno
>
>
>
> On Thu, Aug 9, 2018 at 4:18 PM, xiongqi
s already been covered in KIP-280? Could you check out
> that one and see if it is the case.
>
>
> Guozhang
>
>
> On Mon, Aug 13, 2018 at 1:23 PM, xiongqi wu wrote:
>
> > Hi Kafka,
> >
> > Just updated the confluence page to include the link to this KIP.
>
) or time-based log retention
> to meet the use-case requirement. If no, do you know what is the GDPR's
> requirement on time-to-deletion after user explicitly requests the deletion
> (e.g. 1 hour, 1 day, 7 day)?
>
> Thanks,
> Dong
>
>
> On Mon, Aug 13, 2018 at 3:44
would prefer solution A and rely on
record timestamp.
Two questions:
Do we have use case 3? Is it nice to have or must have?
If we have use case 3 and want to go with solution A, should we introduce
a new configuration to enforce deletion by timestamp?
On Tue, Aug 14, 2018 at 1:52 PM, xiongqi wu
that mechanism would be it started to feel
> like
> > it would be a complicated implementation so we've put it aside for now.
> But
> > maybe we just haven't seen the clean way yet.
> >
> >
> >
> > On Thu, Aug 16, 2018 at 11:22 AM xiongqi wu wrote:
> >
&
ou to avoid double effort
> on this. I am in confluent community slack during the work time. My name is
> Xiaohe Dong. :)
>
> Rgds
> Xiaohe Dong
>
>
>
> On 2018/08/16 01:22:22, xiongqi wu wrote:
> > Brett,
> >
> > Thank you for your comments.
> > I w
sure that the segment will be compacted again within the given time after
> the deletion is requested, right?
>
> Thanks,
> Dong
>
> On Thu, Aug 16, 2018 at 10:27 AM, xiongqi wu wrote:
>
> > Hi Xiaohe,
> >
> > Quick note:
> > 1) Use minimum of segment.ms a
ut I didn't look at code or test for that.
>
> On Fri, Aug 17, 2018 at 10:57 AM xiongqi wu wrote:
>
> > 1, Owner of data (in this sense, kafka is the not the owner of data)
> > should keep track of lifecycle of the data in some external storage/DB.
> > The owner
Hi Kafka,
This KIP tries to address GDPR concern to fulfill deletion request on time
through time-based log compaction on a compaction enabled topic:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Time-based+log+compaction+policy
Any feedback will be appreciated.
Xiongqi
Hi All,
Do you have any additional comments on this KIP?
On Thu, Aug 16, 2018 at 9:17 PM, xiongqi wu wrote:
> on 2)
> The offsetmap is built starting from dirty segment.
> The compaction starts from the beginning of the log partition. That's how
> it ensure the deletion of to
Colin,
I will keep the title for now, since all the previous discussions and links
are tied to this title.
I can change the title at the end or add a clarification note in the doc.
Xiongqi (Wesley) Wu
On Tue, Sep 4, 2018 at 5:47 PM xiongqi wu wrote:
> Colin,
> Thank you for comments.
etermined if we use "0". On the other
> > hand, we can already set "min.cleanable.dirty.ratio" to achieve the
> same
> > goal. So here we choose "0" as "disabled".
>
> I would prefer -1 to be the invalid setting. Treating 0 dif
wrote:
> On Tue, Sep 4, 2018, at 17:47, xiongqi wu wrote:
> > Colin,
> > Thank you for comments.
> > see my inline reply below.
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Tue, Sep 4, 2018 at 5:24 PM Colin McCabe wrote:
> >
> > >
Let's VOTE for this KIP.
KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
%3A+Time-based+log+compaction+policy
Implementation:
https://github.com/apache/kafka/pull/5611
Xiongqi (Wesley) Wu
nt. We should make 0 = disable, to be consistent with
> the
> > > other settings.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > > -- Note that an alternative configuration is to use -1 as
> "disabled"
> > > a
:25, xiongqi wu wrote:
> > Thanks for comments.
> >
> > Today, when creating topic, client only does simple local validation
> > and doesn't check against broker's configurations.
> >
> > We cannot just let users to create a configuration in zookeeper and
> > dish
This KIP enables broker to remove orphan partitions automatically.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-370%3A+Remove+Orphan+Partitions
Xiongqi (Wesley) Wu
And how is it relating to segment.ms
> ?
> Is it that you're proposing to have 0 mean "use segment.ms instead?" as a
> kind of third option?
>
>
>
> On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu wrote:
>
> > To make it clear,
> > I don't against using -1 as dis
set to 0, then we're not being consistent
> > by using 0 for this new setting? I throw out -1 for consideration
> > again :)
> >
> > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu wrote:
> >
> >> Thanks. I will document after PR is merged.
> >>
> >&
that too
> if its not set and null for others means disabled!)
>
>
> On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu wrote:
>
> > If we use 0 to indicate immediate compaction, the compaction lag is
> > determined by segment.ms in worst case. If segment.ms is 24 hours,
> >
falls
> > now.
> >
> > Don't want to spend more time on this though, It's well into
> bikeshedding
> > territory now. :)
> >
> >
> >
> > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu wrote:
> >
> > > I want to honor the minimum value
xiongqi wu wrote:
>
> This KIP enables broker to remove orphan partitions automatically.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-370%3A+Remove+Orphan+Partitions
>
>
> Xiongqi (Wesley) Wu
>
> Mayuresh
>
> On Mon, Sep 3, 2018 at 7:28 PM Brett Rann
> wrote:
>
> > Might also be worth moving to a vote thread? Discussion seems to have
> gone
> > as far as it can.
> >
> > > On 4 Sep 2018, at 12:08, xiongqi wu wrote:
> > >
> > &
these metrics.
8) The Performance Impact suggests user to use the existing metrics to
> monitor the performance impact of this KIP. It i useful to list mean of
> each jmx metrics that we want user to monitor, and possibly explain how to
> interpret the value of these me
Hi Dong,
I have updated the KIP to address your comments.
One correction to previous Email:
after offline discussion with Dong, we decide to use MAX_LONG as default
value for max.compaction.lag.ms.
Xiongqi (Wesley) Wu
On Mon, Oct 29, 2018 at 12:15 PM xiongqi wu wrote:
> Hi Dong,
>
&
confluence/display/KAFKA/KIP-237%3A+More+Controller+Health+Metrics
> .
>
> Thanks,
> Dong
>
> On Thu, Sep 20, 2018 at 5:40 PM xiongqi wu wrote:
>
> > Colin,
> >
> > Thanks for the comment.
> > 1)
> > auto.orphan.partition.removal.delay.ms r
Thanks Dong.
I have updated the KIP.
Instead of using a configure to specify the timeout, I switch it to use
internal timer. User doesn't need a new configuration to use this feature.
Xiongqi (Wesley) Wu
On Mon, Oct 29, 2018 at 4:40 PM xiongqi wu wrote:
> Dong,
>
> Thanks for the
bump
Xiongqi (Wesley) Wu
On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu wrote:
>
> Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
> Can I have more feedback or VOTE on this KIP?
>
>
> Xiongqi (Wesley) Wu
>
>
> On Wed, Sep 19, 2018 at 10:52 AM xiongq
ion.lag.ms
> can be overridden on per-topic basis.
>
>
>
> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu wrote:
>
> > Thanks Joel.
> > Tracking the delay at second granularity makes sense
> > I have updated KIP.
> >
> > Xiongqi (Wesley) Wu
> >
> &
t;
> Joel
>
> On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu wrote:
>
> > bump
> > Xiongqi (Wesley) Wu
> >
> >
> > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu wrote:
> >
> > >
> > > Thanks Eno, Brett, Dong, Guozhang, Colin,
Lincong,
Thanks for the KIP.
I have a question about the lifecycle of request and response.
With the current (requestAdapter, responseAdapter) implementation,
the observer can potentially extend the lifecycle of request and response
through adapter.
Anyone can implement own observer, and some
KIP to address your comments.
> > One correction to previous Email:
> > after offline discussion with Dong, we decide to use MAX_LONG as default
> > value for max.compaction.lag.ms.
> >
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Mon, Oct 29,
Hi all,
Can I have one more vote on this KIP?
Any comment is appreciated.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
Xiongqi (Wesley) Wu
On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu wrote:
> Thanks Dong.
> I have updated the KIP.
>
ches: what's documented in the KIP, and what Xiaohe Dong was working
> on
> here:
>
> https://github.com/dongxiaohe/kafka/tree/dongxiaohe/log-cleaner-compaction-max-lifetime-2.0
>
> If you have code working already Xiongqi Wu could you share a PR? I'd be
> happy
> to start testing.
since the broker started up?
>
> Is there any logic to remove the partition folders on disk? I can only
> find references to removing older log segments, but not the folder, in the
> KIP.
>
> best,
> Colin
>
> On Wed, Sep 19, 2018, at 10:53, xiongqi wu wrote:
> >
Any other votes or comments?
Xiongqi (Wesley) Wu
On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu wrote:
> Yes, more votes and code review.
>
> Xiongqi (Wesley) Wu
>
>
> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> wrote:
>
>> +1 (non binding) from on 0 then, and on
Any comments?
Xiongqi (Wesley) Wu
On Mon, Sep 10, 2018 at 3:04 PM xiongqi wu wrote:
> Here is the implementation for the KIP 370.
>
>
> https://github.com/xiowu0/kafka/commit/f1bd3085639f41a7af02567550a8e3018cfac3e9
>
>
> The purpose is to do one time cleanup (aft
Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
Can I have more feedback or VOTE on this KIP?
Xiongqi (Wesley) Wu
On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu wrote:
> Any other votes or comments?
>
> Xiongqi (Wesley) Wu
>
>
> On Tue, Sep 11, 2018 at 1
in wrote:
>
> > Thanks for the update. +1 (binding)
> >
> > On Wed, Dec 5, 2018 at 8:19 AM Colin McCabe wrote:
> >
> > > Thanks, Xiongqi Wu. +1 (binding)
> > >
> > > regards,
> > > Colin
> > >
> > >
> > &
nce we can get what we need with the combination of
> this and log.message.timestamp.difference.max.ms.
>
> best,
> Colin
>
>
> On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > Thanks for binding and non-binding votes.
> > Can I get one more binding vote?
> >
> > Thanks in advanc
can't go into details, but
> figuring out how to achieve this effect gave me quite a headache. :)
>
> On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu wrote:
>
> > Hi all,
> >
> > Can I have one more vote on this KIP?
> > Any comment is appreciated.
> >
>
, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > Thank you for comments. I will use '0' for now.
> > >
> > > If we create topics through admin client in the future, we might
> perform
> > > some useful checks. (but the assumption is all brokers in the sa
xiongqi wu created KAFKA-7321:
-
Summary: ensure timely processing of deletion requests in Kafka
topic (Time-based log compaction)
Key: KAFKA-7321
URL: https://issues.apache.org/jira/browse/KAFKA-7321
xiongqi wu created KAFKA-7322:
-
Summary: race between compaction thread and retention thread when
changing topic cleanup policy
Key: KAFKA-7322
URL: https://issues.apache.org/jira/browse/KAFKA-7322
xiongqi wu created KAFKA-7362:
-
Summary: enable kafka broker to remove orphan partitions
automatically
Key: KAFKA-7362
URL: https://issues.apache.org/jira/browse/KAFKA-7362
Project: Kafka
xiongqi wu created KAFKA-7501:
-
Summary: double deallocation of producer batch upon expiration of
inflight requests and error response
Key: KAFKA-7501
URL: https://issues.apache.org/jira/browse/KAFKA-7501
xiongqi wu created KAFKA-7650:
-
Summary: make "auto.create.topics.enable" dynamically
configurable.
Key: KAFKA-7650
URL: https://issues.apache.org/jira/browse/KAFKA-7650
Project: Kafka
xiongqi wu created KAFKA-8249:
-
Summary: partition reassignment may never finish if topic deletion
completes first
Key: KAFKA-8249
URL: https://issues.apache.org/jira/browse/KAFKA-8249
Project: Kafka
xiongqi wu created KAFKA-8527:
-
Summary: add dynamic maintenance broker config
Key: KAFKA-8527
URL: https://issues.apache.org/jira/browse/KAFKA-8527
Project: Kafka
Issue Type: Improvement
xiongqi wu created KAFKA-10806:
--
Summary: throwable from user callback on
completeFutureAndFireCallbacks can lead to unhandled exceptions
Key: KAFKA-10806
URL: https://issues.apache.org/jira/browse/KAFKA-10806
55 matches
Mail list logo
