Re: [DISCUSS] KIP-82 - Add Record Headers

[Jun Rao]

Hi, Michael,

We do have online KIP discussion meetings from time to time. How about we
discuss this KIP Wed (Oct 19) at 11:00am PST? I will send out an invite (we
typically do the meeting through Zoom and will post the video recording to
Kafka wiki).

Thanks,

Jun

On Wed, Oct 12, 2016 at 1:22 AM, Michael Pearce 
wrote:

> @Jay and Dana
>
> We have internally had a few discussions of how we may address this if we
> had a common apache kafka message wrapper for headers that can be used
> client side only to, and address the compaction issue.
> I have detailed this solution separately and linked from the main KIP-82
> wiki.
>
> Here’s a direct link –
> https://cwiki.apache.org/confluence/display/KAFKA/
> Headers+Value+Message+Wrapper
>
> We feel this solution though doesn’t manage to address all the use cases
> being mentioned still and also has some compatibility drawbacks e.g.
> backwards forwards compatibility especially on different language clients
> Also we still require with this solution, as still need to address
> compaction issue / tombstones, we need to make server side changes and as
> many message/record version changes.
>
> We believe the proposed solution in KIP-82 does address all these needs
> and is cleaner still, and more benefits.
> Please have a read, and comment. Also if you have any improvements on the
> proposed KIP-82 or an alternative solution/option your input is appreciated.
>
> @All
> As Joel has mentioned to get this moving along, and able to discuss more
> fluidly, it would be great if we can organize to meet up virtually online
> e.g. webex or something.
> I am aware, that the majority are based in America, myself is in the UK.
> @Kostya I assume you’re in Eastern Europe or Russia based on your email
> address (please correct this assumption), I hope the time difference isn’t
> too much that the below would suit you if you wish to join
>
> Can I propose next Wednesday 19th October at 18:30 BST , 10:30 PST, 20:30
> MSK we try meetup online?
>
> Would this date/time suit the majority?
> Also what is the preferred method? I can host via Adobe Connect style
> webex (which my company uses) but it isn’t the best IMHO, so more than
> happy to have someone suggest a better alternative.
>
> Best,
> Mike
>
>
>
>
> On 10/8/16, 7:26 AM, "Michael Pearce"  wrote:
>
> >> I agree with the critique of compaction not having a value. I think
> we should consider fixing that directly.
>
> > Agree that the compaction issue is troubling: compacted "null"
> deletes
> are incompatible w/ headers that must be packed into the message
> value. Are there any alternatives on compaction delete semantics that
> could address this? The KIP wiki discussion I think mostly assumes
> that compaction-delete is what it is and can't be changed/fixed.
>
> This KIP is about dealing with quite a few use cases and issues,
> please see both the KIP use cases detailed by myself and also the
> additional use cases wiki added by LinkedIn linked from the main KIP.
>
> The compaction is something that happily is addressed with headers,
> but most defiantly isn't the sole reason or use case for them, headers
> solves many issues and use cases. Thus their elegance and simplicity, and
> why they're so common in transport mechanisms and so succesfull, as stated
> like http, tcp, jms.
>
> 
> From: Dana Powers 
> Sent: Friday, October 7, 2016 11:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> > I agree with the critique of compaction not having a value. I think
> we should consider fixing that directly.
>
> Agree that the compaction issue is troubling: compacted "null" deletes
> are incompatible w/ headers that must be packed into the message
> value. Are there any alternatives on compaction delete semantics that
> could address this? The KIP wiki discussion I think mostly assumes
> that compaction-delete is what it is and can't be changed/fixed.
>
> -Dana
>
> On Fri, Oct 7, 2016 at 1:38 PM, Michael Pearce 
> wrote:
> >
> > Hi Jay,
> >
> > Thanks for the comments and feedback.
> >
> > I think its quite clear that if a problem keeps arising then it is
> clear that it needs resolving, and addressing properly.
> >
> > Fair enough at linkedIn, and historically for the very first use
> cases addressing this maybe not have been a big priority. But as Kafka is
> now Apache open source and being picked up by many including my company, it
> is clear and evident that this is a requirement and issue that needs to be
> now addressed to address these needs.
> >
> > The fact in almost every transport mechanism including networking
> layers in the enterprise ive worked in, there has always been headers i
> think clearly shows their need and success 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jungtaek Lim]

I guess no one doubts its power on REST server or even UI. I understand the
difficulty to add a module to project, but it's maximized when there is
less support expected hence maintenance issue is likely to rise, and IMHO
this seems to be not the case.

There're also pain points when project doesn't maintain features and
delegates to ecosystem. Based on some points (last commit date, pull
request open and closed, and contributor graph), kafka-manager seems to
have similar activity to kafka-rest, but it doesn't show any responses for
pull request supporting Kafka 0.10.0 even though numerous users leave
comments wish to support. What Kafka community can do for that project to
follow up? Nothing but just persuading by leaving comments hoping that will
be merged. (or finally come up another implementation) Kafka project keeps
agile but in point of whole ecosystem it can be less agile.

Yes decisions and roadmap of the project are driven by PMCs and I think
it's valid right. But we also imagine ASF projects as driven by community
aspect, though it's alike to ideal world. KIP makes innovation on adopting
new feature transparently, which makes many developers inspiring and
adopting it to their projects. Hopes that Kafka community continuously
drives the transparency model among the ASF projects, and beyond.

- Jungtaek Lim (HeartSaVioR)

2016년 10월 17일 (월) 오전 7:56, Jay Kreps 님이 작성:

Hey Nacho,

Yeah, I think it is definitely a call we have to make case by case. We have
some experience with this: originally we attempted to maintain things like
non-java clients, a hadoop connector, etc all in the main project. The
difficulty of that lead us to the current federated approach. In terms of
what is included now, yes, I agree you could potentially have even less
included.

-Jay

On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis 
wrote:

> What is the criteria for keeping things in and out of Kafka, what code
goes
> in or out and what is part of the architecture or not?
>
> The discussion of what goes into a project and what stays out is an always
> evolving question. Different projects treat this in different ways.
>
> Let me paint 2 extremes.  On one side, you have a single monolithic
project
> that brings everything in one tent.  On the other side you have the many
> modules approach.  From what I've learned, Kafka falls in the middle.
> Because of this, the question is bound to come up with respect to the
> criteria used to bring something into the fold.
>
> I'll be the first to point out that the distinction between modules,
> architecture, software, repositories, governance and community are blurry.
> Not to mention that many things are how they are for historical reasons.
>
> I, personally, can't understand why we would not have REST as part of the
> main Kafka project given that a lot of people use it and we include many
> things with the current distribution.  What many things you may ask?
Well,
> if we took the modular approach Kafka is a mixture of components, here's
> the first 4 that come to mind:
> 1. The Kafka protocol
> 2. The Kafka java libraries
> 3. The Kafka broker
> 4. The Kafka stream framework
> 5. Kafka Connect
> 6. MirrorMaker
>
> All of these could be separate products. You should be able to evolve each
> one independently.  Even if they have dependencies on each other, you
could
> potentially replace one part.
>
> The choice of keeping them all in a single repository, with a single
> distribution, under the same governance and community, brings a number of
> trade offs.  It's easy to keep things coherent for example.  There is less
> of a need to rely on inherent versioning and compatibility (which we end
up
> providing anyway because of the way people usually deploy kafka). We all
> focus our efforts on a single code base.
>
> The downside is that it's harder to remove modules that are old or unused.
> Modules that are only used by a small subset of the community will have an
> impact on the rest of the community.  It mixes incentives of what people
> want to work on and what holds them back.  We also need to decide what
> belongs in the blessed bundle and what doesnt.
>
> So, my question boils down to, what criteria is used for bringing stuff
in.
>
> If we have Streams and MirrorMaker and Connect in there, why not have
REST?
> Specially if there is more than one person/group willing to work on it?
> Alternatively, if REST is not included because it's not used by all, then
> why not remove Streams, Connect and MirrorMaker since they're definitely
> not used by all? I realize I say this even though at LinkedIn we have a
> REST setup of our own, just speaking from a community perspective.
>
> Nacho
>
>
> (I'm relatively new and I haven't read all of the mail archive, so I'm
sure
> this has been brought up before, but I decided to chime in anyway)
>
> On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps  wrote:
>
> > I think the questions 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

Hey Nacho,

Yeah, I think it is definitely a call we have to make case by case. We have
some experience with this: originally we attempted to maintain things like
non-java clients, a hadoop connector, etc all in the main project. The
difficulty of that lead us to the current federated approach. In terms of
what is included now, yes, I agree you could potentially have even less
included.

-Jay

On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis 
wrote:

> What is the criteria for keeping things in and out of Kafka, what code goes
> in or out and what is part of the architecture or not?
>
> The discussion of what goes into a project and what stays out is an always
> evolving question. Different projects treat this in different ways.
>
> Let me paint 2 extremes.  On one side, you have a single monolithic project
> that brings everything in one tent.  On the other side you have the many
> modules approach.  From what I've learned, Kafka falls in the middle.
> Because of this, the question is bound to come up with respect to the
> criteria used to bring something into the fold.
>
> I'll be the first to point out that the distinction between modules,
> architecture, software, repositories, governance and community are blurry.
> Not to mention that many things are how they are for historical reasons.
>
> I, personally, can't understand why we would not have REST as part of the
> main Kafka project given that a lot of people use it and we include many
> things with the current distribution.  What many things you may ask?  Well,
> if we took the modular approach Kafka is a mixture of components, here's
> the first 4 that come to mind:
> 1. The Kafka protocol
> 2. The Kafka java libraries
> 3. The Kafka broker
> 4. The Kafka stream framework
> 5. Kafka Connect
> 6. MirrorMaker
>
> All of these could be separate products. You should be able to evolve each
> one independently.  Even if they have dependencies on each other, you could
> potentially replace one part.
>
> The choice of keeping them all in a single repository, with a single
> distribution, under the same governance and community, brings a number of
> trade offs.  It's easy to keep things coherent for example.  There is less
> of a need to rely on inherent versioning and compatibility (which we end up
> providing anyway because of the way people usually deploy kafka). We all
> focus our efforts on a single code base.
>
> The downside is that it's harder to remove modules that are old or unused.
> Modules that are only used by a small subset of the community will have an
> impact on the rest of the community.  It mixes incentives of what people
> want to work on and what holds them back.  We also need to decide what
> belongs in the blessed bundle and what doesnt.
>
> So, my question boils down to, what criteria is used for bringing stuff in.
>
> If we have Streams and MirrorMaker and Connect in there, why not have REST?
> Specially if there is more than one person/group willing to work on it?
> Alternatively, if REST is not included because it's not used by all, then
> why not remove Streams, Connect and MirrorMaker since they're definitely
> not used by all? I realize I say this even though at LinkedIn we have a
> REST setup of our own, just speaking from a community perspective.
>
> Nacho
>
>
> (I'm relatively new and I haven't read all of the mail archive, so I'm sure
> this has been brought up before, but I decided to chime in anyway)
>
> On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps  wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/kafka access that is heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these things.
> > I've participated in open source projects out of LinkedIn via github as
> > well as via the ASF. I don't think there is a "right" answer to how to do
> > these but rather some tradeoffs. We thought about this quite a lot in the
> > context of Kafka based on the experience with the Hadoop ecosystem as
> well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost
> none
> > of them are top level apache projects. I don't think trying to force each
> > of these to 

[jira] [Updated] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian updated KAFKA-2167:
---
Assignee: (was: Neelesh Srinivas Salian)

> ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
> --
>
> Key: KAFKA-2167
> URL: https://issues.apache.org/jira/browse/KAFKA-2167
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jon Bringhurst
>  Labels: newbie
>
> I'm not 100% sure on this, but it seems like "persistent" should instead say 
> "ephemeral" in the JavaDoc. Also, note that "parrent" is misspelled.
> {noformat}
>   /**
>* Update the value of a persistent node with the given path and data.
>* create parrent directory if necessary. Never throw NodeExistException.
>*/
>   def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit 
> = {
> try {
>   client.writeData(path, data)
> }
> catch {
>   case e: ZkNoNodeException => {
> createParentPath(client, path)
> client.createEphemeral(path, data)
>   }
>   case e2 => throw e2
> }
>   }
> {noformat}
> should be:
> {noformat}
>   /**
>* Update the value of an ephemeral node with the given path and data.
>* create parent directory if necessary. Never throw NodeExistException.
>*/
>   def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit 
> = {
> try {
>   client.writeData(path, data)
> }
> catch {
>   case e: ZkNoNodeException => {
> createParentPath(client, path)
> client.createEphemeral(path, data)
>   }
>   case e2 => throw e2
> }
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian updated KAFKA-2167:
---
Comment: was deleted

(was: [~nehanarkhede], could you please help review this JIRA?

Thank you.
Regards,
Neelesh.)

> ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
> --
>
> Key: KAFKA-2167
> URL: https://issues.apache.org/jira/browse/KAFKA-2167
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jon Bringhurst
>Assignee: Neelesh Srinivas Salian
>  Labels: newbie
>
> I'm not 100% sure on this, but it seems like "persistent" should instead say 
> "ephemeral" in the JavaDoc. Also, note that "parrent" is misspelled.
> {noformat}
>   /**
>* Update the value of a persistent node with the given path and data.
>* create parrent directory if necessary. Never throw NodeExistException.
>*/
>   def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit 
> = {
> try {
>   client.writeData(path, data)
> }
> catch {
>   case e: ZkNoNodeException => {
> createParentPath(client, path)
> client.createEphemeral(path, data)
>   }
>   case e2 => throw e2
> }
>   }
> {noformat}
> should be:
> {noformat}
>   /**
>* Update the value of an ephemeral node with the given path and data.
>* create parent directory if necessary. Never throw NodeExistException.
>*/
>   def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit 
> = {
> try {
>   client.writeData(path, data)
> }
> catch {
>   case e: ZkNoNodeException => {
> createParentPath(client, path)
> client.createEphemeral(path, data)
>   }
>   case e2 => throw e2
> }
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: delete topic causing spikes in fetch/metadata requests

[sunil kalva]

Hi
Can you guys help me with this issue

On Oct 12, 2016 10:35 PM, "sunil kalva"  wrote:

>
> We are using kafka 0.8.2.2 (client and server), when ever we delete a
> topic we see lot of errors in broker logs like below, and there is also a
> spike in fetch/metadata requests. Can i correlate these errors with topic
> delete or its a known issue. Since there is spike in metadata requests and
> fetch requests broker throughput has comedown.
>
> 
> 
> 
> --
> [2016-10-12 16:04:55,054] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,056] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,057] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,059] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,060] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,062] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,064] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,065] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,067] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,068] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,070] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,072] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 16:04:55,073] ERROR [Replica Manager on Broker 4]: Error when
> processing fetch request for partition [xyz,0] offset 161946645 from
> consumer with correlation id 0. Possible cause: Request for offset
> 161946645 but we only have log segments in the range 185487049 to
> 202816546. (kafka.server.ReplicaManager)
> [2016-10-12 

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

[????????]

Hi Dong,
The KIP is used to solve both these 2 cases, we specify a small consumed 
log retention time to deleted the consumed data and avoid losing un-consumed 
data.
And the specify a large force log retention time used as higher bound for the 
data.  I will update the KIP for this info.
Another solution I think may be ok is to support an API to delete the 
inactive group?  If the group is in inactive, but it's commit offset is also in 
the __commit_offsets topic and 
stay in the offset cache,  we can delete it via this API.


Thanks,
David


--  --
??: "Dong Lin";;
: 2016??10??14??(??) 5:01
??: "dev"; 

: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention



Hi David,

As explained in the motivation section of the KIP, the problem is that if
log retention is too small, we may lose data; and if log retention is too
large, then we waste disk space. Therefore, we need to solve one if the two
problems -- allow data to be persisted longer for consumption if log
retention is set too small, or allow data to be expired earlier if log
retention is too large. I think the KIP probably needs to make this clear
and explain which one is rejected and why. Note that the choice of the two
affects the solution -- if we want to address the first problem then
log.retention.ms should be used as lower bound on the actual retention
time, and if we want to address the second problem then the log.retention.ms
should be used as higher bound on the actual retention time.

In both cases, we probably need to figure out a way to determine "active
consumer group". Maybe we can compare the time-since-last-commit against a
threshold to determine this. In addition, the threshold can be overridden
either per-topic or per-groupId. If we go along this route, the rejected
solution (per-topic vs. per-groupId) should probably be explained in the
KIP.


Thanks,
Dong



On Thu, Oct 13, 2016 at 10:23 AM, Dong Lin  wrote:

> Hi David,
>
> Thanks for your explanation. There still seems to be issue with this
> solution. Please see my comment inline.
>
>
> On Thu, Oct 13, 2016 at 8:46 AM,  <254479...@qq.com> wrote:
>
>> Hi Dong,
>> Sorry for the delay, here are the comments:
>> 1.I think we should distinguish these two cases:
>> (1) group has no member, but has commit offset :  In this case we should
>> consider its commit offset
>> (2) group has no member, no commit offset:  Skip this group
>> Is it ok?
>>
>>
>> ListGroup API can list the groups,  but this API only show the Online
>> Group, so we should enhance the listGroup API to list those groups in the
>> case (1)
>>
>> Say some user starts a consumer to consume topic A with
> enable.auto.commit = true. Later they change the group name in the config.
> Then the proposed solution will never execute consumed log retention for
> the topic A, right? I think group name change is pretty common and we
> should take care of this issue. One possible solution is to add a config to
> specify the maximum time since last offset commit before we consider a
> group is inactive.
>
>
>>
>> 2. Because every consumer group may appear in different time, say, group
>> 1 start to consume in day 1, group 2 start to consume in day 2.  If we
>> delete the log segment right away,
>> group 2 can not consume these message.  So we hope the messages can hold
>> for a specified time.  I think many use-cases will need there configs, if
>> there are many consumer groups.
>>
>>
> If we want to take care of group 2, can we simply disable consumed log
> retention for the topic and set log retention to 1 day? Can you explain the
> benefit of enabling consumed log retention and set consumed log retention
> to 1 day?
>
> Currently the flow graph in the KIP suggests that we delete data iff
> (consumed log retention is triggered OR forced log retention is triggered).
> And alternative solution is to delete data iff ( (consumed log retention is
> disabled OR consumed log retention is triggered) AND forced log retention
> is triggered). I would argue that the 2nd scheme is better. Say the
> consumed log retention is enabled. The 1st scheme basically interprets
> forced log retention as the upper bound of the time the data can stay in
> Kafka. The 2nd scheme interprets forced log retention as the lower bound of
> the time the data can stay in Kafka, which is more consistent with the
> purpose of having this forced log retention (to save disk space). And if we
> adopt the 2nd solution, the use-case you suggested can be easily addressed
> by setting forced log retention to 1 day and enable consumed log retention.
> What do you think?
>
>
>
>>
>> Thanks,
>> David
>>
>>
>>
>>
>> --  --
>> ??: "Dong Lin";;
>> : 2016??10??10??(??) 4:05
>> ??: "dev";
>>
>> : Re: 

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

[????????]

Hi Dong,
   Thanks for the comments:


1.
Say some user starts a consumer to consume topic A with enable.auto.commit
= true. Later they change the group name in the config. Then the proposed
solution will never execute consumed log retention for the topic A, right?
I think group name change is pretty common and we should take care of this
issue. One possible solution is to add a config to specify the maximum time
since last offset commit before we consider a group is inactive.



If the group name changed, we don't know whether the group name will be reused 
or not. If the group's commit offset
is discard in a specified time, why not used the commit offset timeout? Then 
this inactive group's will actually be remove. 


2.
If we want to take care of group 2, can we simply disable consumed log
retention for the topic and set log retention to 1 day? Can you explain the
benefit of enabling consumed log retention and set consumed log retention
to 1 day?
If we want to take care of group 2 which is starting to consume in day 2,  we 
should at least save the message in day 2.
Like we set the log.retention.commitoffset to day 3.  


Currently the flow graph in the KIP suggests that we delete data iff
(consumed log retention is triggered OR forced log retention is triggered).
And alternative solution is to delete data iff ( (consumed log retention is
disabled OR consumed log retention is triggered) AND forced log retention
is triggered). I would argue that the 2nd scheme is better. Say the
consumed log retention is enabled. The 1st scheme basically interprets
forced log retention as the upper bound of the time the data can stay in
Kafka. The 2nd scheme interprets forced log retention as the lower bound of
the time the data can stay in Kafka, which is more consistent with the
purpose of having this forced log retention (to save disk space). And if we
adopt the 2nd solution, the use-case you suggested can be easily addressed
by setting forced log retention to 1 day and enable consumed log retention.
What do you think?


Yes , that is the KIP's suggestion, the consumed log retention is before the 
force log retention, they can be both triggered.
For my use-case,  the force log retention should be larger than consumed log 
retention, say for 7 days, and consumed log retention
is set to 3 days, so after 3 days, we can quickly clean the messages which are 
consumed. And after 7 days, even if the messages are not 
consumed, we will clean it anyway.






--  --
??: "Dong Lin";;
: 2016??10??14??(??) 1:23
??: "dev"; 

: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention



Hi David,

Thanks for your explanation. There still seems to be issue with this
solution. Please see my comment inline.


On Thu, Oct 13, 2016 at 8:46 AM,  <254479...@qq.com> wrote:

> Hi Dong,
> Sorry for the delay, here are the comments:
> 1.I think we should distinguish these two cases:
> (1) group has no member, but has commit offset :  In this case we should
> consider its commit offset
> (2) group has no member, no commit offset:  Skip this group
> Is it ok?
>
>
> ListGroup API can list the groups,  but this API only show the Online
> Group, so we should enhance the listGroup API to list those groups in the
> case (1)
>
> Say some user starts a consumer to consume topic A with enable.auto.commit
= true. Later they change the group name in the config. Then the proposed
solution will never execute consumed log retention for the topic A, right?
I think group name change is pretty common and we should take care of this
issue. One possible solution is to add a config to specify the maximum time
since last offset commit before we consider a group is inactive.


>
> 2. Because every consumer group may appear in different time, say, group 1
> start to consume in day 1, group 2 start to consume in day 2.  If we delete
> the log segment right away,
> group 2 can not consume these message.  So we hope the messages can hold
> for a specified time.  I think many use-cases will need there configs, if
> there are many consumer groups.
>
>
If we want to take care of group 2, can we simply disable consumed log
retention for the topic and set log retention to 1 day? Can you explain the
benefit of enabling consumed log retention and set consumed log retention
to 1 day?

Currently the flow graph in the KIP suggests that we delete data iff
(consumed log retention is triggered OR forced log retention is triggered).
And alternative solution is to delete data iff ( (consumed log retention is
disabled OR consumed log retention is triggered) AND forced log retention
is triggered). I would argue that the 2nd scheme is better. Say the
consumed log retention is enabled. The 1st scheme basically interprets
forced log retention as the upper bound of the time the data can stay in
Kafka. The 2nd scheme interprets forced log retention as the 

?????? [DISCUSS] KIP-68 Add a consumed log retention before log retention

[????????]

Hi Renu:
Sorry for the delay, here are the comments:  
1. You mean the config for the topic ? We also support the per topic's consumed 
retention configuration. 


2. The consumer group's commit offset timeout is support in the 0.9, the 
consumed retention only concern about the current commit offset.
log retention will not stop even after the consumer disappears.


3. You can set to a very small time, for example 1ms, so the log will be 
deleted after consumed.
Set to 0 will throw error now.


4. Log retention timeout will not change, it only depend on the now time -  
last modified time of the log file.
In the case when a new consumer comes,  will find the new min commit offset in 
the consumed retention process.


Thanks,
David


--  --
??: "Renu Tewari";;
: 2016??10??11??(??) 4:55
??: "dev"; 

: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention



Hi David
  This is a very timely KIP given the number of use cases in the streams
processing pipeline than need consumed log retention management.

Some questions that Becket and Dong asked just wanted to make sure are
described in the KIP.

1. How is the configuration setup per topic to know what is the set of
consumer groups that are "subscribed" to this topic whose committed offsets
will be tracked. Can we have more details on how this will be dynamically
tracked as consumers come and go.
2. Is there a timeout to determine if a consumer group has stopped
committing offsets to topic partitions that they had earlier consumed? Or
the consumed log retention will track each known consumer/consumers groups
committed offset and stop any cleaning if a consumer disappears after
consuming. This is to Dong's earlier question.
3. Can the log.retention value be set to 0 to indicate the log is set to be
cleaned to the min committed offset immediately after it has been consumed?

4. What guarantee is given on when the consumed log will eventually be
cleaned. If the log.retention timeout is enabled for a consumed offset and
a new consumer starts consuming from the beginning then is the min
committed offset value changed and the timer based on log.retention timeout
restarted?

 This kind of all relates to active and inactive consumers and if the set
changes dynamically how does the consumed log retention actually make
progress.

regards
renu


On Mon, Oct 10, 2016 at 1:05 AM, Dong Lin  wrote:

> Hey David,
>
> Thanks for reply. Please see comment inline.
>
> On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L) 
> wrote:
>
> > Hi Dong
> >Thanks for the questions:
> >
> > 1.  Now we don't distinguish inactive or active groups. Because in some
> > case maybe inactive group will become active again, and using the
> previous
> > commit offset.
> >
> > So we will not delete the log segment in the consumer retention if there
> > are some groups consume but not commit, but the log segment can be
> delete by
> >  the force retention.
> >
>
> So in the example I provided, the consumed log retention will be
> effectively disabled, right? This seems to be a real problem in operation
> -- we don't want log retention to be un-intentionally disabled simply
> because someone start a tool to consume from that topic. Either this KIP
> should provide a way to handle this, or there should be a way for operator
> to be aware of such case and be able to re-eanble consumed log retention
> for the topic. What do you think?
>
>
>
> > 2.  These configs are used to determine the out of date time of the
> > consumed retention, like the parameters of the force retention
> > (log.retention.hours, log.retention.minutes, log.retention.ms). For
> > example, users want the save the log for 3 days, after 3 days, kafka will
> > delete the log segments which are
> >
> > consumed by all the consumer group.  The log retention thread need these
> > parameters.
> >
> > It makes sense to have configs such as log.retention.ms -- it is used to
> make data available for up to a configured amount of time before it is
> deleted. My question is what is the use-case for making log available for
> another e.g. 3 days after it has been consumed by all consumer groups. The
> purpose of this KIP is to allow log to be deleted right as long as all
> interested consumer groups have consumed it. Can you provide a use-case for
> keeping log available for longer time after it has been consumed by all
> groups?
>
>
> >
> > Thanks,
> > David
> >
> >
> > > Hey David,
> > >
> > > Thanks for the KIP. Can you help with the following two questions:
> > >
> > > 1) If someone start a consumer (e.g. kafka-console-consumer) to
> consume a
> > > topic for debug/validation purpose, a randome consumer group may be
> > created
> > > and offset may be committed for this consumer group. If no offset
> commit
> > is
> > > made for this consumer group in the future, will