Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Colin, that is definitely a good option and will cover 90% of all use cases (probaby more). However strictly speaking it only addresses one half of the issue unless I am mistaken. The internal behavior of the KafkaConsumer (which partition the fetcher gets data from next and which buffered

Re: [VOTE] KIP-349 Priorities for Source Topics

On Sat, Mar 23, 2019, at 18:41, nathank...@gmail.com wrote: > > > On 2019/01/28 02:26:31, n...@afshartous.com wrote: > > Hi Sönke, > > > > Thanks for taking the time to review. I’ve put KIP-349 into hibernation. > > > > Thanks also to everyone who participated in the discussion. > > > >

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Nathan, I have a couple of remarks/questions about your mail, if I may. First of all, the javadoc for the pause operation of KafkaConsumer states: "Suspend fetching from the requested partitions. Future calls to poll(Duration)

Re: [VOTE] KIP-349 Priorities for Source Topics

On 2019/01/28 02:26:31, n...@afshartous.com wrote: > Hi Sönke, > > Thanks for taking the time to review. I’ve put KIP-349 into hibernation. > > Thanks also to everyone who participated in the discussion. > > Best regards, > -- > Nick > > > On Jan 25, 2019, at 5:51 AM, Sönke Liebau

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Sönke, Thanks for taking the time to review. I’ve put KIP-349 into hibernation. Thanks also to everyone who participated in the discussion. Best regards, -- Nick > On Jan 25, 2019, at 5:51 AM, Sönke Liebau > wrote: > > a bit late to the party, sorry. I recently spent some time

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Nick, a bit late to the party, sorry. I recently spent some time looking into this / a similar issue [1]. After some investigation and playing around with settings I think that the benefit that could be gained from this is somewhat limited and probably outweighed by the implementation effort.

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Colin, > On Jan 24, 2019, at 12:14 PM, Colin McCabe wrote: > > Users almost always like the idea of new features, whatever they are. But > that doesn't mean that the feature would necessarily work well or be > necessary. Yes, though we should certainly consider the responses on the user

Re: [VOTE] KIP-349 Priorities for Source Topics

On Thu, Jan 24, 2019, at 05:31, n...@afshartous.com wrote: > > > On Jan 17, 2019, at 8:49 PM, n...@afshartous.com wrote: > > > >> On Jan 15, 2019, at 2:26 PM, Colin McCabe >> > wrote: > >> > >> I think it makes sense to go back to use-cases again. So far, all of the

Re: [VOTE] KIP-349 Priorities for Source Topics

On 24.01.2019 15:51, Thomas Becker wrote: > Yes, I think this type of strategy interface would be valuable. > Thank you for leaving this here!

Re: [VOTE] KIP-349 Priorities for Source Topics

Yes, I think this type of strategy interface would be valuable. On Wed, 2019-01-16 at 15:41 +, Jan Filipiak wrote: On 16.01.2019 14:05, Thomas Becker wrote: I'm going to bow out of this discussion since it's been made clear that the feature is not targeted at streams. But for the record,

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Jan 17, 2019, at 8:49 PM, n...@afshartous.com wrote: > >> On Jan 15, 2019, at 2:26 PM, Colin McCabe > > wrote: >> >> I think it makes sense to go back to use-cases again. So far, all of the >> use-cases we discussed could be handled by pause and resume. So it

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Jan 15, 2019, at 2:26 PM, Colin McCabe wrote: > > I think it makes sense to go back to use-cases again. So far, all of the > use-cases we discussed could be handled by pause and resume. So it makes > sense to try to figure out what the issue with those APIs is. Are they not >

Re: [VOTE] KIP-349 Priorities for Source Topics

On 16.01.2019 14:05, Thomas Becker wrote: > I'm going to bow out of this discussion since it's been made clear that > the feature is not targeted at streams. But for the record, my desire is > to have an alternative to the timestamp based message choosing strategy > streams currently imposes,

Re: [VOTE] KIP-349 Priorities for Source Topics

I'm going to bow out of this discussion since it's been made clear that the feature is not targeted at streams. But for the record, my desire is to have an alternative to the timestamp based message choosing strategy streams currently imposes, and I thought topic prioritization in the consumer

Re: [VOTE] KIP-349 Priorities for Source Topics

On Sun, Jan 13, 2019, at 18:13, n...@afshartous.com wrote: > Thanks Colin and Mathias. > > > On Jan 12, 2019, at 8:27 PM, Matthias J. Sax wrote: > > > > Thus, I would suggest to limit this KIP to the consumer only, otherwise, > > the scope will be too large and this KIP will drag on even

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Jan, As discussed, I’ve adopted the position that MessageChooser is orthogonal to topic prioritization and hence outside the scope of KIP-349. -- Nick > On Jan 14, 2019, at 12:47 AM, Jan Filipiak wrote: > > On 14.01.2019 02:48, n...@afshartous.com

Re: [VOTE] KIP-349 Priorities for Source Topics

On 14.01.2019 02:48, n...@afshartous.com wrote: > > On reflection, it would be hard to describe the semantics of an API that > tried to address starvation by temporarily disabling prioritization, and then > oscillating back and forth. > Thus I agree that it makes sense not to try and address

Re: [VOTE] KIP-349 Priorities for Source Topics

Thanks Colin and Mathias. > On Jan 12, 2019, at 8:27 PM, Matthias J. Sax wrote: > > Thus, I would suggest to limit this KIP to the consumer only, otherwise, > the scope will be too large and this KIP will drag on even longer. If we > really want to add this to Kafka Streams, I expect a long and

Re: [VOTE] KIP-349 Priorities for Source Topics

n >> >> >> On Thu, Jan 10, 2019, at 10:46, Adam Bellemare wrote: >>> Looks good to me then! >>> >>> +1 non-binding >>> >>> >>> >>>> On Jan 10, 2019, at 1:22 PM, Afshartous, Nick >> wrote: >>>> >>&g

Re: [VOTE] KIP-349 Priorities for Source Topics

> > > Hi Adam, > > > > > > > > > This change is only intended for the basic consumer API. > > > > > > > > > Cheers, > > > > > > -- > > > > > >Nick > > > > > > > > > _

Re: [VOTE] KIP-349 Priorities for Source Topics

gt; > > > > This change is only intended for the basic consumer API. > > > > > > Cheers, > > > > -- > > > > Nick > > > > > > > > From: Adam Bellemare > > Sent: Sunday, January 6, 2019 11:45 AM > > To: dev@kafka.ap

Re: [VOTE] KIP-349 Priorities for Source Topics

> From: Adam Bellemare > Sent: Sunday, January 6, 2019 11:45 AM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-349 Priorities for Source Topics > > Hi Nick > > Is this change only for the basic consumer? How would this affect anything > with Kafka

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Adam, This change is only intended for the basic consumer API. Cheers, -- Nick From: Adam Bellemare Sent: Sunday, January 6, 2019 11:45 AM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-349 Priorities for Source Topics Hi Nick Is this change

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Nick Is this change only for the basic consumer? How would this affect anything with Kafka Streams? Thanks > On Jan 5, 2019, at 10:52 PM, n...@afshartous.com wrote: > > Bumping again for more votes. > -- > Nick > > >> On Dec 26, 2018, at 12:36 PM, n...@afshartous.com wrote: >>

Re: [VOTE] KIP-349 Priorities for Source Topics

Bumping again for more votes. -- Nick > On Dec 26, 2018, at 12:36 PM, n...@afshartous.com wrote: > > Bumping this thread for more votes > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349:+Priorities+for+Source+Topics > >

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi All, Bumping this thread for more votes https://cwiki.apache.org/confluence/display/KAFKA/KIP-349:+Priorities+for+Source+Topics Cheers, -- Nick

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Oct 26, 2018, at 2:00 PM, Colin McCabe wrote: >> > Priorities won't help for this use-case, right? If the "web" partition has a > higher priority, and data is always available, there will *never* be any > events reported for "sync". Priorities don't prevent starvation-- they cause >

Re: [VOTE] KIP-349 Priorities for Source Topics

On Thu, Oct 25, 2018, at 18:16, n...@afshartous.com wrote: > > The reporter of KAFKA-6690 (Bala) replied in the JIra ticket to my > question to elaborate about his use-case. I don’t think he’s on the dev > list. Here’s his response: > > Bala: Sorry about the delay in reply. We use Kafka

Re: [VOTE] KIP-349 Priorities for Source Topics

The reporter of KAFKA-6690 (Bala) replied in the JIra ticket to my question to elaborate about his use-case. I don’t think he’s on the dev list. Here’s his response: Bala: Sorry about the delay in reply. We use Kafka to process the asynchronous events of our Document Management System

Re: [VOTE] KIP-349 Priorities for Source Topics

On Thu, Oct 18, 2018, at 09:23, n...@afshartous.com wrote: > > > > On Oct 12, 2018, at 5:06 PM, Colin McCabe wrote: > > > > Maybe there's some really cool use-case that I haven't thought of. But so > > far I can't really think of any time I would need topic priorities if I was > > muting

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Oct 12, 2018, at 5:06 PM, Colin McCabe wrote: > > Maybe there's some really cool use-case that I haven't thought of. But so > far I can't really think of any time I would need topic priorities if I was > muting topics and offloading blocking operations in a reasonable way. It > would

Re: [VOTE] KIP-349 Priorities for Source Topics

On Mon, Oct 8, 2018, at 12:35, Thomas Becker wrote: > Well my (perhaps flawed) understanding of topic priorities is that lower > priority topics are not consumed as long as higher priority ones have > unconsumed messages (which means our position < HW). So if I'm doing > this manually, I have

Re: [VOTE] KIP-349 Priorities for Source Topics

Well my (perhaps flawed) understanding of topic priorities is that lower priority topics are not consumed as long as higher priority ones have unconsumed messages (which means our position < HW). So if I'm doing this manually, I have to make some determination as to whether my high priority

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Oct 5, 2018, at 2:25 PM, Colin McCabe wrote: > > t's possible for the change to be 100% backwards compatible, but still not > have a separate code path for people who don't want to use this feature, > right? What I am getting at is basically: will this feature increase > broker-side

Re: [VOTE] KIP-349 Priorities for Source Topics

On Fri, Oct 5, 2018, at 10:58, Thomas Becker wrote: > Colin, > Would you mind sharing your vision for how this looks with multiple > consumers? I'm still getting my bearings with the new consumer but it's > not immediately obvious to me how this would work. Hi Thomas, I was just responding to

Re: [VOTE] KIP-349 Priorities for Source Topics

On Wed, Oct 3, 2018, at 16:01, n...@afshartous.com wrote: > > > > On Oct 3, 2018, at 12:41 PM, Colin McCabe wrote: > > > > Will there be a separate code path for people who don't want to use this > > feature? > > > Yes, I tried to capture this in the KIP by indicating that this API >

Re: [VOTE] KIP-349 Priorities for Source Topics

Colin, Would you mind sharing your vision for how this looks with multiple consumers? I'm still getting my bearings with the new consumer but it's not immediately obvious to me how this would work. In particular, it doesn't seem particularly easy to know when you are at the high watermark of a

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Oct 3, 2018, at 12:41 PM, Colin McCabe wrote: > > Will there be a separate code path for people who don't want to use this > feature? Yes, I tried to capture this in the KIP by indicating that this API change is 100% backwards compatible. Current consumer semantics and performance

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi all, I feel like the DISCUSS thread didn't really come to a conclusion, so a vote would be premature here. In particular, I still don't really understand the use-case for this feature. Can someone give a concrete scenario where you would need this? The control plane / data plane example

Re: [VOTE] KIP-349 Priorities for Source Topics

Great. +1 (non-binding) On Mon, Oct 1, 2018 at 4:23 AM Matthias J. Sax wrote: > +1 (binding) > > As Dongjin pointed out, the community is working on upcoming 2.1 > release, and thus it might take some time until people find time to > follow up on this an vote. > > > -Matthias > > On 9/30/18

Re: [VOTE] KIP-349 Priorities for Source Topics

+1 (binding) As Dongjin pointed out, the community is working on upcoming 2.1 release, and thus it might take some time until people find time to follow up on this an vote. -Matthias On 9/30/18 11:11 AM, n...@afshartous.com wrote: > >> On Sep 30, 2018, at 5:16 AM, Dongjin Lee wrote: >> >> 1.

Re: [VOTE] KIP-349 Priorities for Source Topics

> On Sep 30, 2018, at 5:16 AM, Dongjin Lee wrote: > > 1. Your KIP document > > > > lacks hyperlink to the

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Nick, Thanks for your proposal. However, I have two things I would like to point out: 1. Your KIP document lacks hyperlink to the discussion thread. And I couldn`t find the discussion thread from the

Re: [VOTE] KIP-349 Priorities for Source Topics

That sound correct, Colin. At runtime (we just merged an improvement this week, cf KIP-353), Kafka Streams synchronizes different topics based on record timestamps. Records are buffered internally before processed and we `pause()` partitions for which the number of records in the buffer exceeds a

Re: [VOTE] KIP-349 Priorities for Source Topics

Hmm. My understanding is that streams doesn't need anything like this since streams pauses topics when it doesn't need more data from them. (Matthias, can you confirm?) best, Colin On Mon, Aug 20, 2018, at 06:01, Thomas Becker wrote: > I agree with Jan. A strategy interface for choosing

Re: [VOTE] KIP-349 Priorities for Source Topics

also: https://cwiki.apache.org/confluence/display/KAFKA/KIP-353%3A+Improve+Kafka+Streams+Timestamp+Synchronization On 20.08.2018 15:01, Thomas Becker wrote: I agree with Jan. A strategy interface for choosing processing order is nice, and would hopefully be a step towards getting this in

Re: [VOTE] KIP-349 Priorities for Source Topics

I agree with Jan. A strategy interface for choosing processing order is nice, and would hopefully be a step towards getting this in streams. -Tommy On Mon, 2018-08-20 at 12:52 +0200, Jan Filipiak wrote: On 20.08.2018 00:19, Matthias J. Sax wrote: @Nick: A KIP is only accepted if it got 3

Re: [VOTE] KIP-349 Priorities for Source Topics

On 20.08.2018 00:19, Matthias J. Sax wrote: @Nick: A KIP is only accepted if it got 3 binding votes, ie, votes from committers. If you close the vote before that, the KIP would not be accepted. Note that committers need to pay attention to a lot of KIPs and it can take a while until people can

Re: [VOTE] KIP-349 Priorities for Source Topics

@Nick: A KIP is only accepted if it got 3 binding votes, ie, votes from committers. If you close the vote before that, the KIP would not be accepted. Note that committers need to pay attention to a lot of KIPs and it can take a while until people can look into it. Thanks for your understanding.

Re: [VOTE] KIP-349 Priorities for Source Topics

While I am not sure if I can or can’t vote, my question re: Jan’s comment is, “should we be implementing it as Samza does?” I am not familiar with the drawbacks of the current approach vs how samza does it. > On Aug 18, 2018, at 5:06 PM, n...@afshartous.com wrote: > > > I only saw one vote

Re: [VOTE] KIP-349 Priorities for Source Topics

I only saw one vote on KIP-349, just checking to see if anyone else would like to vote before closing this out. -- Nick > On Aug 13, 2018, at 9:19 PM, n...@afshartous.com wrote: > > > Hi All, > > Calling for a vote on KIP-349 > >

Re: [VOTE] KIP-349 Priorities for Source Topics

Sorry for missing the discussion -1 nonbinding see https://samza.apache.org/learn/documentation/0.7.0/api/javadocs/org/apache/samza/system/chooser/MessageChooser.html Best Jan On 14.08.2018 03:19, n...@afshartous.com wrote: Hi All, Calling for a vote on KIP-349

Re: [VOTE] KIP-349 Priorities for Source Topics

Both use-case: Command queue for a streams job and work-queue in general seem reasonable. Thank you for explaining. On Sun, Aug 12, 2018 at 6:34 AM, Matt Farmer wrote: > The work-queue use case is mostly how I see this being used, yes. > > In the most generic sense I can see its use in a

Re: [VOTE] KIP-349 Priorities for Source Topics

In thinking on it, another solution for this is another consumer external to the stream - but then we run into timing issues and complexity with using a state store as the storage of record. :/ On Sun, Aug 12, 2018 at 9:34 AM Matt Farmer wrote: > The work-queue use case is mostly how I see this

Re: [VOTE] KIP-349 Priorities for Source Topics

The work-queue use case is mostly how I see this being used, yes. In the most generic sense I can see its use in a situation where the business dictates that we have to guarantee quality of service for some set of low number of messages while there's some high number of messages being processed

Re: [VOTE] KIP-349 Priorities for Source Topics

Can you guys spell it out for me? I just don't really see when I want to subscribe to two topics but not get events from both at the same time. Is this a work-queue type pattern? On Wed, Aug 8, 2018 at 6:10 PM, Matt Farmer wrote: > Oh, almost forgot, thanks for the KIP - I can see this being a

Re: [VOTE] KIP-349 Priorities for Source Topics

Oh, almost forgot, thanks for the KIP - I can see this being a very useful addition. :) On Wed, Aug 8, 2018 at 9:09 PM Matt Farmer wrote: > Is it worth spelling out explicitly what the behavior is when two topics > have the same priority? I'm a bit fuzzy on how we choose what topics to >

Re: [VOTE] KIP-349 Priorities for Source Topics

Is it worth spelling out explicitly what the behavior is when two topics have the same priority? I'm a bit fuzzy on how we choose what topics to consume from right now, if I'm being honest, so it could be useful to outline the current behavior in the background and to spell out how that would