Hi all, Just as a quick reminder, this is not really a complete proposal. There are a bunch of unresolved issues with this KIP. One example is how this interacts with incremental fetch sessions. It is not mentioned anywhere in the KIP text. Previously we discussed some approaches, but there was no clear consensus.
Another example is the issue of starvation. The KIP discusses "an idea" for handling starvation, but the details are very sparse-- just a sentence of two. At minimum we would need some kind of configuration for the proposed "lag deltas". It's also not clear that the proposed mechanism would work, since we don't receive lag metrics for partitions that we don't fetch. But if we do fetch from the partitions, we may receive data, which would cause our policy to not be strict prioties. Keep in mind, even attempting to fetch 1 byte may cause us to read an entire message, as described in KIP-74. It seems that we don't understand the potential use-cases. The only use-case referenced by the KIP is this one, by Bala Prassanna: > We use Kafka to process the asynchronous events of our Document Management > System such as preview generation, indexing for search etc. > The traffic gets generated via Web and Desktop Sync application. In such > cases, we had to prioritize the traffic from web and consume them first. > But this might lead to the starvation of events from sync if the consumer > speed is slow and the event rate is high from web. A solution to handle > the starvation with a timeout after which the events are consumed normally > for a specified period of time would be great and help us use our > resources effectively. Reading this carefully, it seems that the problem is actually starvation, not implementing priorities. Bala already implemented priorities outside of Kafka. If you read the discussion on KAFKA-6690, Bala also makes this comment: "We would need this in both Consumer API and Streams API." The current KIP does not discuss adding priorities to Streams-- only to the basic consumer API. So it seems clear that KIP-349 does not address Bala's use-case at all. Stepping back a little bit, it seems like a few people have spoken up recently asking for some way to re-order the messages they receive from the Kafka consumer. For example, ChienHsing Wu has discussed a use-case where he wants to receive messages in a "round robin" order. All of this is possible by doing some local buffering and using the pause and resume APIs. Perhaps we should consider better documenting these APIs, and adding some examples. Or perhaps we should consider some kind of API to do pluggable buffering on the client side. In any case, this needs more discussion. We need to be clear and definite about what use cases we want to solve, and the tradeoffs we're making to solve them. For now, I have to reiterate my -1 (binding). Colin On Thu, Jan 10, 2019, at 10:46, Adam Bellemare wrote: > Looks good to me then! > > +1 non-binding > > > > > On Jan 10, 2019, at 1:22 PM, Afshartous, Nick <nafshart...@wbgames.com> > > wrote: > > > > > > Hi Adam, > > > > > > This change is only intended for the basic consumer API. > > > > > > Cheers, > > > > -- > > > > Nick > > > > > > ________________________________ > > From: Adam Bellemare <adam.bellem...@gmail.com> > > Sent: Sunday, January 6, 2019 11:45 AM > > To: dev@kafka.apache.org > > Subject: Re: [VOTE] KIP-349 Priorities for Source Topics > > > > Hi Nick > > > > Is this change only for the basic consumer? How would this affect anything > > with Kafka Streams? > > > > Thanks > > > > > >> On Jan 5, 2019, at 10:52 PM, n...@afshartous.com wrote: > >> > >> Bumping again for more votes. > >> -- > >> Nick > >> > >> > >>> On Dec 26, 2018, at 12:36 PM, n...@afshartous.com wrote: > >>> > >>> Bumping this thread for more votes > >>> > >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= > >>> > >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e=><https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e= > >>> > >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e=>> > >> > >> > >> > >> >