[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770704#comment-17770704 ] Domenico Di Giulio commented on KAFKA-6690: --- I see this issue was closed but we also really need it A LOT. In our case two co-partitioned source topics are used in the same topology, one (let's call it topic "A") to accept new business requests, the other (let's call it topic "B") as a loopback (from downstream components) about business transactions already in progress, which come back to the same topology to update a transaction status (in a store) and proceed. We found that when the event rate on topic A is too high, Kafka Streams has no way to apply any backpressure on it and favor topic B. As a result, the processing on topic B progressively goes in starvation and the system proceeds much slower than it could. We're looking at how we could apply a backpressure on topic A, fetching more events from topic B so that the processing of ongoing business transactions can proceed with higher priority. So far we only thought about adding another upstream component (may be just a simple consumer and producer), which should read topic A and forward events to our topology based on a metric which looks for symptoms of congestion (business latency or whatever). But it would be very beneficial instead to just define B as a priority topic on Kafka Streams. Any news on this KIP ? Does the above use case look meaningful ? > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111093#comment-17111093 ] Nick Afshartous commented on KAFKA-6690: After long discussion on the dev list there was not enough support for this feature. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111073#comment-17111073 ] Wladimir Schmidt commented on KAFKA-6690: - any news on this? > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659806#comment-16659806 ] Bala Prassanna I commented on KAFKA-6690: - [~nafshartous] Sorry about the delay in reply. We use Kafka to process the aynchronous events of our Document Management System such as preview generation, indexing for search etc. The traffic gets generated via Web and Desktop Sync application. In such cases, we had to prioritise the traffic from web and consume them first. Though this may lead to starvation of events from sync, an API as such will help us effectively manage the available resources and prioritise our traffic without allocating separate server to handle the sync load. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659562#comment-16659562 ] Nick Afshartous commented on KAFKA-6690: [~balaprassanna] Just checking again if you can help with elaborating on your use-cases ? This would make it easier to facilitate a vote on this feature request. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597433#comment-16597433 ] Matthias J. Sax commented on KAFKA-6690: [~nafshartous] It's your KIP and thus your decision if you want to follow this suggesting or not. It would be good to understand the advantages and disadvantage of your current (simplified?) proposal compared to the suggestion. What you can do/not do with the or the other? Would it be possible to extend your current proposal with something more generic later on? You can also just continue the VOTE if you are not interested to change the proposal. A single non-binding downvote does not block the KIP. I will also follow up on the mailing list. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596862#comment-16596862
]
Nick Afshartous commented on KAFKA-6690:
There was a comment during the vote that we should look at Samza's
{{MessageChoose}}
https://samza.apache.org/learn/documentation/0.7.0/api/javadocs/org/apache/samza/system/chooser/MessageChooser.html
> Priorities for Source Topics
>
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: consumer
>Reporter: Bala Prassanna I
>Assignee: Nick Afshartous
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582410#comment-16582410 ] Nick Afshartous commented on KAFKA-6690: [~guozhang] I called for a VOTE on the dev list a few days ago. Only one person has voted so far (-1 nonbinding). Any suggestions on how to proceed, i.e. send a reminder soon or just drop the effort ? > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564809#comment-16564809 ] Bala Prassanna I commented on KAFKA-6690: - [~nafshartous] I am okay with the KIP > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564413#comment-16564413 ] Nick Afshartous commented on KAFKA-6690: [~balaprassanna] Please review before I take this to the dev mailing list for discussion https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics#KIP-349:PrioritiesforSourceTopics-Motivation > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562046#comment-16562046 ] Guozhang Wang commented on KAFKA-6690: -- I've granted you the permission and you should be able to create new KIP pages now. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561264#comment-16561264 ] Nick Afshartous commented on KAFKA-6690: [~guozhang] After I asked about how to proceed on the dev list,, [~enether] asked that I write a KIP. Last Monday I requested permission on the dev list to create a KIP and I didn't see a reply granting permission. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560402#comment-16560402 ] Guozhang Wang commented on KAFKA-6690: -- If we add it on the consumer API, then Streams may be automatically geared with this feature since it is leveraging on consumers (some different considerata would be done on Streams since today its messaging choosing is purely dependent on timestamp synchronization, but this can be deferred to a follow-up discussion). Maybe you can start the discussion thread on the mailing list and ask for the community's opinion on this feature request. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559356#comment-16559356 ] Bala Prassanna I commented on KAFKA-6690: - [~guozhang] We would need this in both Consumer API and also in Streams API > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555919#comment-16555919 ] Guozhang Wang commented on KAFKA-6690: -- [~balaprassanna] Your description on the JIRA seems to be requesting this feature on the Consumer API, while your comment above is suggesting adding it on the Streams API (note the Streams client actually has embedded Consumer client internally). Which type of client are you really thinking about? > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555472#comment-16555472
]
Bala Prassanna I commented on KAFKA-6690:
-
Would be helpful if the same option is also present in the following API
{code:java}
org.apache.kafka.streams.Topology.addSource(String name, String[] topics)
{code}
> Priorities for Source Topics
>
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bala Prassanna I
>Assignee: Nick Afshartous
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528109#comment-16528109
]
Nick Afshartous commented on KAFKA-6690:
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPrioritiy(java.lang.String topic, int priority)
{code}
then there would be a new {{KafkaConsumer.subscribe}} method
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture then intent ?
> Priorities for Source Topics
>
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bala Prassanna I
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
