[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770704#comment-17770704 ] Domenico Di Giulio edited comment on KAFKA-6690 at 9/30/23 12:54 PM: - I see this issue was closed but we also really need it A LOT. In our case two co-partitioned source topics are used in the same topology, one (let's call it topic "A") to accept new business requests, the other (let's call it topic "B") as a loopback (from downstream components) about business transactions already in progress, which come back to the same topology to update a transaction status (in a store) and proceed. We found that when the event rate on topic A is too high, Kafka Streams has no way to apply any backpressure on it and favor topic B. As a result, the processing on topic B progressively goes in starvation and the system proceeds much slower than it could. We're looking at how we could apply a backpressure on topic A, fetching more events from topic B so that the processing of ongoing business transactions can proceed with higher priority. So far we only thought about adding another upstream component (may be just a simple consumer and producer), which should read topic A and forward events to our topology based on a metric which looks for symptoms of congestion (business latency or whatever). But it would be very beneficial instead to just define B as a priority topic on Kafka Streams. Any news on this issue ? Does the above use case look meaningful ? was (Author: domenico74): I see this issue was closed but we also really need it A LOT. In our case two co-partitioned source topics are used in the same topology, one (let's call it topic "A") to accept new business requests, the other (let's call it topic "B") as a loopback (from downstream components) about business transactions already in progress, which come back to the same topology to update a transaction status (in a store) and proceed. We found that when the event rate on topic A is too high, Kafka Streams has no way to apply any backpressure on it and favor topic B. As a result, the processing on topic B progressively goes in starvation and the system proceeds much slower than it could. We're looking at how we could apply a backpressure on topic A, fetching more events from topic B so that the processing of ongoing business transactions can proceed with higher priority. So far we only thought about adding another upstream component (may be just a simple consumer and producer), which should read topic A and forward events to our topology based on a metric which looks for symptoms of congestion (business latency or whatever). But it would be very beneficial instead to just define B as a priority topic on Kafka Streams. Any news on this KIP ? Does the above use case look meaningful ? > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659806#comment-16659806 ] Bala Prassanna I edited comment on KAFKA-6690 at 10/23/18 3:31 AM: --- [~nafshartous] Sorry about the delay in reply. We use Kafka to process the asynchronous events of our Document Management System such as preview generation, indexing for search etc. The traffic gets generated via Web and Desktop Sync application. In such cases, we had to prioritize the traffic from web and consume them first. But this might lead to the starvation of events from sync if the consumer speed is slow and the event rate is high from web. A solution to handle the starvation with a timeout after which the events are consumed normally for a specified period of time would be great and help us use our resources effectively. was (Author: balaprassanna): [~nafshartous] Sorry about the delay in reply. We use Kafka to process the aynchronous events of our Document Management System such as preview generation, indexing for search etc. The traffic gets generated via Web and Desktop Sync application. In such cases, we had to prioritise the traffic from web and consume them first. Though this may lead to starvation of events from sync, an API as such will help us effectively manage the available resources and prioritise our traffic without allocating separate server to handle the sync load. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605047#comment-16605047 ] Nick Afshartous edited comment on KAFKA-6690 at 9/7/18 1:22 PM: [~balaprassanna] There is a question today in the dev list, see subject [DISCUSS] KIP-349 Priorities for Source Topics about the use-cases and the need for this feature. Perhaps you could respond and elaborate on your motivational use-cases. [~balaprassanna] Or if its easier you could elaborate the use-cases here and I could post on the dev list. Its important to clearly establish the use-cases in order to gain support for this feature. was (Author: nafshartous): [~balaprassanna] There is a question today in the dev list, see subject [DISCUSS] KIP-349 Priorities for Source Topics about the use-cases and the need for this feature. Perhaps you could respond and elaborate on your motivational use-cases. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597433#comment-16597433 ] Matthias J. Sax edited comment on KAFKA-6690 at 8/30/18 1:21 PM: - [~nafshartous] It's your KIP and thus your decision if you want to follow this suggesting or not. It would be good to understand the advantages and disadvantage of your current (simplified?) proposal compared to the suggestion. What you can do/not do with the or the other? Would it be possible to extend your current proposal with something more generic later on? You can also just continue the VOTE if you are not interested to change the proposal. A single non-binding downvote does not block the KIP. I will also follow up on the mailing list. Note, that anybody can do a KIP and she/he is in the "driver seat" to lead the discussion. Thus, you can agree with any other proposal and change your KIP or argue against it (why do you think it's not necessary etc.). In the end, if you can convince the community that your KIP is valuable you will get it through. Just keep discussion and sharing your point of view on the mailing list, why you want to do what, and lay out the advantages/disadvantages compare to alternative proposals. was (Author: mjsax): [~nafshartous] It's your KIP and thus your decision if you want to follow this suggesting or not. It would be good to understand the advantages and disadvantage of your current (simplified?) proposal compared to the suggestion. What you can do/not do with the or the other? Would it be possible to extend your current proposal with something more generic later on? You can also just continue the VOTE if you are not interested to change the proposal. A single non-binding downvote does not block the KIP. I will also follow up on the mailing list. > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564413#comment-16564413 ] Nick Afshartous edited comment on KAFKA-6690 at 7/31/18 10:00 PM: -- [~balaprassanna] Please review before I take this to the dev mailing list for discussion https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics was (Author: nafshartous): [~balaprassanna] Please review before I take this to the dev mailing list for discussion https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics#KIP-349:PrioritiesforSourceTopics-Motivation > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559356#comment-16559356 ] Bala Prassanna I edited comment on KAFKA-6690 at 7/27/18 7:08 AM: -- [~guozhang] We would need this in both Consumer API and Streams API was (Author: balaprassanna): [~guozhang] We would need this in both Consumer API and also in Streams API > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528109#comment-16528109
]
Nick Afshartous edited comment on KAFKA-6690 at 6/29/18 7:24 PM:
-
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPriority(java.lang.String topic, int priority)
{code}
then there would be new {{KafkaConsumer.subscribe}} methods like
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture the intent ?
was (Author: nafshartous):
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPriority(java.lang.String topic, int priority)
{code}
then there would be a new {{KafkaConsumer.subscribe}} method
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture the intent ?
> Priorities for Source Topics
>
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bala Prassanna I
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528109#comment-16528109
]
Nick Afshartous edited comment on KAFKA-6690 at 6/29/18 7:16 PM:
-
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPriority(java.lang.String topic, int priority)
{code}
then there would be a new {{KafkaConsumer.subscribe}} method
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture then intent ?
was (Author: nafshartous):
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPrioritiy(java.lang.String topic, int priority)
{code}
then there would be a new {{KafkaConsumer.subscribe}} method
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture then intent ?
> Priorities for Source Topics
>
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bala Prassanna I
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528109#comment-16528109
]
Nick Afshartous edited comment on KAFKA-6690 at 6/29/18 7:16 PM:
-
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPriority(java.lang.String topic, int priority)
{code}
then there would be a new {{KafkaConsumer.subscribe}} method
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture the intent ?
was (Author: nafshartous):
This seems like it would require an API change. For example, if we added a new
class
{code}
TopicPriority(java.lang.String topic, int priority)
{code}
then there would be a new {{KafkaConsumer.subscribe}} method
{code}
public void subscribe(java.util.List topicPriorities)
{code}
Doe this capture then intent ?
> Priorities for Source Topics
>
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bala Prassanna I
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a
> consumer is listening more than one topic, say, HighPriorityTopic and
> LowPriorityTopic, it should consume events from LowPriorityTopic only when
> all the events from HighPriorityTopic are consumed. This is needed in Kafka
> Streams processor topologies as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
