[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics

2023-09-30 Thread Domenico Di Giulio (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770704#comment-17770704
 ] 

Domenico Di Giulio edited comment on KAFKA-6690 at 9/30/23 12:54 PM:
-

I see this issue was closed but we also really need it A LOT.

In our case two co-partitioned source topics are used in the same topology, one 
(let's call it topic "A") to accept new business requests, the other (let's 
call it topic "B") as a loopback (from downstream components) about business 
transactions already in progress, which come back to the same topology to 
update a transaction status (in a store) and proceed.

We found that when the event rate on topic A is too high, Kafka Streams has no 
way to apply any backpressure on it and favor topic B. As a result, the 
processing on topic B progressively goes in starvation and the system proceeds 
much slower than it could. We're looking at how we could apply a backpressure 
on topic A, fetching more events from topic B so that the processing of ongoing 
business transactions can proceed with higher priority.

So far we only thought about adding another upstream component (may be just a 
simple consumer and producer), which should read topic A and forward events to 
our topology based on a metric which looks for symptoms of congestion (business 
latency or whatever). But it would be very beneficial instead to just define B 
as a priority topic on Kafka Streams.

Any news on this issue ? Does the above use case look meaningful ?

 


was (Author: domenico74):
I see this issue was closed but we also really need it A LOT.

In our case two co-partitioned source topics are used in the same topology, one 
(let's call it topic "A") to accept new business requests, the other (let's 
call it topic "B") as a loopback (from downstream components) about business 
transactions already in progress, which come back to the same topology to 
update a transaction status (in a store) and proceed.

We found that when the event rate on topic A is too high, Kafka Streams has no 
way to apply any backpressure on it and favor topic B. As a result, the 
processing on topic B progressively goes in starvation and the system proceeds 
much slower than it could. We're looking at how we could apply a backpressure 
on topic A, fetching more events from topic B so that the processing of ongoing 
business transactions can proceed with higher priority.

So far we only thought about adding another upstream component (may be just a 
simple consumer and producer), which should read topic A and forward events to 
our topology based on a metric which looks for symptoms of congestion (business 
latency or whatever). But it would be very beneficial instead to just define B 
as a priority topic on Kafka Streams.

Any news on this KIP ? Does the above use case look meaningful ?

 

> Priorities for Source Topics
> 
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer
>Reporter: Bala Prassanna I
>Assignee: Nick Afshartous
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a 
> consumer is listening more than one topic, say, HighPriorityTopic and 
> LowPriorityTopic, it should consume events from LowPriorityTopic only when 
> all the events from HighPriorityTopic are consumed. This is needed in Kafka 
> Streams processor topologies as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (KAFKA-6690) Priorities for Source Topics

2023-09-30 Thread Domenico Di Giulio (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770704#comment-17770704
 ] 

Domenico Di Giulio commented on KAFKA-6690:
---

I see this issue was closed but we also really need it A LOT.

In our case two co-partitioned source topics are used in the same topology, one 
(let's call it topic "A") to accept new business requests, the other (let's 
call it topic "B") as a loopback (from downstream components) about business 
transactions already in progress, which come back to the same topology to 
update a transaction status (in a store) and proceed.

We found that when the event rate on topic A is too high, Kafka Streams has no 
way to apply any backpressure on it and favor topic B. As a result, the 
processing on topic B progressively goes in starvation and the system proceeds 
much slower than it could. We're looking at how we could apply a backpressure 
on topic A, fetching more events from topic B so that the processing of ongoing 
business transactions can proceed with higher priority.

So far we only thought about adding another upstream component (may be just a 
simple consumer and producer), which should read topic A and forward events to 
our topology based on a metric which looks for symptoms of congestion (business 
latency or whatever). But it would be very beneficial instead to just define B 
as a priority topic on Kafka Streams.

Any news on this KIP ? Does the above use case look meaningful ?

 

> Priorities for Source Topics
> 
>
> Key: KAFKA-6690
> URL: https://issues.apache.org/jira/browse/KAFKA-6690
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer
>Reporter: Bala Prassanna I
>Assignee: Nick Afshartous
>Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a 
> consumer is listening more than one topic, say, HighPriorityTopic and 
> LowPriorityTopic, it should consume events from LowPriorityTopic only when 
> all the events from HighPriorityTopic are consumed. This is needed in Kafka 
> Streams processor topologies as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Issue Comment Deleted] (KAFKA-5016) Consumer hang in poll method while rebalancing is in progress

2017-07-27 Thread Domenico Di Giulio (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Domenico Di Giulio updated KAFKA-5016:
--
Comment: was deleted

(was: I am currently out of the office, with no access to my e-mail.
I will be back at work on July 27.

** Le e-mail provenienti dalla Banca d'Italia sono trasmesse in buona fede e 
non comportano alcun vincolo nè creano obblighi per la Banca stessa, salvo che 
ciò non sia espressamente previsto da un accordo scritto. Questa e-mail è 
confidenziale. Qualora l'avesse ricevuta per errore, La preghiamo di 
comunicarne via e-mail la ricezione al mittente e di distruggere il contenuto. 
La informiamo inoltre che l'utilizzo non autorizzato del messaggio o dei suoi 
allegati potrebbe costituire reato. Grazie per la collaborazione.
-- E-mail from Bank of Italy are sent in good faith but they are neither 
binding on the Bank nor to be understood as creating any obligation on its part 
except where provided for in a written agreement. This e-mail is confidential. 
If you have received it by mistake, please inform the sender by reply e-mail 
and delete it from your system. Please also note that the unauthorized 
disclosure or use of the message or any attachments could be an offence. Thank 
you for your cooperation. **
)

> Consumer hang in poll method while rebalancing is in progress
> -
>
> Key: KAFKA-5016
> URL: https://issues.apache.org/jira/browse/KAFKA-5016
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0, 0.10.2.0
>Reporter: Domenico Di Giulio
>Assignee: Vahid Hashemian
> Attachments: Kafka 0.10.2.0 Issue (TRACE) - Server + Client.txt, 
> Kafka 0.10.2.0 Issue (TRACE).txt, KAFKA_5016.java
>
>
> After moving to Kafka 0.10.2.0, it looks like I'm experiencing a hang in the 
> rebalancing code. 
> This is a test case, not (still) production code. It does the following with 
> a single-partition topic and two consumers in the same group:
> 1) a topic with one partition is forced to be created (auto-created)
> 2) a producer is used to write 10 messages
> 3) the first consumer reads all the messages and commits
> 4) the second consumer attempts a poll() and hangs indefinitely
> The same issue can't be found with 0.10.0.0.
> See the attached logs at TRACE level. Look for "SERVER HANGS" to see where 
> the hang is found: when this happens, the client keeps failing any hearbeat 
> attempt, as the rebalancing is in progress, and the poll method hangs 
> indefinitely.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (KAFKA-5016) Consumer hang in poll method while rebalancing is in progress

2017-07-14 Thread Domenico Di Giulio (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088288#comment-16088288
 ] 

Domenico Di Giulio commented on KAFKA-5016:
---

I am currently out of the office, with no access to my e-mail.
I will be back at work on July 27.

** Le e-mail provenienti dalla Banca d'Italia sono trasmesse in buona fede e 
non comportano alcun vincolo nè creano obblighi per la Banca stessa, salvo che 
ciò non sia espressamente previsto da un accordo scritto. Questa e-mail è 
confidenziale. Qualora l'avesse ricevuta per errore, La preghiamo di 
comunicarne via e-mail la ricezione al mittente e di distruggere il contenuto. 
La informiamo inoltre che l'utilizzo non autorizzato del messaggio o dei suoi 
allegati potrebbe costituire reato. Grazie per la collaborazione.
-- E-mail from Bank of Italy are sent in good faith but they are neither 
binding on the Bank nor to be understood as creating any obligation on its part 
except where provided for in a written agreement. This e-mail is confidential. 
If you have received it by mistake, please inform the sender by reply e-mail 
and delete it from your system. Please also note that the unauthorized 
disclosure or use of the message or any attachments could be an offence. Thank 
you for your cooperation. **


> Consumer hang in poll method while rebalancing is in progress
> -
>
> Key: KAFKA-5016
> URL: https://issues.apache.org/jira/browse/KAFKA-5016
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0, 0.10.2.0
>Reporter: Domenico Di Giulio
>Assignee: Vahid Hashemian
> Attachments: Kafka 0.10.2.0 Issue (TRACE) - Server + Client.txt, 
> Kafka 0.10.2.0 Issue (TRACE).txt, KAFKA_5016.java
>
>
> After moving to Kafka 0.10.2.0, it looks like I'm experiencing a hang in the 
> rebalancing code. 
> This is a test case, not (still) production code. It does the following with 
> a single-partition topic and two consumers in the same group:
> 1) a topic with one partition is forced to be created (auto-created)
> 2) a producer is used to write 10 messages
> 3) the first consumer reads all the messages and commits
> 4) the second consumer attempts a poll() and hangs indefinitely
> The same issue can't be found with 0.10.0.0.
> See the attached logs at TRACE level. Look for "SERVER HANGS" to see where 
> the hang is found: when this happens, the client keeps failing any hearbeat 
> attempt, as the rebalancing is in progress, and the poll method hangs 
> indefinitely.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)