Re: [akka-user] Slow reactive-kafka consumer when running multiple streams

[Kilic Ali-Firat]

Hi, 

I did a lot of tried and in my case, the bottleneckness was the 
akka-streams-kafka version. It was not aligned to my kafka server version. 

You have a look also to field "receive.buffer.bytes" parameter, make some 
tests with differents values.

You also can have a look to your CPUs consumption and hardware use for your 
streams (maybe you have reach bandwith of your machine?)

Le mardi 13 février 2018 08:47:49 UTC+1, Tal Pressman a écrit :
>
> Hi,
>
> We have several streams in the same runtime and it works fine. We did, 
> however, have problems when trying _a lot_ of consumers (~100, iirc), and 
> ended up sharing.
>
> Tal
>
>
> On Tuesday, February 13, 2018 at 1:37:24 AM UTC+2, rrodseth wrote:
>>
>> Anyone else running more than one stream in a deployable? Surely the 
>> answer must be yes. Would love to know what the problem might be.
>>
>>
>> On Wed, Dec 27, 2017 at 10:02 AM, Harshit Patel  
>> wrote:
>>
>>> We have an application that uses reactive-kafka to consume messages from 
>>> a kafka topic, do some validations and transformations on the messages, and 
>>> then persist to two different datastores. We are using two different 
>>> reactive kafka consumer streams for each of the datastores, using different 
>>> consumer-group-ids. Each of the stream can be enabled/disabled via 
>>> configuration in the application. 
>>>
>>> When we run either one of the consumer streams, they seem to run fine 
>>> and are able to consume messages and keep up with the topic reasonably 
>>> well. But when both streams are run together, we see a pretty high latency 
>>> (4-5 minutes) and both consumers are unable to keep up with the messages 
>>> from the topic. We are publishing about 15 messages/minute to the 
>>> topic. We are making sure that both consumers are managing blocking where 
>>> applicable, and are using separate dedicated dispatchers for the same. 
>>>
>>> Any ideas what could be going on ?
>>>
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: 
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives: 
>>> https://groups.google.com/group/akka-user
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Slow reactive-kafka consumer when running multiple streams

[Tal Pressman]

Hi,

We have several streams in the same runtime and it works fine. We did, 
however, have problems when trying _a lot_ of consumers (~100, iirc), and 
ended up sharing.

Tal


On Tuesday, February 13, 2018 at 1:37:24 AM UTC+2, rrodseth wrote:
>
> Anyone else running more than one stream in a deployable? Surely the 
> answer must be yes. Would love to know what the problem might be.
>
>
> On Wed, Dec 27, 2017 at 10:02 AM, Harshit Patel  > wrote:
>
>> We have an application that uses reactive-kafka to consume messages from 
>> a kafka topic, do some validations and transformations on the messages, and 
>> then persist to two different datastores. We are using two different 
>> reactive kafka consumer streams for each of the datastores, using different 
>> consumer-group-ids. Each of the stream can be enabled/disabled via 
>> configuration in the application. 
>>
>> When we run either one of the consumer streams, they seem to run fine and 
>> are able to consume messages and keep up with the topic reasonably well. 
>> But when both streams are run together, we see a pretty high latency (4-5 
>> minutes) and both consumers are unable to keep up with the messages from 
>> the topic. We are publishing about 15 messages/minute to the topic. We 
>> are making sure that both consumers are managing blocking where applicable, 
>> and are using separate dedicated dispatchers for the same. 
>>
>> Any ideas what could be going on ?
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Slow reactive-kafka consumer when running multiple streams

[Richard Rodseth]

Anyone else running more than one stream in a deployable? Surely the answer
must be yes. Would love to know what the problem might be.


On Wed, Dec 27, 2017 at 10:02 AM, Harshit Patel 
wrote:

> We have an application that uses reactive-kafka to consume messages from a
> kafka topic, do some validations and transformations on the messages, and
> then persist to two different datastores. We are using two different
> reactive kafka consumer streams for each of the datastores, using different
> consumer-group-ids. Each of the stream can be enabled/disabled via
> configuration in the application.
>
> When we run either one of the consumer streams, they seem to run fine and
> are able to consume messages and keep up with the topic reasonably well.
> But when both streams are run together, we see a pretty high latency (4-5
> minutes) and both consumers are unable to keep up with the messages from
> the topic. We are publishing about 15 messages/minute to the topic. We
> are making sure that both consumers are managing blocking where applicable,
> and are using separate dedicated dispatchers for the same.
>
> Any ideas what could be going on ?
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.