Re: [akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Yaroslav Klymko]

Hi Patrik,

Yes that was an idea and the Future passed would be of something 
like 
https://github.com/evolution-gaming/akka-tools/blob/master/src/main/scala/com/evolutiongaming/util/FutureSequentialForKey.scala#L17

In general the goal is to make sure large messages do not affect other 
small messages. So having custom `lane` for large and your default for 
small msgs could satisfy my needs. 
Do you have an own idea on how we can achieve this goal?



On Tuesday, May 2, 2017 at 12:56:23 PM UTC+3, Patrik Nordwall wrote:
>
> Are you thinking of adding mapAsync to the Artery lanes? It could be 
> interesting to look at a suggestion for that. Adding more async boundaries 
> will be worse for latency of normal messages, so it should not be the 
> default.
>
> Also note that compression and serialization is CPU bound tasks so if you 
> try to do more than you have cores the performance will only be worse. You 
> also want to leave some cores for other tasks in your application, such as 
> processing the actual messages.
>
> /Patrik
>
>
>
> On Mon, May 1, 2017 at 8:30 PM, Yaroslav Klymko  > wrote:
>
>> I'm looking further in order to understand on how artery fixes it and 
>> truly saying I'm not in favour of configurable number of inbound-lanes and 
>> outbound-lanes.
>>
>> Right now we have an issue with passing messages in cluster. Sometimes we 
>> want to send large messages and make sure those are compressed to minimise 
>> network use. However adding compression will have negative impact for other 
>> messages that have to share same single actor. So I have an idea to 
>> basically parallelize serialisation + compression for messages that for 
>> instance are bigger then some configurable size and use `recipient` as key 
>> for parallelisation, this way I will be able to preserver order.
>>
>> Question: 
>> * Is akka team interested in such idea and will implement solution? 
>> * Interested and will accept the pull request ? 
>> * Interested in pull request that will expose dispatching api for custom 
>> implementation?
>> * None of the above
>>
>>
>>
>>
>> On Friday, April 28, 2017 at 6:32:43 PM UTC+3, Akka Team wrote:
>>>
>>> That is correct, there is one writer and one reader per connection in 
>>> the stable remoting, so if serialization is slow/heavy that may be a 
>>> bottleneck.
>>>
>>> In the new remoting, "artery", we have made it possible to run multiple 
>>> in and outbound "lanes", but that functionality is not yet hardened enough 
>>> to use.
>>> You can see the settings in reference.conf here: 
>>> http://doc.akka.io/docs/akka/2.5.0/general/configuration.html#config-akka-remote-artery
>>>  
>>> - inbound-lanes and outbound-lanes.
>>>
>>> --
>>> Johan
>>> Akka Team
>>>
>>> On Fri, Apr 28, 2017 at 12:45 PM, Yaroslav Klymko  
>>> wrote:
>>>
 Hi guys,

 Do I understand correctly that we have a single instance of 
 `EndpointReader` actor per connection ? 
 Thus deserialisation of remote messages being done sequentially, so we 
 can hit this bottleneck much faster then a network throughput limits?

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.
>>>
>>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
>
> Patrik Nordwall
> Akka Tech Lead
> Lightbend  -  Reactive apps on the JVM
> Twitter: @patriknw
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and 

Re: [akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Patrik Nordwall]

Are you thinking of adding mapAsync to the Artery lanes? It could be
interesting to look at a suggestion for that. Adding more async boundaries
will be worse for latency of normal messages, so it should not be the
default.

Also note that compression and serialization is CPU bound tasks so if you
try to do more than you have cores the performance will only be worse. You
also want to leave some cores for other tasks in your application, such as
processing the actual messages.

/Patrik



On Mon, May 1, 2017 at 8:30 PM, Yaroslav Klymko  wrote:

> I'm looking further in order to understand on how artery fixes it and
> truly saying I'm not in favour of configurable number of inbound-lanes and
> outbound-lanes.
>
> Right now we have an issue with passing messages in cluster. Sometimes we
> want to send large messages and make sure those are compressed to minimise
> network use. However adding compression will have negative impact for other
> messages that have to share same single actor. So I have an idea to
> basically parallelize serialisation + compression for messages that for
> instance are bigger then some configurable size and use `recipient` as key
> for parallelisation, this way I will be able to preserver order.
>
> Question:
> * Is akka team interested in such idea and will implement solution?
> * Interested and will accept the pull request ?
> * Interested in pull request that will expose dispatching api for custom
> implementation?
> * None of the above
>
>
>
>
> On Friday, April 28, 2017 at 6:32:43 PM UTC+3, Akka Team wrote:
>>
>> That is correct, there is one writer and one reader per connection in the
>> stable remoting, so if serialization is slow/heavy that may be a bottleneck.
>>
>> In the new remoting, "artery", we have made it possible to run multiple
>> in and outbound "lanes", but that functionality is not yet hardened enough
>> to use.
>> You can see the settings in reference.conf here: http://doc.akka.io/docs/
>> akka/2.5.0/general/configuration.html#config-akka-remote-artery -
>> inbound-lanes and outbound-lanes.
>>
>> --
>> Johan
>> Akka Team
>>
>> On Fri, Apr 28, 2017 at 12:45 PM, Yaroslav Klymko 
>> wrote:
>>
>>> Hi guys,
>>>
>>> Do I understand correctly that we have a single instance of
>>> `EndpointReader` actor per connection ?
>>> Thus deserialisation of remote messages being done sequentially, so we
>>> can hit this bottleneck much faster then a network throughput limits?
>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>>> urrent/additional/faq.html
>>> >> Search the archives: https://groups.google.com/grou
>>> p/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 

Patrik Nordwall
Akka Tech Lead
Lightbend  -  Reactive apps on the JVM
Twitter: @patriknw

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Yaroslav Klymko]

I'm looking further in order to understand on how artery fixes it and truly 
saying I'm not in favour of configurable number of inbound-lanes and 
outbound-lanes.

Right now we have an issue with passing messages in cluster. Sometimes we 
want to send large messages and make sure those are compressed to minimise 
network use. However adding compression will have negative impact for other 
messages that have to share same single actor. So I have an idea to 
basically parallelize serialisation + compression for messages that for 
instance are bigger then some configurable size and use `recipient` as key 
for parallelisation, this way I will be able to preserver order.

Question: 
* Is akka team interested in such idea and will implement solution? 
* Interested and will accept the pull request ? 
* Interested in pull request that will expose dispatching api for custom 
implementation?
* None of the above




On Friday, April 28, 2017 at 6:32:43 PM UTC+3, Akka Team wrote:
>
> That is correct, there is one writer and one reader per connection in the 
> stable remoting, so if serialization is slow/heavy that may be a bottleneck.
>
> In the new remoting, "artery", we have made it possible to run multiple in 
> and outbound "lanes", but that functionality is not yet hardened enough to 
> use.
> You can see the settings in reference.conf here: 
> http://doc.akka.io/docs/akka/2.5.0/general/configuration.html#config-akka-remote-artery
>  
> - inbound-lanes and outbound-lanes.
>
> --
> Johan
> Akka Team
>
> On Fri, Apr 28, 2017 at 12:45 PM, Yaroslav Klymko  > wrote:
>
>> Hi guys,
>>
>> Do I understand correctly that we have a single instance of 
>> `EndpointReader` actor per connection ? 
>> Thus deserialisation of remote messages being done sequentially, so we 
>> can hit this bottleneck much faster then a network throughput limits?
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Akka Team]

That is correct, there is one writer and one reader per connection in the
stable remoting, so if serialization is slow/heavy that may be a bottleneck.

In the new remoting, "artery", we have made it possible to run multiple in
and outbound "lanes", but that functionality is not yet hardened enough to
use.
You can see the settings in reference.conf here:
http://doc.akka.io/docs/akka/2.5.0/general/configuration.html#config-akka-remote-artery
- inbound-lanes and outbound-lanes.

--
Johan
Akka Team

On Fri, Apr 28, 2017 at 12:45 PM, Yaroslav Klymko  wrote:

> Hi guys,
>
> Do I understand correctly that we have a single instance of
> `EndpointReader` actor per connection ?
> Thus deserialisation of remote messages being done sequentially, so we can
> hit this bottleneck much faster then a network throughput limits?
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Yaroslav Klymko]

Hi guys,

Do I understand correctly that we have a single instance of 
`EndpointReader` actor per connection ? 
Thus deserialisation of remote messages being done sequentially, so we can 
hit this bottleneck much faster then a network throughput limits?

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.