Re: [akka-user] Role of Akka in SMACK

[Roland Kuhn]

Whoever this is: would you please abstain from such anonymous unsubstantiated 
claims? I cannot comment on the truthfulness of your critique, common courtesy 
would entail that I know at least your name to ask for supporting evidence.

Regards, Roland 

Sent from my iPhone

> On 3. May 2017, at 06:36, 'Ryan Tanner' via Akka User List 
>  wrote:
> 
> Be careful with Flink.  IME it's got a long way to go.  The core model is 
> fantastic but there's a lot of low-hanging fruit that needs to be fixed.
> 
>> On Tuesday, May 2, 2017 at 4:03:12 PM UTC-6, Evan Chan wrote:
>> Hi Shiva,
>> 
>> Spark will likely be too high latency for you.  Practical minimal batch size 
>> is a couple seconds.
>> 
>> Think of Akka as a best fit for if you want to deploy individual apps each 
>> reading from some set of fixed Kafka partitions.  Each one could then write 
>> to HDFS.  However you would need to handle failover, state, etc. etc.
>> It is likely that Flink has much more built in for you - HDFS integration, 
>> checkpointing, failover, shuffling/routing of messages, integration with 
>> Kafka, etc.
>> 
>> You might want to look into Intel Gearpump - this is Akka-based very low 
>> latency dynamic stream processing, and they have handled distribution 
>> already.
>> 
>> -Evan
>> 
>>> On Friday, April 28, 2017 at 7:52:50 AM UTC-7, Shivakumar Ramagopal wrote:
>>> Viktor,
>>> 
>>> 
 On Fri, Apr 28, 2017 at 5:03 PM, Viktor Klang  wrote:
 
 
> On Fri, Apr 28, 2017 at 1:12 PM, Shiva Ramagopal  wrote:
> Hi Viktor,
> 
>> On Fri, Apr 28, 2017 at 2:55 PM, Viktor Klang  
>> wrote:
>> Hi Shiva,
>> 
>>> On Fri, Apr 28, 2017 at 11:20 AM, Shiva Ramagopal  
>>> wrote:
>>> I'm looking to compare Kafka Streams vs Akka Streams in two areas: 
>>> 
>>> 1. For ingesting between Kafka and HDFS/RDBMS 
>>> 
>>> Requirements are mainly around performance and latency. A Kafka topic 
>>> can have several million events, each corresponding to a database 
>>> change capture. When ingesting this topic into HDFS I'm also looking to 
>>> partition the data by day, typically based on a timestamp field in the 
>>> event record, aggregations on-the-fly (say by a userid field) and 
>>> parquetizing (preferably on the fly to optimize performance by reducing 
>>> two I/O operations).
>> 
>> Looking forward to see your benchmark!
> 
> Hey, you wanted requirements! :) 
>>  
>>> 
>>> 2. Low-latency processing
>>> 
>>> Experiences around performances of Storm/Flink and Akka Streams would 
>>> be *very* nice. Typical use-cases are de-duping and enrichment with 
>>> metrics computation (# duplicate events/records, aggregate metrics 
>>> etc). Low latency and scalability are main considerations
>> 
>> Low latency is not a metric, and scalability is not a profile. :)
>> In other words: What latency distribution are you targeting and what 
>> scalability curve?
> 
> I know latency is a nuanced topic. I'm just looking for broad experiences 
> on performance comparisons if anyone has done that. Currently we have a 
> Storm topology over 5 nodes doing enrichment of events from Kafka that 
> involves a lookup into a db per event. 90th percentile of latency of this 
> processing is under 200 ms and we are happy with this. While Storm is 
> mature, Akka Streams seems more general purpose than Storm. I'd like to 
> use Akka Streams for this reason if performance is comparable to Storm. 
 
 There's one important architectural difference here tho: Akka Streams are 
 local-only (as in materialization). You can of course materialize Akka 
 Streams on multiple nodes and use a transport to coordinate data 
 processing. Interestingly it doesn't lock you in to a particular backend 
 such that Kafka Streams or even Storm would do.
>>> 
>>> The materialization part was something I was not aware of. Thanks for 
>>> pointing it out, really appreciate it.
 
 All of this boils down to requirements. Something like Flink or Google 
 Beam could be viable options here as well. In your situation I'd look at 
 the requirements and make a couple of prototypes before picking a winner.
>>> 
>>> I was also looking for something to unify batch and streaming - which is 
>>> how I came to look at SMACK. I'd really like to keep the tech stack small 
>>> and have the parts integrate tightly with each other. Guess I have to pick 
>>> between Spark, Flink, Beam and Storm. 
  
>  
>>  
>>> 
 On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang  
 wrote:
 
 
> On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal  
> wrote:
> Hi,
> 
> I have read through multiple 

Re: [akka-user] Role of Akka in SMACK

['Ryan Tanner' via Akka User List]

Be careful with Flink.  IME it's got a long way to go.  The core model is 
fantastic but there's a lot of low-hanging fruit that needs to be fixed.

On Tuesday, May 2, 2017 at 4:03:12 PM UTC-6, Evan Chan wrote:
>
> Hi Shiva,
>
> Spark will likely be too high latency for you.  Practical minimal batch 
> size is a couple seconds.
>
> Think of Akka as a best fit for if you want to deploy individual apps each 
> reading from some set of fixed Kafka partitions.  Each one could then write 
> to HDFS.  However you would need to handle failover, state, etc. etc.
> It is likely that Flink has much more built in for you - HDFS integration, 
> checkpointing, failover, shuffling/routing of messages, integration with 
> Kafka, etc.
>
> You might want to look into Intel Gearpump - this is Akka-based very low 
> latency dynamic stream processing, and they have handled distribution 
> already.
>
> -Evan
>
> On Friday, April 28, 2017 at 7:52:50 AM UTC-7, Shivakumar Ramagopal wrote:
>>
>> Viktor,
>>
>>
>> On Fri, Apr 28, 2017 at 5:03 PM, Viktor Klang  
>> wrote:
>>
>>>
>>>
>>> On Fri, Apr 28, 2017 at 1:12 PM, Shiva Ramagopal  
>>> wrote:
>>>
 Hi Viktor,

 On Fri, Apr 28, 2017 at 2:55 PM, Viktor Klang  
 wrote:

> Hi Shiva,
>
> On Fri, Apr 28, 2017 at 11:20 AM, Shiva Ramagopal  
> wrote:
>
>> I'm looking to compare Kafka Streams vs Akka Streams in two areas: 
>>
>> 1. For ingesting between Kafka and HDFS/RDBMS 
>>
>> Requirements are mainly around performance and latency. A Kafka topic 
>> can have several million events, each corresponding to a database change 
>> capture. When ingesting this topic into HDFS I'm also looking to 
>> partition 
>> the data by day, typically based on a timestamp field in the event 
>> record, 
>> aggregations on-the-fly (say by a userid field) and parquetizing 
>> (preferably on the fly to optimize performance by reducing two I/O 
>> operations).
>>
>
> Looking forward to see your benchmark!
>

 Hey, you wanted requirements! :) 

>  
>
>>
>> 2. Low-latency processing
>>
>> Experiences around performances of Storm/Flink and Akka Streams would 
>> be *very* nice. Typical use-cases are de-duping and enrichment with 
>> metrics 
>> computation (# duplicate events/records, aggregate metrics etc). Low 
>> latency and scalability are main considerations
>>
>
> Low latency is not a metric, and scalability is not a profile. :)
> In other words: What latency distribution are you targeting and what 
> scalability curve?
>

 I know latency is a nuanced topic. I'm just looking for broad 
 experiences on performance comparisons if anyone has done that. Currently 
 we have a Storm topology over 5 nodes doing enrichment of events from 
 Kafka 
 that involves a lookup into a db per event. 90th percentile of latency of 
 this processing is under 200 ms and we are happy with this. While Storm is 
 mature, Akka Streams seems more general purpose than Storm. I'd like to 
 use 
 Akka Streams for this reason if performance is comparable to Storm. 

>>>
>>> There's one important architectural difference here tho: Akka Streams 
>>> are local-only (as in materialization). You can of course materialize Akka 
>>> Streams on multiple nodes and use a transport to coordinate data 
>>> processing. Interestingly it doesn't lock you in to a particular backend 
>>> such that Kafka Streams or even Storm would do.
>>>
>>
>> The materialization part was something I was not aware of. Thanks for 
>> pointing it out, really appreciate it.
>>
>>>
>>> All of this boils down to requirements. Something like Flink or Google 
>>> Beam could be viable options here as well. In your situation I'd look at 
>>> the requirements and make a couple of prototypes before picking a winner.
>>>
>>
>> I was also looking for something to unify batch and streaming - which is 
>> how I came to look at SMACK. I'd really like to keep the tech stack small 
>> and have the parts integrate tightly with each other. Guess I have to pick 
>> between Spark, Flink, Beam and Storm. 
>>
>>>  
>>>
  

>  
>
>>
>> On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang  
>> wrote:
>>
>>>
>>>
>>> On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal  
>>> wrote:
>>>
 Hi,

 I have read through multiple articles describing the SMACK stack 
 but I'm having difficulty understanding the role of Akka in the stack. 
 How 
 does Akka fit in?

>>>
>>> Akka is for building the application itself.
>>>  
>>>

 Also I would like to know of experiences in using Akka Streams vs 
 Kafka Connect for ingesting from 

Re: [akka-user] Akka-streams supervision: best practices

[Konrad Malawski]

Hi Alexander,
since you asked the same question on the issue tracker at the same time
allow me to link those and let's continue the discussion here if you want;

https://github.com/akka/akka/issues/21192#issuecomment-298806620


Firstly: I don't think that's entirely true. Existing supervision indeed
only works when stages "support it". One has to notice that it's not as
simple as it may seem from a high level perspective, since there's buffers
involved between stages so what would "restart a section", or what *should*
 it *really* mean etc.

However, there's many ways to add restarting or other behaviours.

Including, but not limited to:

   - Retry -
   
https://github.com/akka/akka-stream-contrib/blob/master/contrib/src/main/scala/akka/stream/contrib/Retry.scala
   - recoverWith
   
http://doc.akka.io/docs/akka/2.5.1/scala/stream/stages-overview.html#recoverWith
   - recoverWithRetries
   
http://doc.akka.io/docs/akka/2.5.1/scala/stream/stages-overview.html#recoverWithRetries
   - mapError
   http://doc.akka.io/docs/akka/2.5.1/scala/stream/stages-overview.html#mapError
   - and more stages, such as watchTermination, monitor etc.
   - the existing supervision when it's enough
   - or hosting within Actors and reacting to a termination with how you
   want.

Do bear in mind that Akka was, is and remains a toolkit. The tools are
there, with killswitches, a spec how failure is propagated (reactive
streams, which will be in JDK9)

Would we want to improve the current supervision scheme?
Yes, we'd like to. Are we right now going to work on it? No, we're focused
on other areas right now that are more pressing.

What would be helpful now: What I think would be the best next step here,
is to actually *come up with patterns* - be it as blogs, or writeups in
issues or PRs that pre-package behaviour such as the Retry stage did, and
once we have the real use cases we can think about it again, because a
*generic* "supervision" is very hard (note the generic part in that phrase,
specialized ones are simple), and *very* likely not what you'd actually
want (it would do unexpected surprising things, we experimented with some
restarting schemes).

I do agree though that all those patterns should be then collected and put
to: http://doc.akka.io/docs/akka/2.5.1/scala/stream/stream-error.html We're
right now working on a new docs and website page, so once that's done we'll
focus on such content.

Hope this answers the question about status, next steps, and how you can
help :-)



-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 

On 3 May 2017 at 08:20:50, Alexander Temerev (sor...@gmail.com) wrote:

Hi everyone,

Is there any example or document describing best practices for supervising
and monitoring akka streams? E.g. restarting Websocket streams on
disconnect, backing off on repeated failures, creating supervision groups
(e.g. if some stream failed, I also want to fail other streams in the group
and restart them at once), making supervision hierarchies, etc.

Streams are incredibly handy and well-typed, but I don't see yet how I can
recreate these powerful supervision tools available in plain Akka,
especially in dynamic flow graphs where streams can be attached and
detached at runtime.

"Introduction to streams lifecycle" would be incredibly handy, if it exists
somewhere.

Best,
Alexander
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Akka-streams supervision: best practices

[Alexander Temerev]

Hi everyone,

Is there any example or document describing best practices for supervising 
and monitoring akka streams? E.g. restarting Websocket streams on 
disconnect, backing off on repeated failures, creating supervision groups 
(e.g. if some stream failed, I also want to fail other streams in the group 
and restart them at once), making supervision hierarchies, etc.

Streams are incredibly handy and well-typed, but I don't see yet how I can 
recreate these powerful supervision tools available in plain Akka, 
especially in dynamic flow graphs where streams can be attached and 
detached at runtime.

"Introduction to streams lifecycle" would be incredibly handy, if it exists 
somewhere.

Best,
Alexander

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Role of Akka in SMACK

[Evan Chan]

Hi Shiva,

Spark will likely be too high latency for you.  Practical minimal batch 
size is a couple seconds.

Think of Akka as a best fit for if you want to deploy individual apps each 
reading from some set of fixed Kafka partitions.  Each one could then write 
to HDFS.  However you would need to handle failover, state, etc. etc.
It is likely that Flink has much more built in for you - HDFS integration, 
checkpointing, failover, shuffling/routing of messages, integration with 
Kafka, etc.

You might want to look into Intel Gearpump - this is Akka-based very low 
latency dynamic stream processing, and they have handled distribution 
already.

-Evan

On Friday, April 28, 2017 at 7:52:50 AM UTC-7, Shivakumar Ramagopal wrote:
>
> Viktor,
>
>
> On Fri, Apr 28, 2017 at 5:03 PM, Viktor Klang  > wrote:
>
>>
>>
>> On Fri, Apr 28, 2017 at 1:12 PM, Shiva Ramagopal > > wrote:
>>
>>> Hi Viktor,
>>>
>>> On Fri, Apr 28, 2017 at 2:55 PM, Viktor Klang >> > wrote:
>>>
 Hi Shiva,

 On Fri, Apr 28, 2017 at 11:20 AM, Shiva Ramagopal  wrote:

> I'm looking to compare Kafka Streams vs Akka Streams in two areas: 
>
> 1. For ingesting between Kafka and HDFS/RDBMS 
>
> Requirements are mainly around performance and latency. A Kafka topic 
> can have several million events, each corresponding to a database change 
> capture. When ingesting this topic into HDFS I'm also looking to 
> partition 
> the data by day, typically based on a timestamp field in the event 
> record, 
> aggregations on-the-fly (say by a userid field) and parquetizing 
> (preferably on the fly to optimize performance by reducing two I/O 
> operations).
>

 Looking forward to see your benchmark!

>>>
>>> Hey, you wanted requirements! :) 
>>>
  

>
> 2. Low-latency processing
>
> Experiences around performances of Storm/Flink and Akka Streams would 
> be *very* nice. Typical use-cases are de-duping and enrichment with 
> metrics 
> computation (# duplicate events/records, aggregate metrics etc). Low 
> latency and scalability are main considerations
>

 Low latency is not a metric, and scalability is not a profile. :)
 In other words: What latency distribution are you targeting and what 
 scalability curve?

>>>
>>> I know latency is a nuanced topic. I'm just looking for broad 
>>> experiences on performance comparisons if anyone has done that. Currently 
>>> we have a Storm topology over 5 nodes doing enrichment of events from Kafka 
>>> that involves a lookup into a db per event. 90th percentile of latency of 
>>> this processing is under 200 ms and we are happy with this. While Storm is 
>>> mature, Akka Streams seems more general purpose than Storm. I'd like to use 
>>> Akka Streams for this reason if performance is comparable to Storm. 
>>>
>>
>> There's one important architectural difference here tho: Akka Streams are 
>> local-only (as in materialization). You can of course materialize Akka 
>> Streams on multiple nodes and use a transport to coordinate data 
>> processing. Interestingly it doesn't lock you in to a particular backend 
>> such that Kafka Streams or even Storm would do.
>>
>
> The materialization part was something I was not aware of. Thanks for 
> pointing it out, really appreciate it.
>
>>
>> All of this boils down to requirements. Something like Flink or Google 
>> Beam could be viable options here as well. In your situation I'd look at 
>> the requirements and make a couple of prototypes before picking a winner.
>>
>
> I was also looking for something to unify batch and streaming - which is 
> how I came to look at SMACK. I'd really like to keep the tech stack small 
> and have the parts integrate tightly with each other. Guess I have to pick 
> between Spark, Flink, Beam and Storm. 
>
>>  
>>
>>>  
>>>
  

>
> On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang  > wrote:
>
>>
>>
>> On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal > > wrote:
>>
>>> Hi,
>>>
>>> I have read through multiple articles describing the SMACK stack but 
>>> I'm having difficulty understanding the role of Akka in the stack. How 
>>> does 
>>> Akka fit in?
>>>
>>
>> Akka is for building the application itself.
>>  
>>
>>>
>>> Also I would like to know of experiences in using Akka Streams vs 
>>> Kafka Connect for ingesting from Kafka into HDFS (Hive) and RDBMS. Has 
>>> anyone used Akka Streams for say, dynamic partitioning of events from a 
>>> Kafka topic into HDFS?
>>>
>>
>> Based on what requirements?
>>  
>>
>>>
>>> TIA
>>> -Shiva
>>>
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: 
>>> 

[akka-user] SSH Tunneling and Akka remote

[Evan Chan]

Hi guys,

I have a host which is Akka 2.3.15 (yes I know that's ancient, but can't 
help it for now) running Akka cluster.
Now I have a remote client connecting to it, using 
system.actorSelection(.).

The client can connect from the host that the server is running on, no 
problem -- if I use the exact hostname that the server was started with in 
the akka remote TCP config.

Now, I wish to use ssh tunneling to connect to the server.  This does not 
appear to work.   I get errors like:

[2017-05-02 14:19:30,560] ERROR akka.remote.EndpointWriter [] 
[akka.tcp://filo-standal...@xx.yy.65.65:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Ffilo-cli%40192.168.1.108%3A60669-0/endpointWriter]
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local 
recipient [Actor[akka.tcp://filo-standalone@localhost:1235/]] arriving at 
[akka.tcp://filo-standalone@localhost:1235] inbound addresses are 
[akka.tcp://filo-standal...@xx.yy.65.65:2552]


Basically xx.yy.65.65 is the local server IP address and also the address 
the server is set to for Akka config.
"localhost" 1235 is the SSH tunnel address and port which forwards to the 
server and real Akka port (2552).

Any configs to get SSH tunneling to work?

thanks,
Evan

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Detecting connection failures - documentation appears to be incorrect

[richard]

This  document 

 describes 
how to capture a connection failure that is caused by the client. 

However, I found that the recovery clause of  reactToConnectionFailure  is 
never executed.

I tested by replacing 

.map { request =>
// simple streaming (!) "echo" response:
HttpResponse(entity = HttpEntity(ContentTypes.`text/plain(UTF-8)`, 
request.entity.dataBytes))
  }

with

.mapAsync(1) { request =>
*Future.never*
  }

so I have time to break the connection from the client side.


Akka logging indicates

> [DEBUG] [05/02/2017 13:04:33.645] 
> [default-akka.actor.default-dispatcher-6] 
> [akka://default/system/IO-TCP/selectors/$a/3] Closing connection due to IO 
> error java.io.IOException: An established connection was aborted by the 
> software in your host machine

 
The .log() operation simply indicates that "Upstream finished".


-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: akka kafka stream parallel processing

[Shannon Ma]

The above code does not seem to consume any message, from the log 

12:04| INFO | AbstractCoordinator.java 349 | Successfully joined group 
part1 with generation 1
12:04| INFO | ConsumerCoordinator.java 225 | Setting newly assigned 
partitions [sanitation-1, sanitation-0, sanitation-2] for group part1
12:04| DEBUG | KafkaConsumer.java 1358 | Pausing partition sanitation-1
12:04| DEBUG | KafkaConsumer.java 1358 | Pausing partition sanitation-0
12:04| DEBUG | KafkaConsumer.java 1358 | Pausing partition sanitation-2

12:04| DEBUG | ConsumerCoordinator.java 708 | Group part1 fetching 
committed offsets for partitions: [sanitation-1, sanitation-0, sanitation-2]
12:04| DEBUG | ConsumerCoordinator.java 751 | Group part1 has no committed 
offset for partition sanitation-1
12:04| DEBUG | ConsumerCoordinator.java 751 | Group part1 has no committed 
offset for partition sanitation-0
12:04| DEBUG | ConsumerCoordinator.java 751 | Group part1 has no committed 
offset for partition sanitation-2

12:04| DEBUG | Fetcher.java 340 | Resetting offset for partition 
sanitation-1 to earliest offset.
12:04| DEBUG | Fetcher.java 340 | Resetting offset for partition 
sanitation-0 to earliest offset.
12:04| DEBUG | Fetcher.java 340 | Resetting offset for partition 
sanitation-2 to earliest offset.

12:04| DEBUG | Fetcher.java 583 | Fetched {timestamp=-1, offset=58} for 
partition sanitation-1
12:04| DEBUG | Fetcher.java 583 | Fetched {timestamp=-1, offset=16} for 
partition sanitation-0
12:04| DEBUG | Fetcher.java 583 | Fetched {timestamp=-1, offset=16} for 
partition sanitation-2

12:04| DEBUG | KafkaConsumer.java 1358 | Pausing partition sanitation-1
12:04| DEBUG | KafkaConsumer.java 1358 | Pausing partition sanitation-0
12:04| DEBUG | KafkaConsumer.java 1358 | Pausing partition sanitation-2

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: akka kafka stream parallel processing

[Shannon Ma]

Thanks you are right, once i used all javadsl classes it gets compiled. 

I got the simple app works, i like to try how the partition works. The 
inner apply() is similar to the simple logic (no partition), now what i am 
not sure if 

1, what should completionStage contain, i am putting

CompletionStage>>

not sure if this is right.If so how/what should i return. Or should  i 
return CompletionStage, that leads to my 2nd question

2, where should i put runwith(), in the outer apply() or inner(), i have 
been looking at the examples, but still not clear to me.



Source, NotUsed>>, Control> s 
= Consumer.committablePartitionedSource(consumerSettings, 
Subscriptions.topics("sanitation"));

s.mapAsync(3, new 
Function,NotUsed>>,
 
CompletionStage>>() {

public CompletionStage> apply(
Pair, 
NotUsed>> param)
throws Exception {
param.second().map(new 
Function, 
ProducerMessage.Message>() {

public ProducerMessage.Message apply(
CommittableMessage msg)
throws Exception {
System.out.println(Thread.currentThread().getName() + "msg" 
+ msg); 
return new ProducerMessage.Message(
new ProducerRecord("akkatest", 
msg.record().key(), msg.record().value()), msg.committableOffset());
}
});
 
CompletableFuture>
 
fu = new 
CompletableFuture>();;
return fu;
}
}).runWith(Producer.commitableSink(producerSettings), materializer);
   

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] experiences with serialization: json vs protobuf vs avro vs kryo...

[Guido Medina]

You might want to try Json Smile which is a Json binary, it is very 
effective and can favor of re-using Jackson ObjectMapper
https://github.com/FasterXML/jackson-dataformats-binary/tree/master/smile

I also use Kryo for mainly Java classes where schema evolution and 
compatibility is not important as all message classes are shared and 
cluster restarted fully.

HTH,

Guido.

On Monday, May 1, 2017 at 6:08:25 PM UTC+1, Alan Burlison wrote:
>
> On 01/05/2017 17:15, Justin du coeur wrote: 
>
> > My general impression is that protobuf is the well-established route, 
> and 
> > the one that most people are using 
>
> I wrote a wrapper around Akka-Spray to save JSON in flat files. It is 
> fairly primitive but has the (for my use case) big advantage that I can 
> examine and manipulate the persisted Actor data using other tools, which 
> is extremely useful during development. 
>
> -- 
> Alan Burlison 
> -- 
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] ANNOUNCE: Akka 2.5.1

[Arnout Engelen]

Dear hakkers,

We are proud to announce Akka 2.5.1, which is the first maintenance release
of Akka 2.5.

Some notable improvements and bug fixes are:


   -

   Limiting the number of concurrent PersistentActor recoveries (#22638
   )
   -

   Fix resources leaks in some stages on materializer shutdown (#22652
   )
   -

   Fix memory usage problem when using Artery large-message-destinations (
   #22723 )
   -

   Do not create durable data files for cluster sharding unless needed (
   #22759 )
   -

   startPosition option for FileIO.fromFile (#19483
   )
   -

   Use java.nio in favour of FileInputStream/FileOutputStream (#22733
   )
   -

   Update to Aeron 1.2.5 (#22738 )


Credits

A total of 25 issues were closed since 2.5.0.

The complete list of closed issues can be found on the 2.5.1
 milestone on github.

For this release we had the help of 23 committers – thank you all very much!

commits  added  removed

11638   85 Patrik Nordwall

 7144   55 Arnout Engelen

 2333   71 Johan Andrén

 2 61   22 Enno

 1521  502 henrikengstrom

 1331   47 Gilad Hoch

 1171   20 Cedric CORBIERE

 1 954 astonchev

 1 25   23 Guido Medina

 1 18   17 Konrad `ktoso` Malawski

 1 160 loostro

 1 122 Juan Jose Lopez Martin

 1  24 Gergő Törcsvári

 1  22 Heiko Seeberger

 1  22 Tim Moore

 1  11 Martynas Mickevičius

 1  11 James Roper

 1  11 Guillermo Lammers

 1  11 Evgeny Veretennikov

 1  11 everetennikov

 1  10 gosubpl

 1  00 Hyungsuk Yoon


Happy hakking!

– The Akka Team

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Graph materialized value of websocket upgrade

[Konrad Malawski]

-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 

On 2 May 2017 at 23:28:51, Mathieu Bruyen (mathbru...@gmail.com) wrote:

It looks a bit unnatural, should there be a way of doing this similar to
the client one (in which the materialized value is directly returned)?

There is one big difference, the client *you* materialize, so you can get
back the value easily, the server however materializes it for you, so
you've passed it out and don't have a simple way of getting it back in hand.

Please open a ticket about that; technically it will be the exact same
operation, however perhaps we can add a different name for it, like expose
or something hm.


If mapMaterializedValue is the way to go, does it deserve a mention in the
documentation (I can create the PR for that)?

Please open a ticket or send in a PR about that :-)




On Tuesday, May 2, 2017 at 12:42:43 PM UTC+2, Konrad Malawski wrote:
>
> Sorry for the brief response, quite a lot of text in there :)
>
> Yeah, mapMaterializedValue is a way to getting it out and side effecting
> it into somewhere.
>
> --
> Konrad `ktoso` Malawski
> Akka  @ Lightbend 
>
> On 2 May 2017 at 19:35:32, Mathieu Bruyen (mathb...@gmail.com) wrote:
>
> Hello,
>
> (this is more related to akka-http but there seem to be no dedicated group)
>
> I am trying to setup a websocket server for which there is no 1-1 relation
> between incoming and outgoing messages. The client can send messages at any
> time which may or not have a reply from the server. The server can push
> messages at any time which may or not have a reply from the client. I
> wanted to let actors handle and produce these messages. Following the doc
> 
>  I am using Sink#actorRefWithAck to push incoming messages to the
> entrypoint/router actor and let it do it's job. For outgoing messages in
> the options suggested by the doc
> 
>  I was planning to use Source#queue and give this queue to all the actors
> which might need to push to the queue. I targeted this solution because it
> allows for backpressure management.
>
> My issue is that I need access to the queue materialized value (in order
> to call offer) but WebSocket#handleWebSocketRequestWith does not provide
> access to it. It returns only the HttpResponse to send back to the client.
>
> Source queue =
> Source.queue(5, OverflowStrategy.backpressure());
> Sink sink = Sink.actorRefWithAck(tunnel, Messages.INIT,
> Messages.ACK, Messages.COMPLETE, this::onError);
> Flow flow =
> Flow.fromSinkAndSourceMat(sink, queue, Keep.right());
> HttpResponse response = WebSocket.handleWebSocketRequestWith(request,
> greeterFlow);
>
> I tried the "lower level" UpgradeToWebSocket#handleMessagesWith but got
> to the same conclusion: only the HTTP response is returned back:
>
> HttpResponse response = ((UpgradeToWebSocket)header).handleMessagesWith(sink,
> queue);
>
> I worked around the issue by using mapMaterializedValue, which gives me
> the materialized value which I can give to my handler actor for them to
> publish on. But it sounds like a hack and I would prefer not having side
> effects in mapMaterliazedValue:
>
> Source.queue(5, OverflowStrategy.backpressure())
> .mapMaterializedValue(sourceQueue -> { 
> tunnel.tell(InitTunnel.create(sourceQueue),
> Actor.noSender()); return sourceQueue; });
>
> On the other side, the client gives it back:
>
> Pair SourceQueueWithComplete>
> pair =
> http.singleWebSocketRequest(WebSocketRequest.create(connect),
> flow, materializer);
>
> I wonder if I am misusing the API or there is a miss here. If nothing in
> the API lets users do that, would that be a desirable addition?
>
> While debugging I saw that the graph is materialized after the response is
> sent back, so there might be a technical limitation which I am not aware of
> that forces to materialize the graph later.
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
> --
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html

Re: [akka-user] Graph materialized value of websocket upgrade

[Mathieu Bruyen]

It looks a bit unnatural, should there be a way of doing this similar to 
the client one (in which the materialized value is directly returned)? If 
mapMaterializedValue is the way to go, does it deserve a mention in the 
documentation (I can create the PR for that)?

On Tuesday, May 2, 2017 at 12:42:43 PM UTC+2, Konrad Malawski wrote:
>
> Sorry for the brief response, quite a lot of text in there :)
>
> Yeah, mapMaterializedValue is a way to getting it out and side effecting 
> it into somewhere.
>
> -- 
> Konrad `ktoso` Malawski
> Akka  @ Lightbend 
>
> On 2 May 2017 at 19:35:32, Mathieu Bruyen (mathb...@gmail.com 
> ) wrote:
>
> Hello,
>
> (this is more related to akka-http but there seem to be no dedicated group)
>
> I am trying to setup a websocket server for which there is no 1-1 relation 
> between incoming and outgoing messages. The client can send messages at any 
> time which may or not have a reply from the server. The server can push 
> messages at any time which may or not have a reply from the client. I 
> wanted to let actors handle and produce these messages. Following the doc 
> 
>  
> I am using Sink#actorRefWithAck to push incoming messages to the 
> entrypoint/router actor and let it do it's job. For outgoing messages in 
> the options suggested by the doc 
> 
>  
> I was planning to use Source#queue and give this queue to all the actors 
> which might need to push to the queue. I targeted this solution because it 
> allows for backpressure management.
>
> My issue is that I need access to the queue materialized value (in order 
> to call offer) but WebSocket#handleWebSocketRequestWith does not provide 
> access to it. It returns only the HttpResponse to send back to the client.
>
> Source queue = 
> Source.queue(5, OverflowStrategy.backpressure());
> Sink sink = Sink.actorRefWithAck(tunnel, Messages.INIT, 
> Messages.ACK, Messages.COMPLETE, this::onError);
> Flow flow = 
> Flow.fromSinkAndSourceMat(sink, queue, Keep.right());
> HttpResponse response = WebSocket.handleWebSocketRequestWith(request, 
> greeterFlow);
>
> I tried the "lower level" UpgradeToWebSocket#handleMessagesWith but got to 
> the same conclusion: only the HTTP response is returned back:
>
> HttpResponse response = 
> ((UpgradeToWebSocket)header).handleMessagesWith(sink, queue);
>
> I worked around the issue by using mapMaterializedValue, which gives me 
> the materialized value which I can give to my handler actor for them to 
> publish on. But it sounds like a hack and I would prefer not having side 
> effects in mapMaterliazedValue:
>
> Source.queue(5, OverflowStrategy.backpressure())
> .mapMaterializedValue(sourceQueue -> { 
> tunnel.tell(InitTunnel.create(sourceQueue), Actor.noSender()); return 
> sourceQueue; });
>
> On the other side, the client gives it back:
>
> Pair SourceQueueWithComplete> pair =
> http.singleWebSocketRequest(WebSocketRequest.create(connect), 
> flow, materializer);
>
> I wonder if I am misusing the API or there is a miss here. If nothing in 
> the API lets users do that, would that be a desirable addition?
>
> While debugging I saw that the graph is materialized after the response is 
> sent back, so there might be a technical limitation which I am not aware of 
> that forces to materialize the graph later.
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+...@googlegroups.com .
> To post to this group, send email to akka...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] experiences with serialization: json vs protobuf vs avro vs kryo...

[Joost de Vries]

Thanks for your experiences.

It sounds that protobuf is the most often used serialisation. And perhaps 
the need to write IDL is something where we'll have to bite the bullet. 
It's a bit of double book keeping but at least it should be easily 
testable

I think the standard serialisation for Lagom is json. I haven't seen 
information on the why of that decision anywhere. Would be interesting to 
know.

On Monday, May 1, 2017 at 7:08:25 PM UTC+2, Alan Burlison wrote:
>
> On 01/05/2017 17:15, Justin du coeur wrote: 
>
> > My general impression is that protobuf is the well-established route, 
> and 
> > the one that most people are using 
>
> I wrote a wrapper around Akka-Spray to save JSON in flat files. It is 
> fairly primitive but has the (for my use case) big advantage that I can 
> examine and manipulate the persisted Actor data using other tools, which 
> is extremely useful during development. 
>
> -- 
> Alan Burlison 
> -- 
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] ANNOUNCE: Akka 2.4.18

[Patrik Nordwall]

Dear hakkers,

We are proud to announce Akka 2.4.18, which is another maintenance release
of Akka 2.4.

Some notable improvements and bug fixes are:


   -

   Limiting number of concurrent PersistentActor recoveries, #22638
   
   -

   Some stream stages may leak resources on materializer shutdown, #22652
   
   -

   Stream FileIO should truncate existing files (before, existing files
   were appended) #22782 
   -

   Improve serialization of remote deployment Props #4
   
   -

   Fix memory usage problem when using Artery large-message-destinations,
   #22723 
   -

   Update to Aeron 1.2.5

Credits

A total 40 issues were closed since 2.4.17.

The complete list of closed issues can be found on the 2.4.18
 milestone on github.

For this release we had the help of 16 committers – thank you all very much!

commits  added  removed

15   1669  335 Johan Andrén

 8527   76 Patrik Nordwall

 7139   51 Johannes Rudolph

 6626   40 Konrad `ktoso` Malawski

 4 70   14 ortigali

 3453   86 Björn Antonsson

 2176   25 Cedric CORBIERE

 1 954 Arnout Engelen

 1 46   11 Kirill Yankov

 1 471 Javier Puerto

 1 15   10 drewhk

 1 148 Seth Tisue

 1  77 Nafer Sanabria

 1 131 Sergey

 1  22 sebastian.harko

 1  12 Heiko Seeberger




Happy hakking!

– The Akka Team

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka and RDMA

[Konrad Malawski]

I'd read that note as "it is possible to put it onto RDMA" :-)

-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 

On 2 May 2017 at 19:35:31, Marc Christoph (facelessto...@gmail.com) wrote:

Apologies, Michal. The resource containing the (brief) mention of RDMA can
be found here
, under
"Design Assumptions". I'll quote it here for convenience.

*"The Aeron protocol is designed to be run directly over many different
types of transmission media, including shared memory/IPC, InfiniBand/RDMA,
UDP, TCP, Raw IP, HTTP, WebSocket, BLE, etc. This means that the following
assumptions are made:*

   - *Transmission Media may be a stream media, such as TCP or RDMA without
   inherent frame boundaries.*
   - *..."*

Marc

On Tuesday, 2 May 2017 11:00:42 UTC+2, Michal Borowiecki wrote:
>
> Hi Marc,
>
> Sounds very interesting but I couldn't find evidence of Aeron directly
> supporting RDMA in the links your provided or otherwise. Can you please
> point me to your sources?
>
> I found this github ticket https://github.com/real-logic/Aeron/issues/220
> but it's still open.
>
> Thanks,
>
> Michal
>
> On 02/05/17 09:43, Marc Christoph wrote:
>
> Hi, everyone
>
> I am having some issues understanding the implementation details of RDMA
> and whether it is supported by Akka. According to this blog post
> , Akka is using
> Aeron  as an underlying transport in
> the remoting implementation, and Aeron supports RDMA. However, I am unsure
> of how to leverage it in my own applications. From what I understand, RDMA
> is a feature of the network adapter, but I do not know if I need to
> explicitly make use of it in my code? Or if Akka provides functionality to
> do so?
>
> Thanks in Advance
>
> Marc
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
>  Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600
>
>
> +44 203 249 8448
>
>
>
> E: michal.b...@openbet.com
> W: www.openbet.com
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
> 
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> ...@openbet.com and delete it from your system as well as any copies. The
> content of e-mails as well as traffic data may be monitored by OpenBet for
> employment and security purposes. To protect the environment please do not
> print this e-mail unless necessary. OpenBet Ltd. Registered Office:
> Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United
> Kingdom. A company registered in England and Wales. Registered no. 3134634.
> VAT no. GB927523612
>
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Graph materialized value of websocket upgrade

[Konrad Malawski]

Sorry for the brief response, quite a lot of text in there :)

Yeah, mapMaterializedValue is a way to getting it out and side effecting it
into somewhere.

-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 

On 2 May 2017 at 19:35:32, Mathieu Bruyen (mathbru...@gmail.com) wrote:

Hello,

(this is more related to akka-http but there seem to be no dedicated group)

I am trying to setup a websocket server for which there is no 1-1 relation
between incoming and outgoing messages. The client can send messages at any
time which may or not have a reply from the server. The server can push
messages at any time which may or not have a reply from the client. I
wanted to let actors handle and produce these messages. Following the doc

I am using Sink#actorRefWithAck to push incoming messages to the
entrypoint/router actor and let it do it's job. For outgoing messages in
the options suggested by the doc

I was planning to use Source#queue and give this queue to all the actors
which might need to push to the queue. I targeted this solution because it
allows for backpressure management.

My issue is that I need access to the queue materialized value (in order to
call offer) but WebSocket#handleWebSocketRequestWith does not provide
access to it. It returns only the HttpResponse to send back to the client.

Source queue =
Source.queue(5, OverflowStrategy.backpressure());
Sink sink = Sink.actorRefWithAck(tunnel, Messages.INIT,
Messages.ACK, Messages.COMPLETE, this::onError);
Flow flow =
Flow.fromSinkAndSourceMat(sink, queue, Keep.right());
HttpResponse response = WebSocket.handleWebSocketRequestWith(request,
greeterFlow);

I tried the "lower level" UpgradeToWebSocket#handleMessagesWith but got to
the same conclusion: only the HTTP response is returned back:

HttpResponse response =
((UpgradeToWebSocket)header).handleMessagesWith(sink, queue);

I worked around the issue by using mapMaterializedValue, which gives me the
materialized value which I can give to my handler actor for them to publish
on. But it sounds like a hack and I would prefer not having side effects in
mapMaterliazedValue:

Source.queue(5, OverflowStrategy.backpressure())
.mapMaterializedValue(sourceQueue -> {
tunnel.tell(InitTunnel.create(sourceQueue), Actor.noSender()); return
sourceQueue; });

On the other side, the client gives it back:

Pair pair =
http.singleWebSocketRequest(WebSocketRequest.create(connect), flow,
materializer);

I wonder if I am misusing the API or there is a miss here. If nothing in
the API lets users do that, would that be a desirable addition?

While debugging I saw that the graph is materialized after the response is
sent back, so there might be a technical limitation which I am not aware of
that forces to materialize the graph later.
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Graph materialized value of websocket upgrade

[Mathieu Bruyen]

Hello,

(this is more related to akka-http but there seem to be no dedicated group)

I am trying to setup a websocket server for which there is no 1-1 relation 
between incoming and outgoing messages. The client can send messages at any 
time which may or not have a reply from the server. The server can push 
messages at any time which may or not have a reply from the client. I 
wanted to let actors handle and produce these messages. Following the doc 

 
I am using Sink#actorRefWithAck to push incoming messages to the 
entrypoint/router actor and let it do it's job. For outgoing messages in 
the options suggested by the doc 

 
I was planning to use Source#queue and give this queue to all the actors 
which might need to push to the queue. I targeted this solution because it 
allows for backpressure management.

My issue is that I need access to the queue materialized value (in order to 
call offer) but WebSocket#handleWebSocketRequestWith does not provide 
access to it. It returns only the HttpResponse to send back to the client.

Source queue = 
Source.queue(5, OverflowStrategy.backpressure());
Sink sink = Sink.actorRefWithAck(tunnel, Messages.INIT, 
Messages.ACK, Messages.COMPLETE, this::onError);
Flow flow = 
Flow.fromSinkAndSourceMat(sink, queue, Keep.right());
HttpResponse response = WebSocket.handleWebSocketRequestWith(request, 
greeterFlow);

I tried the "lower level" UpgradeToWebSocket#handleMessagesWith but got to 
the same conclusion: only the HTTP response is returned back:

HttpResponse response = 
((UpgradeToWebSocket)header).handleMessagesWith(sink, queue);

I worked around the issue by using mapMaterializedValue, which gives me the 
materialized value which I can give to my handler actor for them to publish 
on. But it sounds like a hack and I would prefer not having side effects in 
mapMaterliazedValue:

Source.queue(5, OverflowStrategy.backpressure())
.mapMaterializedValue(sourceQueue -> { 
tunnel.tell(InitTunnel.create(sourceQueue), Actor.noSender()); return 
sourceQueue; });

On the other side, the client gives it back:

Pair pair =
http.singleWebSocketRequest(WebSocketRequest.create(connect), flow, 
materializer);

I wonder if I am misusing the API or there is a miss here. If nothing in 
the API lets users do that, would that be a desirable addition?

While debugging I saw that the graph is materialized after the response is 
sent back, so there might be a technical limitation which I am not aware of 
that forces to materialize the graph later.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka and RDMA

[Marc Christoph]

Apologies, Michal. The resource containing the (brief) mention of RDMA can 
be found here 
, under 
"Design Assumptions". I'll quote it here for convenience.

*"The Aeron protocol is designed to be run directly over many different 
types of transmission media, including shared memory/IPC, InfiniBand/RDMA, 
UDP, TCP, Raw IP, HTTP, WebSocket, BLE, etc. This means that the following 
assumptions are made:*

   - *Transmission Media may be a stream media, such as TCP or RDMA without 
   inherent frame boundaries.*
   - *..."*

Marc

On Tuesday, 2 May 2017 11:00:42 UTC+2, Michal Borowiecki wrote:
>
> Hi Marc,
>
> Sounds very interesting but I couldn't find evidence of Aeron directly 
> supporting RDMA in the links your provided or otherwise. Can you please 
> point me to your sources?
>
> I found this github ticket https://github.com/real-logic/Aeron/issues/220 
> but it's still open.
>
> Thanks,
>
> Michal
>
> On 02/05/17 09:43, Marc Christoph wrote:
>
> Hi, everyone 
>  
> I am having some issues understanding the implementation details of RDMA 
> and whether it is supported by Akka. According to this blog post 
> , Akka is using 
> Aeron  as an underlying transport in 
> the remoting implementation, and Aeron supports RDMA. However, I am unsure 
> of how to leverage it in my own applications. From what I understand, RDMA 
> is a feature of the network adapter, but I do not know if I need to 
> explicitly make use of it in my code? Or if Akka provides functionality to 
> do so?
>
> Thanks in Advance
>
> Marc
> -- 
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+...@googlegroups.com .
> To post to this group, send email to akka...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
> -- 
>  Michal Borowiecki 
> Senior Software Engineer L4 
> T: +44 208 742 1600 
>
>
> +44 203 249 8448 
>
>
>   
> E: michal.b...@openbet.com  
> W: www.openbet.com 
> OpenBet Ltd 
>
> Chiswick Park Building 9 
>
> 566 Chiswick High Rd 
>
> London 
>
> W4 5XT 
>
> UK 
>  
> This message is confidential and intended only for the addressee. If you 
> have received this message in error, please immediately notify the 
> ...@openbet.com  and delete it from your system as well as 
> any copies. The content of e-mails as well as traffic data may be monitored 
> by OpenBet for employment and security purposes. To protect the environment 
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered 
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, 
> United Kingdom. A company registered in England and Wales. Registered no. 
> 3134634. VAT no. GB927523612 
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Yaroslav Klymko]

Hi Patrik,

Yes that was an idea and the Future passed would be of something 
like 
https://github.com/evolution-gaming/akka-tools/blob/master/src/main/scala/com/evolutiongaming/util/FutureSequentialForKey.scala#L17

In general the goal is to make sure large messages do not affect other 
small messages. So having custom `lane` for large and your default for 
small msgs could satisfy my needs. 
Do you have an own idea on how we can achieve this goal?



On Tuesday, May 2, 2017 at 12:56:23 PM UTC+3, Patrik Nordwall wrote:
>
> Are you thinking of adding mapAsync to the Artery lanes? It could be 
> interesting to look at a suggestion for that. Adding more async boundaries 
> will be worse for latency of normal messages, so it should not be the 
> default.
>
> Also note that compression and serialization is CPU bound tasks so if you 
> try to do more than you have cores the performance will only be worse. You 
> also want to leave some cores for other tasks in your application, such as 
> processing the actual messages.
>
> /Patrik
>
>
>
> On Mon, May 1, 2017 at 8:30 PM, Yaroslav Klymko  > wrote:
>
>> I'm looking further in order to understand on how artery fixes it and 
>> truly saying I'm not in favour of configurable number of inbound-lanes and 
>> outbound-lanes.
>>
>> Right now we have an issue with passing messages in cluster. Sometimes we 
>> want to send large messages and make sure those are compressed to minimise 
>> network use. However adding compression will have negative impact for other 
>> messages that have to share same single actor. So I have an idea to 
>> basically parallelize serialisation + compression for messages that for 
>> instance are bigger then some configurable size and use `recipient` as key 
>> for parallelisation, this way I will be able to preserver order.
>>
>> Question: 
>> * Is akka team interested in such idea and will implement solution? 
>> * Interested and will accept the pull request ? 
>> * Interested in pull request that will expose dispatching api for custom 
>> implementation?
>> * None of the above
>>
>>
>>
>>
>> On Friday, April 28, 2017 at 6:32:43 PM UTC+3, Akka Team wrote:
>>>
>>> That is correct, there is one writer and one reader per connection in 
>>> the stable remoting, so if serialization is slow/heavy that may be a 
>>> bottleneck.
>>>
>>> In the new remoting, "artery", we have made it possible to run multiple 
>>> in and outbound "lanes", but that functionality is not yet hardened enough 
>>> to use.
>>> You can see the settings in reference.conf here: 
>>> http://doc.akka.io/docs/akka/2.5.0/general/configuration.html#config-akka-remote-artery
>>>  
>>> - inbound-lanes and outbound-lanes.
>>>
>>> --
>>> Johan
>>> Akka Team
>>>
>>> On Fri, Apr 28, 2017 at 12:45 PM, Yaroslav Klymko  
>>> wrote:
>>>
 Hi guys,

 Do I understand correctly that we have a single instance of 
 `EndpointReader` actor per connection ? 
 Thus deserialisation of remote messages being done sequentially, so we 
 can hit this bottleneck much faster then a network throughput limits?

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.
>>>
>>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
>
> Patrik Nordwall
> Akka Tech Lead
> Lightbend  -  Reactive apps on the JVM
> Twitter: @patriknw
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and 

Re: [akka-user] EndpointReader actor is a remoting bottleneck in case of heavy deserialisation

[Patrik Nordwall]

Are you thinking of adding mapAsync to the Artery lanes? It could be
interesting to look at a suggestion for that. Adding more async boundaries
will be worse for latency of normal messages, so it should not be the
default.

Also note that compression and serialization is CPU bound tasks so if you
try to do more than you have cores the performance will only be worse. You
also want to leave some cores for other tasks in your application, such as
processing the actual messages.

/Patrik



On Mon, May 1, 2017 at 8:30 PM, Yaroslav Klymko  wrote:

> I'm looking further in order to understand on how artery fixes it and
> truly saying I'm not in favour of configurable number of inbound-lanes and
> outbound-lanes.
>
> Right now we have an issue with passing messages in cluster. Sometimes we
> want to send large messages and make sure those are compressed to minimise
> network use. However adding compression will have negative impact for other
> messages that have to share same single actor. So I have an idea to
> basically parallelize serialisation + compression for messages that for
> instance are bigger then some configurable size and use `recipient` as key
> for parallelisation, this way I will be able to preserver order.
>
> Question:
> * Is akka team interested in such idea and will implement solution?
> * Interested and will accept the pull request ?
> * Interested in pull request that will expose dispatching api for custom
> implementation?
> * None of the above
>
>
>
>
> On Friday, April 28, 2017 at 6:32:43 PM UTC+3, Akka Team wrote:
>>
>> That is correct, there is one writer and one reader per connection in the
>> stable remoting, so if serialization is slow/heavy that may be a bottleneck.
>>
>> In the new remoting, "artery", we have made it possible to run multiple
>> in and outbound "lanes", but that functionality is not yet hardened enough
>> to use.
>> You can see the settings in reference.conf here: http://doc.akka.io/docs/
>> akka/2.5.0/general/configuration.html#config-akka-remote-artery -
>> inbound-lanes and outbound-lanes.
>>
>> --
>> Johan
>> Akka Team
>>
>> On Fri, Apr 28, 2017 at 12:45 PM, Yaroslav Klymko 
>> wrote:
>>
>>> Hi guys,
>>>
>>> Do I understand correctly that we have a single instance of
>>> `EndpointReader` actor per connection ?
>>> Thus deserialisation of remote messages being done sequentially, so we
>>> can hit this bottleneck much faster then a network throughput limits?
>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>>> urrent/additional/faq.html
>>> >> Search the archives: https://groups.google.com/grou
>>> p/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 

Patrik Nordwall
Akka Tech Lead
Lightbend  -  Reactive apps on the JVM
Twitter: @patriknw

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka and RDMA

['Michal Borowiecki' via Akka User List]

Hi Marc,

Sounds very interesting but I couldn't find evidence of Aeron directly 
supporting RDMA in the links your provided or otherwise. Can you please 
point me to your sources?


I found this github ticket 
https://github.com/real-logic/Aeron/issues/220 but it's still open.


Thanks,

Michal


On 02/05/17 09:43, Marc Christoph wrote:

Hi, everyone
I am having some issues understanding the implementation details of 
RDMA and whether it is supported by Akka. According to this blog post 
, Akka is using 
Aeron  as an underlying transport 
in the remoting implementation, and Aeron supports RDMA. However, I am 
unsure of how to leverage it in my own applications. From what I 
understand, RDMA is a feature of the network adapter, but I do not 
know if I need to explicitly make use of it in my code? Or if Akka 
provides functionality to do so?


Thanks in Advance

Marc
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
http://doc.akka.io/docs/akka/current/additional/faq.html

>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google 
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to akka-user+unsubscr...@googlegroups.com 
.
To post to this group, send email to akka-user@googlegroups.com 
.

Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.


--
Signature
 Michal Borowiecki
Senior Software Engineer L4
T:  +44 208 742 1600


+44 203 249 8448



E:  michal.borowie...@openbet.com
W:  www.openbet.com 


OpenBet Ltd

Chiswick Park Building 9

566 Chiswick High Rd

London

W4 5XT

UK




This message is confidential and intended only for the addressee. If you 
have received this message in error, please immediately notify the 
postmas...@openbet.com  and delete it 
from your system as well as any copies. The content of e-mails as well 
as traffic data may be monitored by OpenBet for employment and security 
purposes. To protect the environment please do not print this e-mail 
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company 
registered in England and Wales. Registered no. 3134634. VAT no. 
GB927523612


--

 Read the docs: http://akka.io/docs/
 Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
 Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Akka and RDMA

[Marc Christoph]

Hi, everyone
 
I am having some issues understanding the implementation details of RDMA 
and whether it is supported by Akka. According to this blog post 
, Akka is using Aeron 
 as an underlying transport in the 
remoting implementation, and Aeron supports RDMA. However, I am unsure of 
how to leverage it in my own applications. From what I understand, RDMA is 
a feature of the network adapter, but I do not know if I need to explicitly 
make use of it in my code? Or if Akka provides functionality to do so?

Thanks in Advance

Marc

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.