[akka-user] Nested Slick streams, buffer and backpressure

[Richard Rodseth]

I'm still missing something. I thought I had solved my problem of
overwhelming Postgres by using buffer(), but today (after upgrading to
2.4.2-RC3, but that's probably coincidence) I am getting a lot of timeouts.
As you can see below, I have two Slick sources, one nested via
flatMapConcat.

Even with all these superfluous buffer() calls, I still see a ton of
channel-month-for-interval-query
messages

I assume that the Slick DataBasePublisher is not signalling backpressure if
waiting for a connection.

How can I limit the nested slick stream call to something like the size of
the connection pool?

If the flow from second query on was run as a separate stream run in a
mapAsync I suppose I could use the parallelism value and backpressure would
flow back to the buffer, but how can I do it in one flow?

val channels = Sources.channelsForFilter(db, channelFilter, 1000 /*fetchSize
*/ ) // First Slick call

val source = channels

  .buffer(5, OverflowStrategy.backpressure) // Candidate 1

  .flatMapConcat { channel =>

Sources.monthsForChannel(channel, requestedRange)

  }

  .buffer(5, OverflowStrategy.backpressure) // Candidate 2

  .groupBy(10, c => c.channel.channelId)

  .buffer(5,OverflowStrategy.backpressure) // Candidate 3

  .via(Transformations.progress(e => s"channel-month-for-interval-query
$e")) // Message logged

  .flatMapConcat { channelMonth =>

val r = Sources.intervalRowsForChannelAndTime(db, channelMonth.
channel, channelMonth.period, batchSize) // Second Slick call

  .buffer(5, OverflowStrategy.backpressure) // Candidate 4

r.map { x => ChannelAndInstantRangeAndInterval(channelMonth.channel,
channelMonth.period, x) }

  }

  .via(Transformations.channelMonthIntervalToBytesWritten)

  .mergeSubstreams

  .via(Transformations.progress(fileWritten => s"File written $
fileWritten"))

val sink = Sinks.countingSink

val runnable = source.toMat(sink)(Keep.right)

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: [ANNOUNCE] Akka 2.4.2 including Streams and HTTP Released!

[Gary Struthers]

I've migrated and everything compiles except the http route test. Tests 
like the one shown don't compile, "route" isn't found.

  it should "respond with handled = false for partial path" in {

Get(saPath) ~> route ~> check {

  handled shouldEqual false

}

  }

I don't see how to migrate this, I've 
tried "akka-http-testkit-experimental_2.11" % "2.4.2-RC3" and 2.0.3

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] [ANNOUNCE] Akka 2.4.2 including Streams and HTTP Released!

[Richard Rodseth]

Spoke too soon about snappy-flows.
ByteStringParser seems to have moved or disappeared between RC-3 and final.
I'll ping the author.

Caused by: java.lang.ClassNotFoundException: akka.stream.io.ByteStringParser

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

On Wed, Feb 17, 2016 at 12:55 PM, Richard Rodseth 
wrote:

> Congratulations! I'm excited about this release, and I think if I could
> Stream All The Things for the rest of my career, I would die a happy man :)
>
> Migration note from RC3: IOResult and Framing have changed package. The
> migration guide refers to the old location.
> snappy-flows 0.1.2 (built with RC3) seems to work with 2.4.2, but I'm sure
> a 0.1.3 will be forthcoming.
>
> Thanks everyone.
>
> On Wed, Feb 17, 2016 at 7:42 AM, Konrad Malawski 
> wrote:
>
>> *Dear hakkers,*
>>
>> we—the Akka committers—are proud to announce the FINAL RELEASE of Akka
>> 2.4.2. The main change in this release is that it includes Streams & HTTP.
>> Some of these new modules are still marked experimental due to impending
>> API changes that we could not yet finish, in particular these are akka-http
>> (the DSLs) and the HTTP marshaller modules for spray-json, jackson, and
>> scala-xml. In comparison to Streams & HTTP 2.0 the main changes are:
>>
>>- significant performance improvement for HTTP handling, now reaching
>>roughly 75% of Spray’s performance for long-lived HTTP connections—this is
>>not the end of the performance work, we have only just begun (in 
>> particular
>>the number of connections per second needs more work)
>>- replacement of all uses of the Unit type (represented as BoxedUnit
>>in Java) with the more descriptive typesakka.Done (for signaling
>>successful completion) and akka.NotUsed (for materialization results
>>of stages that do not produce a value)
>>- usage of Java 8 types in the Java DSLs: java.util.Optional instead
>>of scala.Option andjava.util.concurrent.CompletionStage instead of
>>scala.concurrent.Future
>>
>> Especially the second and third point mean that porting code from Streams
>> & HTTP 2.0 to Akka 2.4.2 will require some mechanical source code changes,
>> please refer to the migration guide(for Java
>> 
>>  and Scala
>> )
>> for the details.
>>
>> But also in the rest of Akka much work was done, with highlights being:
>>
>>- fixed a possible replay consistency issue
>> in Akka Persistence and a too
>>tight specification  in
>>the TCK
>>- silence heartbeat logging
>> in ClusterClient and add a
>> notification when it cannot reconnect
>>
>>- provide an API for ClusterSharding state retrieval
>>
>>- improvements to BackoffSupervisor
>>
>>- fix documentation for max-pool-size settings
>> (this was a little
>>unintuitive, makes a good trivia question!)
>>- add Java API for ByteString builders
>>
>>- fix a race condition that could lead to lost messages in
>>DistributedPubSub 
>>- add an ask() variant that can be used without sender(), i.e. by putting
>>the promise reference into the message
>>
>>- added Java patterns for CompletionStage, see PatternsCS
>>
>>
>> Closed issues since version 2.4.1 can be found here
>> .
>>
>>
>> *Binary Compatibility*
>>
>> Akka 2.4.2 is backwards binary compatible with previous 2.4.x and 2.3.x
>> versions (exceptions listed below). This means that the new JARs are a
>> drop-in replacement for the old one (but not the other way around) as long
>> as your build does not enable the inliner (Scala-only restriction). It
>> should be noted that Scala 2.11.x is is not binary compatible with Scala
>> 2.10.x, which means that Akka’s binary compatibility property only holds
>> between versions that were built for a given Scala
>> version—akka-actor_2.11-2.4.2-RC1.jar is compatible with
>> akka-actor_2.11-2.3.14.jar but not with akka-actor_2.10-2.3.14.jar.
>>
>> Binary compatibility is *not* maintained for the following:
>>
>>- testkits:
>>   - akka-testkit
>>   - akka-multi-node-testkit
>>   - akka-persistence-tck
>>   - akka-stream-testkit
>>   - akka-http-testkit
>>- experimental modules:
>>   - 

[akka-user] Re: [ANNOUNCE] Akka 2.4.2 including Streams and HTTP Released!

[hbf]

Thanks a lot! – A "humongous" thanks, I want to say :) 

One question: will there be a release for Scala 2.10 (i.e., 
'com.typesafe.akka:akka-actor_2.10:2.4.2')?

– Hbf


On Wednesday, February 17, 2016 at 7:42:18 AM UTC-8, Konrad Malawski wrote:
>
> *Dear hakkers,*
>
> we—the Akka committers—are proud to announce the FINAL RELEASE of Akka 
> 2.4.2. The main change in this release is that it includes Streams & HTTP. 
> Some of these new modules are still marked experimental due to impending 
> API changes that we could not yet finish, in particular these are akka-http 
> (the DSLs) and the HTTP marshaller modules for spray-json, jackson, and 
> scala-xml. In comparison to Streams & HTTP 2.0 the main changes are:
>
>- significant performance improvement for HTTP handling, now reaching 
>roughly 75% of Spray’s performance for long-lived HTTP connections—this is 
>not the end of the performance work, we have only just begun (in 
> particular 
>the number of connections per second needs more work)
>- replacement of all uses of the Unit type (represented as BoxedUnit 
>in Java) with the more descriptive typesakka.Done (for signaling 
>successful completion) and akka.NotUsed (for materialization results 
>of stages that do not produce a value)
>- usage of Java 8 types in the Java DSLs: java.util.Optional instead of
> scala.Option andjava.util.concurrent.CompletionStage instead of 
>scala.concurrent.Future
>
> Especially the second and third point mean that porting code from Streams 
> & HTTP 2.0 to Akka 2.4.2 will require some mechanical source code changes, 
> please refer to the migration guide(for Java 
> 
>  and Scala 
> )
>  
> for the details.
>
> But also in the rest of Akka much work was done, with highlights being:
>
>- fixed a possible replay consistency issue 
> in Akka Persistence and a too 
>tight specification  in the 
>TCK
>- silence heartbeat logging 
> in ClusterClient and add a notification when it cannot reconnect 
>
>- provide an API for ClusterSharding state retrieval 
>
>- improvements to BackoffSupervisor 
>
>- fix documentation for max-pool-size settings 
> (this was a little 
>unintuitive, makes a good trivia question!)
>- add Java API for ByteString builders 
>
>- fix a race condition that could lead to lost messages in 
>DistributedPubSub 
>- add an ask() variant that can be used without sender(), i.e. by putting 
>the promise reference into the message 
>
>- added Java patterns for CompletionStage, see PatternsCS 
>
>
> Closed issues since version 2.4.1 can be found here 
> .
>
>
> *Binary Compatibility*
>
> Akka 2.4.2 is backwards binary compatible with previous 2.4.x and 2.3.x 
> versions (exceptions listed below). This means that the new JARs are a 
> drop-in replacement for the old one (but not the other way around) as long 
> as your build does not enable the inliner (Scala-only restriction). It 
> should be noted that Scala 2.11.x is is not binary compatible with Scala 
> 2.10.x, which means that Akka’s binary compatibility property only holds 
> between versions that were built for a given Scala 
> version—akka-actor_2.11-2.4.2-RC1.jar is compatible with 
> akka-actor_2.11-2.3.14.jar but not with akka-actor_2.10-2.3.14.jar.
>
> Binary compatibility is *not* maintained for the following:
>
>- testkits:
>   - akka-testkit
>   - akka-multi-node-testkit
>   - akka-persistence-tck
>   - akka-stream-testkit
>   - akka-http-testkit
>- experimental modules:
>   - akka-persistence-query-experimental
>   - akka-distributed-data-experimental
>   - akka-typed-experimental
>   - akka-http-experimental
>   - akka-http-spray-json-experimental
>   - akka-http-xml-experimental
>   - akka-http-jackson-experimental
>- features, classes, methods that were deprecated in 2.3.0 or earlier 
>and removed in 2.4.x
>- everything marked as INTERNAL API in the JavaDoc.
>
> *Outlook*
>
> The next version will focus on getting the Akka HTTP Java DSL aligned with 
> the Scala DSL—this is the reason for the “experimental” designation that 
> remains on this module for now. Another 

[akka-user] [ANNOUNCE] Akka 2.4.2 including Streams and HTTP Released!

[Konrad Malawski]

*Dear hakkers,*

we—the Akka committers—are proud to announce the FINAL RELEASE of Akka 
2.4.2. The main change in this release is that it includes Streams & HTTP. 
Some of these new modules are still marked experimental due to impending 
API changes that we could not yet finish, in particular these are akka-http 
(the DSLs) and the HTTP marshaller modules for spray-json, jackson, and 
scala-xml. In comparison to Streams & HTTP 2.0 the main changes are:

   - significant performance improvement for HTTP handling, now reaching 
   roughly 75% of Spray’s performance for long-lived HTTP connections—this is 
   not the end of the performance work, we have only just begun (in particular 
   the number of connections per second needs more work)
   - replacement of all uses of the Unit type (represented as BoxedUnit in 
   Java) with the more descriptive typesakka.Done (for signaling successful 
   completion) and akka.NotUsed (for materialization results of stages that 
   do not produce a value)
   - usage of Java 8 types in the Java DSLs: java.util.Optional instead of 
   scala.Option andjava.util.concurrent.CompletionStage instead of 
   scala.concurrent.Future
   
Especially the second and third point mean that porting code from Streams & 
HTTP 2.0 to Akka 2.4.2 will require some mechanical source code changes, 
please refer to the migration guide(for Java 

 and Scala 
)
 
for the details.

But also in the rest of Akka much work was done, with highlights being:

   - fixed a possible replay consistency issue 
    in Akka Persistence and a too 
   tight specification  in the 
   TCK
   - silence heartbeat logging  in 
   ClusterClient and add a notification when it cannot reconnect 
   
   - provide an API for ClusterSharding state retrieval 
   
   - improvements to BackoffSupervisor 
   
   - fix documentation for max-pool-size settings 
    (this was a little 
   unintuitive, makes a good trivia question!)
   - add Java API for ByteString builders 
   
   - fix a race condition that could lead to lost messages in 
   DistributedPubSub 
   - add an ask() variant that can be used without sender(), i.e. by putting 
   the promise reference into the message 
   
   - added Java patterns for CompletionStage, see PatternsCS 
   
   
Closed issues since version 2.4.1 can be found here 
.


*Binary Compatibility*

Akka 2.4.2 is backwards binary compatible with previous 2.4.x and 2.3.x 
versions (exceptions listed below). This means that the new JARs are a 
drop-in replacement for the old one (but not the other way around) as long 
as your build does not enable the inliner (Scala-only restriction). It 
should be noted that Scala 2.11.x is is not binary compatible with Scala 
2.10.x, which means that Akka’s binary compatibility property only holds 
between versions that were built for a given Scala 
version—akka-actor_2.11-2.4.2-RC1.jar is compatible with 
akka-actor_2.11-2.3.14.jar but not with akka-actor_2.10-2.3.14.jar.

Binary compatibility is *not* maintained for the following:

   - testkits:
  - akka-testkit
  - akka-multi-node-testkit
  - akka-persistence-tck
  - akka-stream-testkit
  - akka-http-testkit
   - experimental modules:
  - akka-persistence-query-experimental
  - akka-distributed-data-experimental
  - akka-typed-experimental
  - akka-http-experimental
  - akka-http-spray-json-experimental
  - akka-http-xml-experimental
  - akka-http-jackson-experimental
   - features, classes, methods that were deprecated in 2.3.0 or earlier 
   and removed in 2.4.x
   - everything marked as INTERNAL API in the JavaDoc.

*Outlook*

The next version will focus on getting the Akka HTTP Java DSL aligned with 
the Scala DSL—this is the reason for the “experimental” designation that 
remains on this module for now. Another focus is to further optimize our 
Stream internals and HTTP usage of same so that the performance will be 
comparable to Spray for all cases.

*Credits*

Due to incorporating the full development of Streams & HTTP this release is 
*HUMONGOUS:* 2493 files changed, 179223 insertions, 4238 deletions. Our 
statistics script counts 101 committers!

The full list and detailed stats are in the news item on akka.io here 

[akka-user] Quite stdout when using the TestEventListener?

[Paul Cleary]

I am using the akka testkit TestEventListener to check for exceptions.

I have a number of test cases spread across the application that use it.

My application.conf for src/test looks like:

akka {
 loglevel = "INFO"
 loggers = ["akka.testkit.TestEventListener"]


But now, I get a TON of output in stdout when running my tests.

Is there a way to use the TestEventListener and not have normal logging 
output to stdout when running your tests?

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] [akka-stream] Difference between Source[Future[T]] .mapAsync(n)(f => f) and .flatMapConcat(f => Source.fromFuture(f))?

[Viktor Klang]

If I were you, I'd use parallelism 1, and if that ever became a problem,
I'd revisit that choice.

Focus on the business value and tune when you have something which works :)

On Wed, Feb 17, 2016 at 11:49 AM, Alexey Shuksto  wrote:

> Hello there,
>
> I have `Source[Future[T]]` which I need to transform to `Source[T]`
> somehow.
>
> Obvious choice would be to use `FlowOps.mapAsync(parallelism)(...)` but
> the problem is -- I'm not quite sure how to determine 'right' `parallelism`
> value -- basically I need mapAsync to produce as many futures, as requested
> by downstream.
>
> Recently I found `FlowOps.flatMapConcat(...)` method, and something along
> the `Flow[Future[T]].flatMapConcat(f => Source.fromFuture(f))` looks very
> similar to mapAsync but without parallelism guessing.
>
> Could anyone tell me is there any significant differences between them?
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Cheers,
√

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] [akka-stream] Is there a way to access Configuration from Flow?

[Alexey Shuksto]

Hello there,

Is there a way to access run-time system configuration in 
GraphDSL.create(...) similar to as any Actor could access one via 
`context.system.settings.config`?

I can, possibly do something like `flow.zip(Source.repeat(config))`, but 
maybe there is some approach that will not require to provide ActorSystem 
of Config to graph creation procedure?

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] How to detect sharding start failures and Singleon start failures?

[Marek Żebrowski]

We observe problems with both cluster sharding and cluster singletons.
With sharders - usually problem is corrupted journal that prevents sharding 
coordinator from starting. In our situation easiest thing to do is to 
delete all data from journal and restart it - problem is that I can't find 
a way to detect that situation in a different way than observe logs - I 
can't find any way to detect such failure from the code. It should be 
pretty easy - as usually `akka.cluster.sharding.ShardCoordinator.State` 
throws exception with requirementFailed, but there is no way I can find a 
way to react on that - no easy way to put `supervisorStrategy` for shard 
coordinator or no other way to detect its state.
We can't use `ddata` mode - as current implementation does not work in our 
environment, when we need to scale nodes up and down - as it requires 
majority of nodes to respond, it fails to work even on simplest cases of 
scaling down in a small cluster.

Similar situation applies to cluster singleton - if cluster singleton is 
stuck, there is no way to detect that situation from the code - only by 
observing logs.

Does anybody have experience in handling such situations?
I'm trying to implement some external monitoring for both things, with 
basically sending `Identity` message to actors that are supposed to exist - 
singletons - but it looks like rewriting already existing code inside akka.

Maybe adding such failure detection capabilities to akka (publish event 
bus, adding ability to set supervisor strategy ) is a better approach?

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.