Hi,
I have a sample akka stream code where my source is a preloaded .5 Gb of
data from disk which is split into lines. Each line is about 2.5Kb.
I create a runnable graph where i add processing parallelism by doing a
balance ~> processing flow ~> merge before it goes into a dummy sink.
The problem is, what ever the value i set for parallelism(split 'N' ways
and merge), the processing time remain the same. ~22 secs.
I varied the parallelism from 1 to 8 and got the same running time and my
CPU usage never goes more than a core.
I tried increasing the initial and max buffer but it didn't help. I also
created my own dispatcher and tried using it in ActorMaterializerSettings
but still got the same result.
I am not sure what I am doing wrong here.
I am running this test on a quad core mac book pro and with
-Xms5G
-Xmx5G
I have attached the code below.
I am using "com.typesafe.akka" %% "akka-stream" % "2.4.2-RC2"
I want to process through the .5 Gb of data as soon as possible using all
my CPU which is not happening now.
I would really appreciate if someone can help me identify the performance
problem.
Thanks,
Vishnu.
object StreamTest2 extends App{
implicit val system = ActorSystem("indexer")
implicit val mat = ActorMaterializer()
val source =
Source(scala.io.Source.fromFile("/tmp/collection512mb").getLines().toList)
val parallelism = 8
val graph = RunnableGraph.fromGraph(GraphDSL.create(Sink.ignore) { implicit
builder =>
(snk) =>
import GraphDSL.Implicits._
val balance = builder.add(Balance[String](parallelism))
val merge = builder.add(Merge[Any](parallelism))
val flow = Flow[String].map { str =>
str.split(" ").groupBy(identity).map(a => (a._1, a._2.length))
}
source ~> balance.in
for (i <- 0 until parallelism) {
balance.out(i) ~> flow ~> merge.in(i)
}
merge.out ~> snk.in
ClosedShape
})
val start = System.currentTimeMillis()
val result = graph.run()
implicit val ec = system.dispatcher
result.onComplete { _ =>
println(s"Done ${(System.currentTimeMillis() - start) / 1000}s.")
system.terminate()
}
}
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
