Hey,
currently our lambda architecture is designed this way :
Tree based hierarchy of View Materializer Actors which is mostly done due
to Actor supervision. Each Materializer Actor triggers an akka-stream that
builds the resulting View. This design works but it leads to complex actors
because each following Materializer must be initialized from its parent
Materializer - that's how Actor hierarchy works. Overall it is not easy to
design a complex lambda architecture tree/flow.
I was thinking about rewriting this to one big Akka Stream where original
materializer Actor hierarchy would become an akka-stream Graph.
Materializer Actors would turn into Flows and these Flows would spin an
akka-stream building a View.
Think of it this way :
start ~> FtpToS3 ~> S3ToElasticsearch ~> ElasticSearchToS3 ~>
SparkJobFoo ~> finish
~> S3ToS3 ~> S3ToFs ~>
~> SparkJobBar ~>
Questions :
1) Is it OK to have an akka-stream Graph that has almost no throughput? In
fact there would be just a single element coming through it. It would just
collect statistics about performance and status.
2) I'm not sure how to deal with the wrapping of self-contained
akka-streams into akka-stream Materializer Flows... Would it be ok if was
done simply this way :
S3ToElasticsearchFlow().mapAsync(1) { x => SomeSource()......runWith(
SomeSink()) }
Or is there a better way?
I've never really thought of akka-streams as just a tool for building a
Graph... I've always thought of it as a
high-throughput-asynchronous-boundary-crossing tool. But now I can see use
cases that have nothing to do with it.
Thanks !!
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
