As always, the right answer depends on your requirements. In our case, we
have taken the following approach.
Our streams process data in a strongly ordered way, but we are able to
create many runnable graphs since this ordered processing correspond to a
relatively small subset of our overall data. That is, each external entity
generates a strongly ordered sequence of events, but we have many external
entities.
At the end of each of these runnable graphs is a flow to group the message
to be saved, and then a mapAsync(1){...} that actually does the saving of
the batches. It is configured to run on the database's fixed size
dispatcher to insure these writes and other reads to the same database in
the application cannot overwhelm the database. Within this mapAsync, we
save both the batched data and a high water mark so a rerun of the flow (an
ETL style job) can pick up where it left off. The mapAsync can return
statistical info, like number of records saved, which can then be
aggregated into the actual sink. This can be logged as an indicator of
progress.
In our case, we are using a relational database.
There is another graph that wraps the above described worker graph that
decides what external entities should be processed, and how many should be
run concurrently, isolating failures between them and reporting accordingly.
This has worked well for us. Hopefully these ideas may help you as well.
- James
On Jul 15, 2016 3:10 AM, "AWel" <andy.zelin...@wel.io> wrote:
> Hello,
>
> high level requirement: need to save stream job results
>
> What is the current status/best practices for using Databases as a Sink
> for Akka Stream jobs.?
>
> Theres so much information out there on the read side (Slick 3), but for
> the writing side, there is not much at all.
>
> Do I need to be looking at write-optimized stores like Cassandra? Has
> anyone had success sinking to an RDBMS via slick?
>
> I could even adjust and save to file with Streaming IO, but theres not
> much info on that either. all thoughts appreciated
>
>
> Thanks!
>
> -Andy
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
