Hello,
Not sure why you need "massive over-provision". Wouldn't you just need
about twice as many store actors than read actors? So maybe three times as
many would solve your problem?
You could (should?) monitor resources (e.g. free heap space) or number of
objects waiting for storage and tell the read actor to suspend if you
approach problematic load and resume after a certain time, if the situation
has improved.
Best, Oliver
On Thursday, October 2, 2014 11:22:00 AM UTC-4, tigerfoot wrote:
>
> Hello,
>
> I have a system that does the following: read/parse from a huge (150+GB)
> file of data, run sanity/cleaning operations, and store the cleaned records
> in my DB.
> I've got this configured using actors such that each time I read an object
> from the input file I send a message to the cleaner actor. When that
> object is cleaned I send a
> message to the storage actor. (The real application is more complex but
> this example illustrates the problem.)
>
> read --> clean --> store
>
> Simple on the surface. My problem is that each of these operations takes
> different amounts of time. Reading takes say 2 time unit, leaning takes 1,
> and storage takes 4.
> That means I've got a bottleneck on storage and even if I use a
> round-robin dispatcher I still run the risk of overflow and data loss on
> the mailbox on the storage actor.
>
> Other than massive over-provision of storage actors, is there another way?
> Can something within the actor ecosystem provide back-pressure so that I
> can slow down
> the reader just enough to keep the pipeline clean?
>
> Thanks,
> Greg
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
