I am trying to implement *Field-aware Factorization Machines* [FFM,
https://www.csie.ntu.edu.tw/~r01922136/libffm ] that use Stochastic
Gradient Descent (SGD). The design I have now is:
*1*. one master actor that spans workers
*2.* one actor that holds model parameters (including big 3dimentional
matrix with weights)
*3.* workers get data from master, and needed parameters and weights from
parameter actor, do some processing and update weights
*NOTE: *I do not care about order of processing and updating.
I think I have already points 1-3, now I needed to feed it with data so I
wanted to have one more actor that would read a file and pass the data to
master actor that then distributes them to workers who are free to work.
That is why I think foreach{thing => doStuff(thing)} is not a good option
for me. I need to be able to use backpressure if computations take too long
so I do not run out of RAM. Also the matrix with weights will probably be
big (tens of GB), so it would be good if I did not have to store too many
rows from file in the memory.
--
Marcin
On Tuesday, 3 January 2017 01:19:54 UTC+1, Konrad Malawski wrote:
>
> That'd work too. It basically will do what a foreach { thing =>
> doStuff(thing) } would do.
> It also somewhat depends what you're doing with these lines - if it's
> heavy or light work etc, then we could give more hints.
>
> Happy hakking
>
> --
> Konrad `ktoso` Malawski
> Akka <http://akka.io> @ Lightbend <http://lightbend.com>
>
> On 3 January 2017 at 00:51:28, Marcin Pękalski ([email protected]
> <javascript:>) wrote:
>
> Thank you Konrad. I still need to get my head around "the stream way" of
> thinking.
>
> Just if someone stumbled upon my post one day I went with
>
> file_source.to(Sink.actorRefWithAck(ModelMaster
> ,onInitMessage = Init
> ,ackMessage = Ack
> ,onCompleteMessage = Complete
> ,onFailureMessage = Status.Failure
> )
> ).run()
>
> and my *ModelMaster* extends *ActorSubscriber*, and after receiving *Init*
> and receiving a message it replies to sender with *Ack*.
>
> For more details see *actorSubscriber* and *actorRefWithAck* in
> http://doc.akka.io/japi/akka/2.4/akka/stream/javadsl/Sink.html
> and examples in test cases
>
>
> https://github.com/akka/akka/blob/448c12d06a62e936c84d5267e221b54813037b9d/akka-stream-tests/src/test/scala/akka/stream/scaladsl/ActorRefBackpressureSinkSpec.scala
>
> --
> Marcin
>
> On Monday, 2 January 2017 10:37:12 UTC+1, Konrad Malawski wrote:
>>
>> There is no "all at once here" it's processing line by line, pulling more
>> data as needed.
>> I don't see why you're worried about the all-at-once – that's exactly
>> what Akka Streams solve, it is bounded memory processing of that file.
>> Amount of data in memory will be chunk-size * number of buffers (between
>> async stages), and thus limited.
>>
>> --
>> Konrad `ktoso` Malawski
>> Akka <http://akka.io> @ Lightbend <http://lightbend.com>
>>
>> On 2 January 2017 at 08:12:44, Marcin Pękalski ([email protected]) wrote:
>>
>>
>> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To post to this group, send email to [email protected]
> <javascript:>.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
