To me, this isn't the correct solution. Passing Source(data) around seems wrong, it would seem dangerous for large files and impossible for remote actors if the source were pointer based. The tl;dr is that IMHO the correct solution, where you process indexing responses in a flow based way, is difficult to get done correctly right now. I'm waiting for this missing component: https://github.com/akka/akka/issues/16416.
Read on for what turned into a pretty long winded brain dump... I think of Actors as little (possibly stateful) message processors. They react to messages, and sometimes produce directed response messages. These messages generally ought to be small and immutable. I see Streams and Flows as higher level constructs. As if someone stepped back, looked at all the little discrete message processors that he had and wanted to model and materialize a "processing engine" out of those parts. In your case I'd probably break things into a few components. Files, IndexingManager, FileToLinesGenerator, any number of LineToX mappers along with XtoY mappers, YtoZ mappers, etc. Finish things off with an Indexer component. You want to get files indexed. That overall responsibility would be owned by the IndexingManager. He'd be responsible for constructing the high level materialized flow, from the building blocks at hand. Maybe he's an actor that gets a NewFile(location) message whenever some file shows up somewhere. He than sets up and executes the indexing flow. I'd probably use a scala.io.Source to get an iterator over that file content. Source(file).getLines() or zipWithIndex() if you want line number information as well (I would). Construct your FlowGraph with whatever intermediate steps you like, and eventually get a small message over to your Indexer. Maybe that message looks like IndexingRequest(fileMetadata, lineNumber, line). It then indexes this bite sized message (also able to be sent across remote boundaries). Eventually the flow completes and your IndexingManager does some final book keeping and declares victory. This is how I'd model an overall indexing system. And it would probably work reasonably well in one direction, from FileData -> Indexing. But we want to have the Indexer tell someone about the indexing results. For me, I'm waiting for https://github.com/akka/akka/issues/16416 to come out. It should make custom request/response protocols much easier to model. Until then, I'm probably not going to mess with cyclic flow graphs with Akka Streams. In the mean time, if I needed custom request/response logic I'd build non-Streams logic into my IndexingManager. I'd have the IndexingManager manually buffer requests/responses and simply have some basic ack-based back-pressure (send a few indexing requests blind, and then wait for responses to show up before I sent more requests). This assumes that the IndexingManager is going to do something special with responses. If all it's going to do is log errors, or send them to a queue for followup and possible reprocessing, the Indexer could do that and you can have a unidirectional flow managed by Akka Steams again. On 2nd thought, the blind send of IndexingRequests is probably poor form, knowing a bit about how Streams works I'd probably start the conversation off with a single IndexingSessionRequest that basically says, "hey I want to send stuff, how many can you handle". The response can help the IndexingManager set internal buffers as well. The Indexer can respond with IndexingSessionResponse(send me 5) or IndexingSessionResponse(too busy, ask again in 5.seconds). -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
