Thank you for the quick response, Andre! After thinking through "The Meat" slide in Dr. Roland Kuhn's fantastic presentation, I was thinking, "just how would the onComplete method for the groupBy know that the individual groupBy flows were complete?" So yes, it makes perfect sense, logically, why this behavior is occurring.
I've commented the ticket https://github.com/typesafehub/activator-akka-stream-scala/issues/18 <https://github.com/typesafehub/activator-akka-stream-scala/issues/18> so perhaps further discussion of mapAsync vs folding futures can be done there. Tim > On Dec 23, 2014, at 13:03, Akka Team <[email protected]> wrote: > > Hi Tim, > > > I have been following this pattern, but am finding that the Future returned > by the foreach on the groupBy completes BEFORE the groupFlows are finished > processing. > > This is completely normal. The master stream finishes as soon as there are no > more gourps, it has nothing to do with group streams being finished (or at > least there is a possible race). In fact every stream finishes as soon as the > last element has been processed. In this case the last element is the last > group, and processing "finishes" from the viewpoint of that stream as soon as > you finished calling foreach on the substream (which is usually earlier when > that foreach itself finishes). > > > Is the Activator example a bad example of how to use streams? > > It is a faulty example it seems, it demonstrates streams properly, but it > demonstrates shutdown order wrongly: shutting down the system happens too > early -- it should happen when all writing streams are finished, not when all > writing streams has been started. I opened a ticket: > https://github.com/typesafehub/activator-akka-stream-scala/issues/18 > <https://github.com/typesafehub/activator-akka-stream-scala/issues/18> > > Or is akka streams misbehaving? > > No, it is not. > > (is it impossible for a group stream to know that it's groupFlows are > completed)? > > Yes, there is a cookbook sample that is similar: > http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-cookbook.html#Implementing_reduce-by-key > > <http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-cookbook.html#Implementing_reduce-by-key> > > Although that recipe is called "reduce-by-key" it follows a similar pattern. > Modification of the activator sample is simple. replace: > > groupFlow.foreach(line => output.println(line)).onComplete(_ => > Try(output.close())) > > with (pseudocode, won't compile) > > groupFlow.foreach(line => output.println(line)).andThen(_ => > Try(output.close())) > > This change causes the above statement to return a Future[Unit] instead of > just Unit. Now on the master stream use > > .mapAsyncUnordered(identity).onComplete(<close input file>) > > These changes basically do the following: The first change makes the stream > of groups become a stream of completion Futures (Future[Unit]), while the > second modification causes the stream to "flatten" these futures by waiting > for them (Unordered since we don't care about the order). Now when this new > stream finishes, it also means that all Futures has been finished, which > means that all "output.close()" has been called (because of the andThen > combinator on the Future), which means that all foreach blocks on the > substreams has been finished. This is exactly what you want. > > Just like it is explained in the recipe you actually want to call: > > .buffer(MaxGroups, > OverflowStrategy.error).mapAsyncUnordered(identity).onComplete(<close input > file>) > > With MaxGroups being large enough to contain all possible groups. The reason > for this is explained in the recipe. > > -Endre -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
