If you groupBy with an upper bound of substreams at Int.MaxValue you can at some point have more than 256 substreams, which is what you limit mergeSubstreamsWithParallelism with, this means that it will backpressure indefinitely (as none of the substreams will complete until upstream completes and that is now being back pressured).
5 million tiny subflows with minimal processing each will be a lot of overhead for materializing those substreams and will take a lot of memory in addition to the large amount of memory you will need for actually collect your values per key in memory. GroupBy is likely not a good option for you to achieve this, I'd say a custom graph stage would be the way to go if you need to do it with streams, or just folding over some suitable datastructure where you can collect elements per key. -- Johan Akka Team On Thu, Sep 29, 2016 at 11:03 AM, Jakub Liska <[email protected]> wrote: > I'm trying to figure out why this is hanging/idling indefinitely : > > Source.fromIterator(() => Iterator.from(0).take(500).map(_ -> 1)) > .groupBy(Int.MaxValue, _._1) > .mergeSubstreamsWithParallelism(256) > .runWith(Sink.seq) > > This is the only way how to avoid instantiating ridiculous amounts of sub > streams. > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
