Re: [akka-user] Re: [akka-stream] How to implement Spark's cogroup/join operation

2016-10-07 Thread Akka Team
If you groupBy with an upper bound of substreams at Int.MaxValue you can at some point have more than 256 substreams, which is what you limit mergeSubstreamsWithParallelism with, this means that it will backpressure indefinitely (as none of the substreams will complete until upstream completes and

[akka-user] Re: [akka-stream] How to implement Spark's cogroup/join operation

2016-09-29 Thread Jakub Liska
I'm trying to figure out why this is hanging/idling indefinitely : Source.fromIterator(() => Iterator.from(0).take(500).map(_ -> 1)) .groupBy(Int.MaxValue, _._1) .mergeSubstreamsWithParallelism(256) .runWith(Sink.seq) This is the only way how to avoid instantiating ridiculous amounts of