Re: Channel indexing with pointwise connection pattern

2014-11-27 Thread Stephan Ewen
Our implication so far was that forwarding means evenly scattering over successors - a balanced load being the important goal. If you find different requirements in streaming, you could define a new type of selector. On Thu, Nov 27, 2014 at 11:02 AM, Gyula Fora wrote: > Thanks Stephan, > > So I

Re: Channel indexing with pointwise connection pattern

2014-11-27 Thread Gyula Fora
Thanks Stephan, So I took a quick look at the ChannelSelectors the batch api uses and I see that for Forward strategy uses round-robin. My question was aimed exactly to avoid having to do this. Isn’t this sub-optimal? Maybe we could pass the channel info to the channel selector, so it can make

Re: Channel indexing with pointwise connection pattern

2014-11-27 Thread Stephan Ewen
This is a bit tricky, since the new scheduling is more flexible... Assume we have a PointWise connection with two receiving tasks per sending task: outgoing channels 0 and 1. When scheduling in the most basic mode, the receivers can go anywhere, but the schedule will try to give them a slot on th

Channel indexing with pointwise connection pattern

2014-11-26 Thread Gyula Fora
Hey, I was hoping that someone can answer this right away without me having to dig through all the code :) How does the channel indexing go when more then one consumer subtask is connected to an intermediate dataset in the pointwise pattern? I am trying to figure out which one is the “in-memor