Hi Michael, What you are seeing is right. Helix as of now does not consider balancing across resources. This is something we would love to have but never got to it.
One workaround is the set max partitions per node constraint. For example, if you set max partitions=1 then each node should have only one task. But this has the drawback that if 1 node fails, it won't assign that task to any of the remaining nodes unless you remove the constraint after the tasks are evenly distributed. thanks, Kishore G On Thu, Nov 10, 2016 at 3:13 PM, Michael Craig <[email protected]> wrote: > This is related to a use case mentioned in a prior thread: > > > .. manage a set of ad-hoc tasks across a cluster of machines. Each task > is a separate resource with 1 partition and 1 replica, intended to run on > exactly 1 node. > > Can Helix balance many small resources like this? E.g. 3 resources {R1, > R2, R3} each with 1 partition and 1 replica, on 3 nodes: > > N1: {R1}, N2: {R2}, N3: {R3} > > Or does Helix balance the resources independently, putting all partitions > on the same node? I.e. > > N1: {R1, R2, R3}, N2: {}, N3: {} > > I've observed the latter behavior, though I'm wondering if I'm misusing > Helix. >
