Not sure that load balancing is what is needed in this case. The problem is to make sure that at any time tasks do not consume more resources than the available ones. Thus, a collision spi is the right strategy to handle that
I've ended up implementing a custom collision spi delegating the job stealing to the default implementation. It looks working fine. You can find the code here: https://github.com/nextflow-io/nextflow/blob/master/subprojects/nxf-ignite/src/main/nextflow/collision/CustomStealingCollisionSpi.groovy#L60 Best, Paolo On Fri, Jan 29, 2016 at 11:26 PM, vkulichenko <[email protected] > wrote: > Paolo, > > This actually sounds like a use case for load balancing [1]. In particular, > I think you can use AdaptiveLoadBalancingSpi for what you described, but > you > will have to provide your own implementation of AdaptiveLoadProbe, because > we don't currently have the one based on memory utilization. > > Will this work for you? > > [1] https://apacheignite.readme.io/docs/load-balancing > > -Val > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Is-it-safe-to-modify-the-activeJobs-and-waitingJobs-collections-in-the-CollisionContext-tp2758p2779.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
