Hello! I think you will have to do it semi-manually: on every node you should know how many resources are available, don't allow jobs to overconsume.
You could try to use LoadBalancingSpi but as far as I have heard, it is not trivial. Ignite it not a scheduler, so we don't have any built-ins for resource control. Regards, -- Ilya Kasnacheev чт, 29 авг. 2019 г. в 14:27, Pascoe Scholle <[email protected]>: > Hi, > > I have a question regarding the workings of the load balancer and running > prcesses which are outside the jvm. > > We have a python commandline tool which is used for processing big data. > The tool is highly optimized and is able to instantly load data into ram. > Some data can be as large as 20 Gb. > > When sending multiple jobs each triggering their own python process, I do > not want this to occur on the same machine, is there any way we can use the > load balancer to ensure that all jobs are evenly distributed or possibly > restrict certain jobs to a node running on a desktop we know has enough > memory available. > > For example I have two machines one which has 32 Gb of ram and the second > which has 64 Gb, one ignite node per machine. Three jobs are sent using > ComputeTaskContinuousMapper. The 32Gb machine received two tasks and > obviously froze up. Having some way of ensuring that two jobs are sent to > the machine with more memory would be really helpful. > > Thanks and kind regards, > Pascoe >
