Properly redistributing the data partitions shared by 20 workers if the next superstep will run on only 15 workers (for example) would be prohibitive performance wise for most use cases I think?
In the "Pure YARN" Giraph profile this functionality and many similar things will be fairly easy to implement, but will have the same limitations/practical constraints in practice, so may not be a great idea. Adding more workers as they become available or are required for load balancing in the middle of a calculation might be more worthwhile, but also has not been implemented yet in Giraph. On Wed, Oct 16, 2013 at 11:24 AM, Manuel Lagang <[email protected]>wrote: > Does anyone have any thoughts on how to have Giraph not require a fixed > amount of workers, but rather be able to start a superstep with a possibly > smaller number of workers? > > > On Thu, Oct 10, 2013 at 4:12 PM, Manuel Lagang <[email protected]>wrote: > >> I'm trying to understand the meaning of the 3 parameters to >> GiraphConfiguration.setWorkerConfiguration: minWorkers, maxWorkers, and >> minPercentResponded. I want my Giraph jobs to co-exist nicely with other >> jobs in the cluster, and it's not always the case that I can get a fixed >> number of map slots for my job before the job times out. Thus, I would like >> the job to be able to start with a possibly smaller set of workers, ideally >> being able to pick up workers in later supersteps. >> >> So I tried <minWorkers=10,maxWorkers=50,minPercentResponded=100>, >> expecting this to mean that it can start with 10 workers provided that 100% >> of those workers respond. But this setting ends up again waiting for all 50 >> workers. >> >> Then I tried <minWorkers=10,maxWorkers=50,minPercentResponded=20>, >> expecting that minPercentResponded was just a redundant expression of >> minWorkers/maxWorkers. But this setting leads to null pointer exceptions in >> org.apache.giraph.comm.SendCache.removeWorkerData(SendCache.java:199). >> >> So I must be confused about the meaning of these variables, and what the >> legal values are. Can anyone enlighten me on how (if possible) I can get >> the behavior I want? >> >> Thanks, >> Manuel Lagang >> > >
