Hi guys, I am trying to run an experiment where I want the reducers to be forced executing on a specific host. The simplest way I found it is to modify in RMContainerAllocator such that whenever there is an addContainerReq(ContainerRequest) for a reducer, I simply do: addResourceRequest(req.priority, host_X_name, req.capability) and return, instead of setting the resourceName to ResourceRequest.ANY.
Unfortunately when I pass such a ResourceRequest for reducers through a makeRemoteRequest, the RM crashes, giving an ugly exception like: FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549) at java.lang.Thread.run(Thread.java:722) Do you have other suggestion how I can enforce reducers execution on a certain host ? Or do you know what I am doing wrong here ? Thanks, robert
