Hi Avery, I am not preempting tasks out of the giraph pool. I have configured pre-emption so that any job submitted to giraph pool will get its min share. Any suggestion on how to make this work?
Thanks so much in advance. Arun Ramani From: Avery Ching <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, May 7, 2013 7:25 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Giraph and Fair Scheduler Can you disable the preemption for the giraph pool? It's not great to preempt those tasks. Avery On 5/6/13 6:37 PM, Ramani, Arun wrote: Hi, I am running Fair scheduler with many applications in hadoop stack in my cluster (like pig, hive, hbase etc). I have dedicated a pool for Giraph and want to run giraph along with those other applications. I have configured pre-emption and and set the "minsharepreemptiontimeout=5" (sec – for the jobs submitted to this pool to wait to get the min share). I am trying to run giraph in this mode. I see that jobs from other pools are getting pre-empted to give the giraph job's pool its configured min share but my job fails with "Unable to create native thread" error. This same job passes if the slots are available immediately without having to wait for the tasks from other queues to be pre-empted. I also tried to tweak the "giraph.minPercentResponded=50.0f". My Giraph job still fails. Please help in this scenario. Basically, I wanted to know how to configure giraph to wait for a threshold for the slots to be available for it through pre-emption. Thanks Arun Ramani
