Can you check the logs of the failed task and report what the error is?
Avery
On 5/7/13 2:26 PM, Ramani, Arun wrote:
Hi Avery,
I am setting "minsharepreemptiontimeout" to 5 sec and my Giraph job
could not even wait for 5 secs to get its slots. Let me explain the
scenario below:
Assume, Cluster capacity is 150
Queue A (min share –10 maps) - I submit a sleep job with 100 map
tasks. Cluster is empty, and hence the first job submitted to Queue A
will take the entire 100 map tasks.
Queue B (Giraph pool with min share – 140 maps) - Now my job 1 is
running with 100 tasks occupied. I submit a giraph shortestpathfirst
example job with 100 workers to Queue B. Queue B has
"minsharepreemptiontimeout" to 5 sec". So, it will first schedule 50
tasks since first job only took 100 tasks and cluster's capacity is
150. Meanwhile, in 5 sec, 50 more tasks would be preempted from Queue
A and would be given to Giraph Job. I see this happening, however, the
job fails with "Unable to create native thread error"
Please let me know if "giraph.maxMasterSuperstepWaitMsecs" will help
in this scenario.
Thanks so much
Arun Ramani
From: Avery Ching <[email protected] <mailto:[email protected]>>
Date: Tuesday, May 7, 2013 2:19 PM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Cc: "Ramani, Arun(aramani)" <[email protected]
<mailto:[email protected]>>
Subject: Re: Giraph and Fair Scheduler
Oh, I see. You can change the timeout of how long the giraph job
waits for tasks before giving up. Try setting
giraph.maxMasterSuperstepWaitMsecs to a higher number. The default is
10 minutes.
Avery
On 5/7/13 2:10 PM, Ramani, Arun wrote:
Hi Avery,
I am not preempting tasks out of the giraph pool. I have configured
pre-emption so that any job submitted to giraph pool will get its min
share. Any suggestion on how to make this work?
Thanks so much in advance.
Arun Ramani
From: Avery Ching <[email protected] <mailto:[email protected]>>
Reply-To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Date: Tuesday, May 7, 2013 7:25 AM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: Re: Giraph and Fair Scheduler
Can you disable the preemption for the giraph pool? It's not great
to preempt those tasks.
Avery
On 5/6/13 6:37 PM, Ramani, Arun wrote:
Hi,
I am running Fair scheduler with many applications in hadoop stack
in my cluster (like pig, hive, hbase etc). I have dedicated a pool
for Giraph and want to run giraph along with those other
applications. I have configured pre-emption and and set the
"minsharepreemptiontimeout=5" (sec – for the jobs submitted to this
pool to wait to get the min share).
I am trying to run giraph in this mode. I see that jobs from other
pools are getting pre-empted to give the giraph job's pool its
configured min share but my job fails with "Unable to create native
thread" error. This same job passes if the slots are available
immediately without having to wait for the tasks from other queues
to be pre-empted. I also tried to tweak the
"giraph.minPercentResponded=50.0f". My Giraph job still fails.
Please help in this scenario.
Basically, I wanted to know how to configure giraph to wait for a
threshold for the slots to be available for it through pre-emption.
Thanks
Arun Ramani
