Shaulboo Zhiao created YARN-5960:
------------------------------------
Summary: yarn capacity DominantResourceCalculator can work on
mapreduce , but got error on tez
Key: YARN-5960
URL: https://issues.apache.org/jira/browse/YARN-5960
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, yarn
Affects Versions: 2.7.1
Environment: HDP 2.4.2.0-258 , hadoop 2.7.1 , hive 1.2 , tez 0.7.0 ,
ranger 0.5.0
Reporter: Shaulboo Zhiao
Priority: Minor
Yestoday ,I configured a queue with 30% resources , 24core CPU & 1000G memory
. Until then everything is OK，while hql runs over tez engine, ranger-hive
is on and yarn uses capacity scheduler . After I configured
'yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator'
, hql executed failed .The message below :
INFO : Session is already open
INFO : Dag name: select local_code , count(1) fr...local_code(Stage-1)
INFO : Tez session was closed. Reopening...
INFO : Session re-established.
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1480664840037_0003)
INFO : Map 1: -/- Reducer 2: 0/14
INFO : Map 1: 0/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-2)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-4)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-5)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-16)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-35)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-38)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-44)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-50)/36 Reducer 2: 0/14
INFO : Map 1: 0(+1,-56)/36 Reducer 2: 0/14
INFO : Map 1: 0(+1,-59)/36 Reducer 2: 0/14
INFO : Map 1: 0(+2,-63)/36 Reducer 2: 0/14
INFO : Map 1: 0(+2,-70)/36 Reducer 2: 0/14
INFO : Map 1: 0(+5,-77)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-84)/36 Reducer 2: 0/14
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1480664840037_0003_1_00, diagnostics=[Task failed,
taskId=task_1480664840037_0003_1_00_000029, diagnostics=[TaskAttempt 0 failed,
info=[Container container_e25_1480664840037_0003_01_000034 finished with
diagnostics set to [Container failed, exitCode=-1000. Application
application_1480664840037_0003 initialization failed (exitCode=255) with
output: main : command provided 0
main : run as user is nobody
main : requested yarn user is hive
Requested user nobody is not whitelisted and has id 99,which is below the
minimum allowed 500
It seems that yarn run as user hive before , because the ranger is on and ask
user hive to execut all hql . But I set capacity resouece calculator to
'org.apache.hadoop.yarn.util.resource.DominantResourceCalculator' , Yarn run
task with user 'NOBODY' , that make the hive run failed everytime . I change
hive execution engine to mr , everything is OK . As you know, I need tez to
improve performance over mapreduce , so is it the yarn capacity problem.
Feedback would be appreciated!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
