[
https://issues.apache.org/jira/browse/YARN-5960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760025#comment-15760025
]
Shaulboo Zhiao commented on YARN-5960:
--------------------------------------
Thank you for you replying . I got the problem ~~~ . Thank you again
> yarn capacity DominantResourceCalculator can work on mapreduce , but got
> error on tez
> --------------------------------------------------------------------------------------
>
> Key: YARN-5960
> URL: https://issues.apache.org/jira/browse/YARN-5960
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
> Affects Versions: 2.7.1
> Environment: HDP 2.4.2.0-258 , hadoop 2.7.1 , hive 1.2 , tez 0.7.0 ,
> ranger 0.5.0
> Reporter: Shaulboo Zhiao
> Priority: Minor
>
> Yestoday ,I configured a queue with 30% resources , 24core CPU & 1000G
> memory . Until then everything is OK，while hql runs over tez engine,
> ranger-hive is on and yarn uses capacity scheduler . After I configured
> 'yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator'
> , hql executed failed .The message below :
> INFO : Session is already open
> INFO : Dag name: select local_code , count(1) fr...local_code(Stage-1)
> INFO : Tez session was closed. Reopening...
> INFO : Session re-established.
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1480664840037_0003)
> INFO : Map 1: -/- Reducer 2: 0/14
> INFO : Map 1: 0/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-2)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-4)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-5)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-16)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-35)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-38)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-44)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-50)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+1,-56)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+1,-59)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+2,-63)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+2,-70)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+5,-77)/36 Reducer 2: 0/14
> INFO : Map 1: 0(+0,-84)/36 Reducer 2: 0/14
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1480664840037_0003_1_00, diagnostics=[Task failed,
> taskId=task_1480664840037_0003_1_00_000029, diagnostics=[TaskAttempt 0
> failed, info=[Container container_e25_1480664840037_0003_01_000034 finished
> with diagnostics set to [Container failed, exitCode=-1000. Application
> application_1480664840037_0003 initialization failed (exitCode=255) with
> output: main : command provided 0
> main : run as user is nobody
> main : requested yarn user is hive
> Requested user nobody is not whitelisted and has id 99,which is below the
> minimum allowed 500
> It seems that yarn run as user hive before , because the ranger is on and ask
> user hive to execut all hql . But I set capacity resouece calculator to
> 'org.apache.hadoop.yarn.util.resource.DominantResourceCalculator' , Yarn
> run task with user 'NOBODY' , that make the hive run failed everytime . I
> change hive execution engine to mr , everything is OK . As you know, I need
> tez to improve performance over mapreduce , so is it the yarn capacity
> problem.
>
> Feedback would be appreciated!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
