[ 
https://issues.apache.org/jira/browse/PIG-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571233#comment-15571233
 ] 

Alexandre Linte commented on PIG-5042:
--------------------------------------

The full error stacktrace when the jobs are failing is:
{noformat}
16/10/13 09:08:31 ERROR tez.TezJob: Cannot submit DAG
java.lang.RuntimeException: java.io.IOException: The ownership on the staging 
directory hdfs://sandbox/Products/TEZ/staging is not as expected. It is owned 
by hdfs. The directory must be owned by the submitter shfs3453 or by shfs3453
        at 
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:105)
        at 
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:228)
        at 
org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:203)
        at 
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:210)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: The ownership on the staging directory 
hdfs://sandbox/Products/TEZ/staging is not as expected. It is owned by hdfs. 
The directory must be owned by the submitter shfs3453 or by shfs3453
        at 
org.apache.tez.client.TezClientUtils.ensureStagingDirExists(TezClientUtils.java:339)
        at 
org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:467)
        at org.apache.tez.client.TezClient.start(TezClient.java:462)
        at 
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:96)
        ... 8 more
199704 [main] WARN  
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher  - Ooops! Some 
job has failed! Specify -stop_on_failure if you want Pig to stop immediately on 
failure.
{noformat}

> Pig with Tez requires a staging directory with submitter ownership
> ------------------------------------------------------------------
>
>                 Key: PIG-5042
>                 URL: https://issues.apache.org/jira/browse/PIG-5042
>             Project: Pig
>          Issue Type: Bug
>          Components: tez
>    Affects Versions: 0.16.0
>         Environment: Hadoop 2.7.2, Pig 0.16.0, Tez 0.8.4
>            Reporter: Alexandre Linte
>
> With Pig 0.15.0, I have been able to work using the Tez execution engine 
> without issues. After upgrading Pig to 0.16.0, I haven't been able to submit 
> my jobs.
> Since the beginning, the tez-site.xml is configured as below:
> {noformat}
> <property>
>   <description>String value to a file path. The location of the Tez libraries 
> which will be localized for DAGs. This follows the following 
> semantics</description>
>   <name>tez.lib.uris</name>
>   <value>hdfs:///Products/TEZ/tez-0.8.4.tar.gz</value>
> </property>
> <property>
>   <description>String value. Specifies a directory where Tez can create 
> temporary job artifacts.</description>
>   <name>tez.staging-dir</name>
>   <value>/Products/TEZ/staging/</value>
> </property>
> {noformat}
> The staging directory was set up with 777 rights, including a sticky bit:
> {noformat}
> [root@spark01 ~]# hadoop fs -ls /Products/TEZ/
> Found 3 items
> drwxrwxrwt+  - hdfs hadoop          0 2014-11-27 10:31 /Products/TEZ/staging
> -rw-r--r--   3 hdfs hadoop   46518370 2016-09-02 11:06 
> /Products/TEZ/tez-0.8.4.tar.gz
> {noformat}
> Pig 0.16.0 seems to need a staging directory with submitter directory, even 
> if the staging directory has full access to "other" like my previous 
> configuration. 
> This issue looks like a regression to me, compared to Pig 0.15.0. Can you 
> check that point?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to