Error while running Hive queries over tez
Hi, I am running Hive with Tez. When I run Hive query below is the exception i am facing. 2015-06-24 05:13:15,240 ERROR [main]: exec.Task (TezTask.java:execute(184)) - Failed to execute tez graph. java.io.IOException: Previous writer likely failed to write hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0. Failing because I am unlikely to write too. at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:803) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:228) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:245) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604) Please help me out what can be the issue. When i do fresh setup, Job runs only first time after that for each Hive query it shows same error. Thanks.
Re: Error while running Hive queries over tez
Did you add file/jar/archive after the first query ? It seems exception happens
when you add resources.
You can check the hive log for more info. By default hive.log is located in
/tmp/${user}/hive.log
Best Regard,
Jeff Zhang
From: amit kumar aks...@gmail.commailto:aks...@gmail.com
Reply-To: user@tez.apache.orgmailto:user@tez.apache.org
user@tez.apache.orgmailto:user@tez.apache.org
Date: Wednesday, June 24, 2015 at 8:19 PM
To: user@tez.apache.orgmailto:user@tez.apache.org
user@tez.apache.orgmailto:user@tez.apache.org
Subject: Error while running Hive queries over tez
Hi,
I am running Hive with Tez. When I run Hive query below is the exception i am
facing.
2015-06-24 05:13:15,240 ERROR [main]: exec.Task (TezTask.java:execute(184)) -
Failed to execute tez graph.
java.io.IOException: Previous writer likely failed to write
hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0.
Failing because I am unlikely to write too.
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:803)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:228)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:245)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
Please help me out what can be the issue. When i do fresh setup, Job runs only
first time after that for each Hive query it shows same error.
Thanks.
Re: Error while running Hive queries over tez
A couple of things to do:
1) (optional) For tez.lib.uris, set it to ${fs.defaultFS}/apps/tez/tez.tar.gz”
- this tez.tar.gz should come from tez-dist/target/. Given that your basic job
is working, you can ignore this for now but it is the recommended way to deploy
tez.
2) On a fresh setup, do a hadoop dfs -mkdir -p /tmp/hive and set permissions
to 777 for both /tmp and /tmp/hive before running any job. You should probably
check whether your setup is using a different dfs.umask that is causing write
permissions later on.
thanks
— Hitesh
On Jun 24, 2015, at 9:20 PM, amit kumar aks...@gmail.com wrote:
Below is the tez-site.xml
configuration
property
nametez.lib.uris/name
value${fs.defaultFS}/apps/tez-0.7.0,${fs.defaultFS}/apps/tez-0.7.0/lib//value
/property
/configuration
I ran OrderWordCount job and that run successfully, only with hive queries i
am facing this error.
Please let me know if i missing something.
Thanks.
On Thu, Jun 25, 2015 at 4:04 AM, Jianfeng (Jeff) Zhang
jzh...@hortonworks.com wrote:
Did you configure tez.lib.uris in tez-site.xml and put it under classpath ?
It looks like you didn’t configure it.
Best Regard,
Jeff Zhang
From: amit kumar aks...@gmail.com
Reply-To: user@tez.apache.org user@tez.apache.org
Date: Thursday, June 25, 2015 at 11:58 AM
To: user@tez.apache.org user@tez.apache.org
Subject: Re: Error while running Hive queries over tez
No, I didn't add anything after first query, its like the hive query runs on
fresh cluster and after that below error is shown for all queries. Below are
the hive logs:
2015-06-24 05:12:50,163 INFO [main]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(108)) - PERFLOG method=task.TEZ.Stage-1
from=org.apache.hadoop.hive.ql.Driver
2015-06-24 05:12:50,163 INFO [main]: ql.Driver
(SessionState.java:printInfo(824)) - Launching Job 1 out of 1
2015-06-24 05:12:50,163 INFO [main]: ql.Driver
(Driver.java:launchTask(1602)) - Starting task [Stage-1:MAPRED] in serial mode
2015-06-24 05:12:50,163 INFO [main]: tez.TezSessionPoolManager
(TezSessionPoolManager.java:canWorkWithSameSession(217)) - The current user:
amitkumar, session user: amitkumar
2015-06-24 05:12:50,163 INFO [main]: tez.TezSessionPoolManager
(TezSessionPoolManager.java:canWorkWithSameSession(234)) - Current queue name
is null incoming queue name is null
2015-06-24 05:12:50,190 INFO [main]: ql.Context
(Context.java:getMRScratchDir(266)) - New scratch dir is
hdfs://localhost:9000/tmp/hive/amitkumar/5dcd0855-107b-471e-8ae6-a332c8b7c919/hive_2015-06-24_05-12-50_013_722448017482877635-1
2015-06-24 05:12:50,204 INFO [main]: exec.Task
(TezTask.java:updateSession(236)) - Session is already open
2015-06-24 05:12:50,207 INFO [main]: tez.DagUtils
(DagUtils.java:localizeResource(951)) - Localizing resource because it does
not exist: file:/home/amitkumar/Apache_tez/TEZ/tez-0.7.0 to dest:
hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0
2015-06-24 05:12:50,212 INFO [main]: tez.DagUtils
(DagUtils.java:localizeResource(955)) - Looks like another thread is writing
the same file will wait.
2015-06-24 05:12:50,212 INFO [main]: tez.DagUtils
(DagUtils.java:localizeResource(962)) - Number of wait attempts: 5. Wait
interval: 5000
2015-06-24 05:13:15,239 ERROR [main]: tez.DagUtils
(DagUtils.java:localizeResource(978)) - Could not find the jar that was being
uploaded
2015-06-24 05:13:15,240 ERROR [main]: exec.Task (TezTask.java:execute(184)) -
Failed to execute tez graph.
java.io.IOException: Previous writer likely failed to write
hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0.
Failing because I am unlikely to write too.
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:803)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:228)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:245)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201
Re: Error while running Hive queries over tez
)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2015-06-24 21:41:16,712 INFO [main]: client.TezClient
(TezClient.java:stop(514)) - Could not connect to AM, killing session via
YARN, sessionName=HIVE-25e6f3e0-b913-46c8-9d87-83962f1dc073,
applicationId=application_1435118621366_0002
2015-06-24 21:41:18,457 INFO [main]: impl.YarnClientImpl
(YarnClientImpl.java:killApplication(364)) - Killed application
application_1435118621366_0002
2015-06-24 21:41:18,457 INFO [main]: tez.TezSessionState
(TezSessionState.java:open(142)) - User of session id
25e6f3e0-b913-46c8-9d87-83962f1dc073 is amitkumar
2015-06-24 21:41:18,463 INFO [main]: tez.DagUtils
(DagUtils.java:localizeResource(951)) - Localizing resource because it does
not exist: file:/home/amitkumar/Apache_tez/TEZ/tez-0.7.0 to dest:
hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0
2015-06-24 21:41:18,467 INFO [main]: tez.DagUtils
(DagUtils.java:localizeResource(955)) - Looks like another thread is
writing the same file will wait.
2015-06-24 21:41:18,467 INFO [main]: tez.DagUtils
(DagUtils.java:localizeResource(962)) - Number of wait attempts: 5. Wait
interval: 5000
2015-06-24 21:41:43,480 ERROR [main]: tez.DagUtils
(DagUtils.java:localizeResource(978)) - Could not find the jar that was
being uploaded
2015-06-24 21:41:43,481 ERROR [main]: exec.Task (TezTask.java:execute(184))
- Failed to execute tez graph.
java.io.IOException: Previous writer likely failed to write
hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0.
Failing because I am unlikely to write too.
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860)
I am not getting it, why session is getting closed.
Thanks
On Wed, Jun 24, 2015 at 2:23 PM, Jianfeng (Jeff) Zhang
jzh...@hortonworks.com wrote:
Did you add file/jar/archive after the first query ? It seems exception
happens when you add resources.
You can check the hive log for more info. By default hive.log is located
in /tmp/${user}/hive.log
Best Regard,
Jeff Zhang
From: amit kumar aks...@gmail.com
Reply-To: user@tez.apache.org user@tez.apache.org
Date: Wednesday, June 24, 2015 at 8:19 PM
To: user@tez.apache.org user@tez.apache.org
Subject: Error while running Hive queries over tez
Hi,
I am running Hive with Tez. When I run Hive query below is the exception
i am facing.
2015-06-24 05:13:15,240 ERROR [main]: exec.Task
(TezTask.java:execute(184)) - Failed to execute tez graph.
java.io.IOException: Previous writer likely failed to write
hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0.
Failing because I am unlikely to write too.
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860)
at
org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:803)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:228)
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:245)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
Please help me out what can be the issue. When i do fresh setup, Job runs
only first time after that for each Hive query it shows same error.
Thanks.
Re: Error while running Hive queries over tez
I have provided the settings in tez-site.xml. I run the OrderWordCount job and that runs successfully. The issue happens only when i fire a Hive query. Thanks. On Wed, Jun 24, 2015 at 12:34 PM, Rajesh Balamohan rajesh.balamo...@gmail.com wrote: Can you provide tez.lib.uris in your setting and the location of tez deployed in HDFS? ~Rajesh.B On Wed, Jun 24, 2015 at 5:49 PM, amit kumar aks...@gmail.com wrote: Hi, I am running Hive with Tez. When I run Hive query below is the exception i am facing. 2015-06-24 05:13:15,240 ERROR [main]: exec.Task (TezTask.java:execute(184)) - Failed to execute tez graph. java.io.IOException: Previous writer likely failed to write hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0. Failing because I am unlikely to write too. at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:803) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:228) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:245) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604) Please help me out what can be the issue. When i do fresh setup, Job runs only first time after that for each Hive query it shows same error. Thanks. -- ~Rajesh.B
Re: Error while running Hive queries over tez
Did you configure tez.lib.uris in tez-site.xml and put it under classpath ? It looks like you didn’t configure it. Best Regard, Jeff Zhang From: amit kumar aks...@gmail.commailto:aks...@gmail.com Reply-To: user@tez.apache.orgmailto:user@tez.apache.org user@tez.apache.orgmailto:user@tez.apache.org Date: Thursday, June 25, 2015 at 11:58 AM To: user@tez.apache.orgmailto:user@tez.apache.org user@tez.apache.orgmailto:user@tez.apache.org Subject: Re: Error while running Hive queries over tez No, I didn't add anything after first query, its like the hive query runs on fresh cluster and after that below error is shown for all queries. Below are the hive logs: 2015-06-24 05:12:50,163 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - PERFLOG method=task.TEZ.Stage-1 from=org.apache.hadoop.hive.ql.Driver 2015-06-24 05:12:50,163 INFO [main]: ql.Driver (SessionState.java:printInfo(824)) - Launching Job 1 out of 1 2015-06-24 05:12:50,163 INFO [main]: ql.Driver (Driver.java:launchTask(1602)) - Starting task [Stage-1:MAPRED] in serial mode 2015-06-24 05:12:50,163 INFO [main]: tez.TezSessionPoolManager (TezSessionPoolManager.java:canWorkWithSameSession(217)) - The current user: amitkumar, session user: amitkumar 2015-06-24 05:12:50,163 INFO [main]: tez.TezSessionPoolManager (TezSessionPoolManager.java:canWorkWithSameSession(234)) - Current queue name is null incoming queue name is null 2015-06-24 05:12:50,190 INFO [main]: ql.Context (Context.java:getMRScratchDir(266)) - New scratch dir is hdfs://localhost:9000/tmp/hive/amitkumar/5dcd0855-107b-471e-8ae6-a332c8b7c919/hive_2015-06-24_05-12-50_013_722448017482877635-1 2015-06-24 05:12:50,204 INFO [main]: exec.Task (TezTask.java:updateSession(236)) - Session is already open 2015-06-24 05:12:50,207 INFO [main]: tez.DagUtils (DagUtils.java:localizeResource(951)) - Localizing resource because it does not exist: file:/home/amitkumar/Apache_tez/TEZ/tez-0.7.0 to dest: hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0 2015-06-24 05:12:50,212 INFO [main]: tez.DagUtils (DagUtils.java:localizeResource(955)) - Looks like another thread is writing the same file will wait. 2015-06-24 05:12:50,212 INFO [main]: tez.DagUtils (DagUtils.java:localizeResource(962)) - Number of wait attempts: 5. Wait interval: 5000 2015-06-24 05:13:15,239 ERROR [main]: tez.DagUtils (DagUtils.java:localizeResource(978)) - Could not find the jar that was being uploaded 2015-06-24 05:13:15,240 ERROR [main]: exec.Task (TezTask.java:execute(184)) - Failed to execute tez graph. java.io.IOException: Previous writer likely failed to write hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0. Failing because I am unlikely to write too. at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:979) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:860) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:803) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:228) at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:245) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:201) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:153) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:364) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:712) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:631) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:570) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 2015-06-24 05:13:15,240 ERROR [main]: ql.Driver (SessionState.java:printError(833)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask On Wed, Jun 24, 2015 at 2:23 PM, Jianfeng (Jeff) Zhang jzh...@hortonworks.commailto:jzh
Re: Error while running Hive queries over tez
Hey Hitesh, I feel the same, issue might because of permissions. I am not allowed to access */tmp* directory on HDFS using hadoop UI. But */tmp* directory is created itself, please let me know what can be the resolution. Thanks. On Wed, Jun 24, 2015 at 6:36 PM, Hitesh Shah hit...@apache.org wrote: The error seems to indicate that the create dir/write to HDFS failed. Can you check compare what user you are running as and whether the user has permissions to create/write to the directories in the path below? Furthermore, you may wish to check if the datanodes are alive and also look for errors earlier in the logs to see what failed ( due to the bit related to Previous writer likely failed” ) . thanks — Hitesh On Jun 24, 2015, at 5:19 AM, amit kumar aks...@gmail.com wrote: java.io.IOException: Previous writer likely failed to write hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0.
Re: Error while running Hive queries over tez
The error seems to indicate that the create dir/write to HDFS failed. Can you check compare what user you are running as and whether the user has permissions to create/write to the directories in the path below? Furthermore, you may wish to check if the datanodes are alive and also look for errors earlier in the logs to see what failed ( due to the bit related to Previous writer likely failed” ) . thanks — Hitesh On Jun 24, 2015, at 5:19 AM, amit kumar aks...@gmail.com wrote: java.io.IOException: Previous writer likely failed to write hdfs://localhost:9000/tmp/hive/amitkumar/_tez_session_dir/25e6f3e0-b913-46c8-9d87-83962f1dc073/tez-0.7.0.
