[
https://issues.apache.org/jira/browse/TEZ-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-1097:
-----------------------------
Description:
In HDInsight Azure, we have hive scratch directory set to using HDFS. However
the default filesystem is WASB.
Due to this configuration, Tez seems to fail.
The relevant error logs are as follows.
{noformat}
2014-05-01 04:09:15,007 INFO tez.TezSessionState
(TezSessionState.java:open(142)) - User of session id
41905b13-36e7-4755-af34-265768778443 is chuan
2014-05-01 04:09:15,054 INFO tez.DagUtils
(DagUtils.java:getHiveJarDirectory(760)) - Jar dir is null/directory doesn't
exist. Choosing HIVE_INSTALL_DIR -
wasb://[email protected]/user/chuan/.hiveJars
2014-05-01 04:09:15,382 INFO tez.DagUtils
(DagUtils.java:localizeResource(839)) - Localizing resource because it does not
exist:
file:/C:/apps/dist/hive-0.13.0.2.1.2.0-1660/lib/hive-exec-0.13.0.2.1.2.0-1660.jar
to dest:
wasb://[email protected]/user/chuan/.hiveJars/hive-exec-0.13.0.2.1.2.0-1660-3abc67a6088fef5988020d9608d28d3025115d4201be785434c57c8a90d4cb5f.jar
2014-05-01 04:09:15,413 INFO tez.DagUtils
(DagUtils.java:localizeResource(843)) - Looks like another thread is writing
the same file will wait.
2014-05-01 04:09:15,413 INFO tez.DagUtils
(DagUtils.java:localizeResource(850)) - Number of wait attempts: 5. Wait
interval: 5000
2014-05-01 04:09:15,444 INFO tez.DagUtils
(DagUtils.java:createLocalResource(603)) - Resource modification time:
1398916679000
2014-05-01 04:09:15,444 ERROR exec.Task (TezTask.java:execute(192)) - Failed to
execute tez graph.
java.lang.IllegalArgumentException: Wrong FS:
hdfs://headnodehost:9000/hive/scratch/_tez_session_dir/41905b13-36e7-4755-af34-265768778443,
expected: wasb://[email protected]
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
at
org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.checkPath(NativeAzureFileSystem.java:436)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
at org.apache.tez.client.AMConfiguration.<init>(AMConfiguration.java:81)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:174)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:154)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-05-01 04:09:15,444 ERROR ql.Driver (SessionState.java:printError(546)) -
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
{noformat}
was:
In HDInsight Azure, we have hive scratch directory set to using HDFS. However
the default filesystem is WASB.
Due to this configuration, Tez seems to fail. Not sure where exactly the code
fix needs to go, so creating a hive bug (plus there is no project for Tez yet
in Monarch)
The relevant error logs are as follows.
{noformat}
2014-05-01 04:09:15,007 INFO tez.TezSessionState
(TezSessionState.java:open(142)) - User of session id
41905b13-36e7-4755-af34-265768778443 is chuan
2014-05-01 04:09:15,054 INFO tez.DagUtils
(DagUtils.java:getHiveJarDirectory(760)) - Jar dir is null/directory doesn't
exist. Choosing HIVE_INSTALL_DIR -
wasb://[email protected]/user/chuan/.hiveJars
2014-05-01 04:09:15,382 INFO tez.DagUtils
(DagUtils.java:localizeResource(839)) - Localizing resource because it does not
exist:
file:/C:/apps/dist/hive-0.13.0.2.1.2.0-1660/lib/hive-exec-0.13.0.2.1.2.0-1660.jar
to dest:
wasb://[email protected]/user/chuan/.hiveJars/hive-exec-0.13.0.2.1.2.0-1660-3abc67a6088fef5988020d9608d28d3025115d4201be785434c57c8a90d4cb5f.jar
2014-05-01 04:09:15,413 INFO tez.DagUtils
(DagUtils.java:localizeResource(843)) - Looks like another thread is writing
the same file will wait.
2014-05-01 04:09:15,413 INFO tez.DagUtils
(DagUtils.java:localizeResource(850)) - Number of wait attempts: 5. Wait
interval: 5000
2014-05-01 04:09:15,444 INFO tez.DagUtils
(DagUtils.java:createLocalResource(603)) - Resource modification time:
1398916679000
2014-05-01 04:09:15,444 ERROR exec.Task (TezTask.java:execute(192)) - Failed to
execute tez graph.
java.lang.IllegalArgumentException: Wrong FS:
hdfs://headnodehost:9000/hive/scratch/_tez_session_dir/41905b13-36e7-4755-af34-265768778443,
expected: wasb://[email protected]
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
at
org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.checkPath(NativeAzureFileSystem.java:436)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
at org.apache.tez.client.AMConfiguration.<init>(AMConfiguration.java:81)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:174)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:154)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-05-01 04:09:15,444 ERROR ql.Driver (SessionState.java:printError(546)) -
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask
{noformat}
> Tez assumes that the scratch directory has to be same as the default
> filesystem
> -------------------------------------------------------------------------------
>
> Key: TEZ-1097
> URL: https://issues.apache.org/jira/browse/TEZ-1097
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Chuan Liu
>
> In HDInsight Azure, we have hive scratch directory set to using HDFS. However
> the default filesystem is WASB.
> Due to this configuration, Tez seems to fail.
> The relevant error logs are as follows.
> {noformat}
> 2014-05-01 04:09:15,007 INFO tez.TezSessionState
> (TezSessionState.java:open(142)) - User of session id
> 41905b13-36e7-4755-af34-265768778443 is chuan
> 2014-05-01 04:09:15,054 INFO tez.DagUtils
> (DagUtils.java:getHiveJarDirectory(760)) - Jar dir is null/directory doesn't
> exist. Choosing HIVE_INSTALL_DIR -
> wasb://[email protected]/user/chuan/.hiveJars
> 2014-05-01 04:09:15,382 INFO tez.DagUtils
> (DagUtils.java:localizeResource(839)) - Localizing resource because it does
> not exist:
> file:/C:/apps/dist/hive-0.13.0.2.1.2.0-1660/lib/hive-exec-0.13.0.2.1.2.0-1660.jar
> to dest:
> wasb://[email protected]/user/chuan/.hiveJars/hive-exec-0.13.0.2.1.2.0-1660-3abc67a6088fef5988020d9608d28d3025115d4201be785434c57c8a90d4cb5f.jar
> 2014-05-01 04:09:15,413 INFO tez.DagUtils
> (DagUtils.java:localizeResource(843)) - Looks like another thread is writing
> the same file will wait.
> 2014-05-01 04:09:15,413 INFO tez.DagUtils
> (DagUtils.java:localizeResource(850)) - Number of wait attempts: 5. Wait
> interval: 5000
> 2014-05-01 04:09:15,444 INFO tez.DagUtils
> (DagUtils.java:createLocalResource(603)) - Resource modification time:
> 1398916679000
> 2014-05-01 04:09:15,444 ERROR exec.Task (TezTask.java:execute(192)) - Failed
> to execute tez graph.
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://headnodehost:9000/hive/scratch/_tez_session_dir/41905b13-36e7-4755-af34-265768778443,
> expected: wasb://[email protected]
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
> at
> org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.checkPath(NativeAzureFileSystem.java:436)
> at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
> at org.apache.tez.client.AMConfiguration.<init>(AMConfiguration.java:81)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:174)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:154)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> 2014-05-01 04:09:15,444 ERROR ql.Driver (SessionState.java:printError(546)) -
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
