[
https://issues.apache.org/jira/browse/TEZ-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989572#comment-16989572
]
Nikita Jaisawal commented on TEZ-4104:
--------------------------------------
To rectify this issue I tried refering the tar file directly from S3 without
changing fs.defaultFS to s3://bucketName.
I did the following changes in tez-site.xml:
* set tez.lib.uris property to s3://bucketName
In hive-site.xml:
* set hive.execution.engine=tez
* fs.defaultFS=hdfs://
But still it is searching for tar file in hdfs.
S3 location where tez.tar.gz has been stored:
s3://srev-emr-dev/F129/tez/tez.tar.gz where s3 bucket name is 'srev-emr-dev'
Facing below error:
java.io.FileNotFoundException: File does not exist: /F129/tez/tez.tar.gz
java.io.FileNotFoundException: File does not exist: /F129/tez/tez.tar.gz at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441)
~[hadoop-hdfs-client-2.8.3-amzn-1.jar:?] at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434)
~[hadoop-hdfs-client-2.8.3-amzn-1.jar:?] at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-2.8.3-amzn-1.jar:?] at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1449)
~[hadoop-hdfs-client-2.8.3-amzn-1.jar:?] at
org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:1031)
~[tez-api-0.8.4.jar:0.8.4] at
org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:275)
~[tez-api-0.8.4.jar:0.8.4] at
org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)
~[tez-api-0.8.4.jar:0.8.4] at
org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)
~[tez-api-0.8.4.jar:0.8.4] at
org.apache.tez.client.TezClient.start(TezClient.java:447)
~[tez-api-0.8.4.jar:0.8.4] at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:376)
~[hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:87)
~[hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:330)
[hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:326)
[hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at
java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at
java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
> Upscale/Downscale of the emr cluster corrupts hdfs jars /apps/tez/tez.tar.gz
> ----------------------------------------------------------------------------
>
> Key: TEZ-4104
> URL: https://issues.apache.org/jira/browse/TEZ-4104
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Nikita Jaisawal
> Priority: Major
>
> I am facing an issue in which the JARs kept in *hdfs:///apps/tez/tez/tar.gz*
> gets corrupted whenever I do emr cluster resizing after setting the property
> in hive-site.xml:
> *set hive.execution.engine=tez*
> (with default file system as *hdfs://)*
> How do I resolve this issue? Can anyone suggest?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
