[
https://issues.apache.org/jira/browse/HIVE-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138320#comment-14138320
]
Vikram Dixit K commented on HIVE-8071:
--------------------------------------
[~rajesh.balamohan] If you put the check in that if clause, there may be a race
condition to localize the resource before it has been completely written to
hdfs. This can happen with hive server 2. If you see the rest of the method, we
wait till the size of the file becomes equal to the size expected and then
proceed in case the file was already present.
> hive shell tries to write hive-exec.jar for each run
> ----------------------------------------------------
>
> Key: HIVE-8071
> URL: https://issues.apache.org/jira/browse/HIVE-8071
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: HIVE-8071.1.patch, HIVE-8071.2.patch
>
>
> For every run of the hive CLI there is a delay for the shell startup
> {noformat}
> 14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> 14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is
> hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/
> 14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not
> exist:
> file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar to
> dest:
> hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8.
> jar
> 14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the
> same file will wait.
> 14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait
> interval: 5000
> 14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963
> 14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id:
> 02d6b558-44cc-4182-b2f2-6a37ffdd25d2, scratch dir:
> hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2)
> {noformat}
> Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE
> even if it is from a common install dir.
> {code}
> public LocalResource localizeResource(Path src, Path dest, Configuration
> conf)
> throws IOException {
> ....
> return createLocalResource(destFS, dest, LocalResourceType.FILE,
> LocalResourceVisibility.PRIVATE);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
