[
https://issues.apache.org/jira/browse/HIVE-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138325#comment-14138325
]
Gopal V commented on HIVE-8071:
-------------------------------
??
That is my change-set up there.
{code}
private boolean checkPreExisting(Path src, Path dest, Configuration conf)
throws IOException {
FileSystem destFS = dest.getFileSystem(conf);
FileSystem sourceFS = src.getFileSystem(conf);
if (destFS.exists(dest)) {
return (sourceFS.getFileStatus(src).getLen() ==
destFS.getFileStatus(dest).getLen());
}
return false;
}
...
if (src != null && checkPreExisting(src, dest, conf) == false) {
...
{code}
Doesn't that already check for size of file?
> hive shell tries to write hive-exec.jar for each run
> ----------------------------------------------------
>
> Key: HIVE-8071
> URL: https://issues.apache.org/jira/browse/HIVE-8071
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: HIVE-8071.1.patch, HIVE-8071.2.patch
>
>
> For every run of the hive CLI there is a delay for the shell startup
> {noformat}
> 14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> 14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is
> hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/
> 14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not
> exist:
> file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar to
> dest:
> hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8.
> jar
> 14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the
> same file will wait.
> 14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait
> interval: 5000
> 14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963
> 14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id:
> 02d6b558-44cc-4182-b2f2-6a37ffdd25d2, scratch dir:
> hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2)
> {noformat}
> Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE
> even if it is from a common install dir.
> {code}
> public LocalResource localizeResource(Path src, Path dest, Configuration
> conf)
> throws IOException {
> ....
> return createLocalResource(destFS, dest, LocalResourceType.FILE,
> LocalResourceVisibility.PRIVATE);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
