[
https://issues.apache.org/jira/browse/HIVE-18623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352229#comment-16352229
]
yangfang edited comment on HIVE-18623 at 2/5/18 10:47 AM:
----------------------------------------------------------
Hive should move the execution result from the source path to the distination
path at the end of the execution, In my test, the source path is
viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-10000
and the distination path is viewfs://nsX/nameservice/hive/test_par6/p=10000/ ,
Moving files from source path to distination path are acrossing mount points.
HDFS use rename to move files , but it does not support renames across mount
points, so it throws an exception.
The exception comes from the following code:
private static Path mvFile(HiveConf conf, FileSystem sourceFs, Path
sourcePath, FileSystem destFs, Path destDirPath,
boolean isSrcLocal, boolean isOverwrite, boolean isRenameAllowed) throws
IOException {
................................................................................
for (int counter = 1; destFs.exists(destFilePath); counter++) {
if (isOverwrite)
{ destFs.delete(destFilePath, false); break; }
destFilePath = new Path(destDirPath, name + (Utilities.COPY_KEYWORD + counter)
+ (!type.isEmpty() ? "." + type : ""));
}
if (isRenameAllowed)
{ destFs.rename(sourcePath, destFilePath); }
else if (isSrcLocal)
{ destFs.copyFromLocalFile(sourcePath, destFilePath); }
else
{ FileUtils.copy(sourceFs, sourcePath, destFs, destFilePath, true, // delete
source false, // overwrite destination conf); }
................................................................................
}
I think we should make isRenameAllowed false because across mount points can
not use rename, we should use copy In this case, so I have changed needToCopy()
method , if I found source path and destination path are acrossing mount point
, needToCopy() return false.Now isRenameAllowed is false , Hive will use
FileUtils.copy() method for moving files.
was (Author: yangfang):
Hive should move the execution result from the source path to the distination
path at the end of the execution, In my test, the source path is
viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-10000
and the distination path is viewfs://nsX/nameservice/hive/test_par6/p=10000/ ,
Moving files from source path to distination path are acrossing mount points.
HDFS use rename to move files , but it does not suport renames across mount
points, so it throws an exception.
The exception comes from the following code:
private static Path mvFile(HiveConf conf, FileSystem sourceFs, Path
sourcePath, FileSystem destFs, Path destDirPath,
boolean isSrcLocal, boolean isOverwrite, boolean isRenameAllowed) throws
IOException {
................................................................................
for (int counter = 1; destFs.exists(destFilePath); counter++) {
if (isOverwrite) {
destFs.delete(destFilePath, false);
break;
}
destFilePath = new Path(destDirPath, name + (Utilities.COPY_KEYWORD + counter)
+ (!type.isEmpty() ? "." + type : ""));
}
if (isRenameAllowed) {
destFs.rename(sourcePath, destFilePath);
}else if (isSrcLocal) {
destFs.copyFromLocalFile(sourcePath, destFilePath);
} else {
FileUtils.copy(sourceFs, sourcePath, destFs, destFilePath,
true, // delete source
false, // overwrite destination
conf);
}
................................................................................
}
I think we should make isRenameAllowed false because across mount points can
not use rename, we should use copy In this case, so I have changed needToCopy()
method , if I found source path and destination path are acrossing mount point
, needToCopy() return false.Now isRenameAllowed is false , Hive will use
FileUtils.copy() method for moving files.
> Hive throws an exception "Renames across Mount points not supported" when
> running in a federated cluster
> --------------------------------------------------------------------------------------------------------
>
> Key: HIVE-18623
> URL: https://issues.apache.org/jira/browse/HIVE-18623
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 2.2.0, 2.3.0, 2.3.1, 2.3.2
> Environment: hadoop 2.7.5, HDFS Federation enabled
> hive 3.0.0
> Reporter: yangfang
> Assignee: yangfang
> Priority: Major
> Attachments: HIVE-18623.1.patch
>
>
>
> I run a sql query in in a federated cluster and I have two namespaces:
> nameservice and nameservice1. I set
> hive.exec.stagingdir=/nameservice1/hive_tmp in hive-site.xml and my data
> tables are located in the directory of nameservice, then I got the exception
> as below:
> hive> create external table test_par6(id int,name string) partitioned by(p
> int);
> OK
> Time taken: 1.527 seconds
> hive> insert into table test_par6 partition(p = 10000) values(1,'Jack');
> Moving data to directory
> viewfs://nsX/nameservice1/hive_tmp_hive_2018-02-05_14-09-36_416_3075179128063595297-1/-ext-10000
> Loading data to table default.test_par6 partition (p=10000)
> Failed with exception java.io.IOException: Renames across Mount points not
> supported
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Renames across
> Mount points not supported
> MapReduce Jobs Launched:
> Stage-Stage-1: Map: 1 Cumulative CPU: 2.08 sec HDFS Read: 3930 HDFS Write: 7
> SUCCESS
> Total MapReduce CPU Time Spent: 2 seconds 80 msec
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
