Dapeng Sun created HIVE-21483:
---------------------------------
Summary: HoS would fail when scratch dir is using remote HDFS
Key: HIVE-21483
URL: https://issues.apache.org/jira/browse/HIVE-21483
Project: Hive
Issue Type: Bug
Reporter: Dapeng Sun
Assignee: Dapeng Sun
HoS would fail when scratch dir is using remote HDFS:
public static URI uploadToHDFS(URI source, HiveConf conf) throws IOException {
Path localFile = new Path(source.getPath());
Path remoteFile = new
Path(SessionState.get().getSparkSession().getHDFSSessionDir(),
getFileName(source));
- FileSystem fileSystem = FileSystem.get(conf);
+ FileSystem fileSystem = remoteFile.getFileSystem(conf);
// Overwrite if the remote file already exists. Whether the file can be
added
// on executor is up to spark, i.e. spark.files.overwrite
fileSystem.copyFromLocalFile(false, true, localFile, remoteFile);
Path fullPath = fileSystem.getFileStatus(remoteFile).getPath();
r
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
