[
https://issues.apache.org/jira/browse/HIVE-21483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated HIVE-21483:
------------------------------
Summary: Fix HoS when scratch dir is using remote HDFS (was: HoS would
fail when scratch dir is using remote HDFS)
> Fix HoS when scratch dir is using remote HDFS
> ---------------------------------------------
>
> Key: HIVE-21483
> URL: https://issues.apache.org/jira/browse/HIVE-21483
> Project: Hive
> Issue Type: Bug
> Reporter: Dapeng Sun
> Assignee: Dapeng Sun
> Priority: Major
>
> HoS would fail when scratch dir is using remote HDFS:
> {noformat}
> public static URI uploadToHDFS(URI source, HiveConf conf) throws
> IOException {
> Path localFile = new Path(source.getPath());
> Path remoteFile = new
> Path(SessionState.get().getSparkSession().getHDFSSessionDir(),
> getFileName(source));
> - FileSystem fileSystem = FileSystem.get(conf);
> + FileSystem fileSystem = remoteFile.getFileSystem(conf);
> // Overwrite if the remote file already exists. Whether the file can be
> added
> // on executor is up to spark, i.e. spark.files.overwrite
> fileSystem.copyFromLocalFile(false, true, localFile, remoteFile);
> Path fullPath = fileSystem.getFileStatus(remoteFile).getPath();
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
