Jonathan Eagles created TEZ-3993:
------------------------------------
Summary: Tez fails to parse windows file paths in local mode
Key: TEZ-3993
URL: https://issues.apache.org/jira/browse/TEZ-3993
Project: Apache Tez
Issue Type: Bug
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
TezLocalCacheManager tries to generate symlinks to files that it puts in the
local cache, but the code that it uses to construct the path names is not safe
on Windows and causes bad file names to be constructed when run in a Windows
environment. On Windows, a path like file:/c:/path/to/my/file should be legal
and transform to c:\path\to\my\file, but with the invalid construct, it turns
into the illegal value /c:/path/to/my/file instead.
In TezLocalCacheManager, there is code that does
{code}
private boolean createSymlink(Path target, Path link) throws IOException {
LOG.info("Creating symlink: {} <- {}", target, link);
String targetPath = target.toUri().getPath();
String linkPath = link.toUri().getPath();
{code}
It looks like there are several other places in the Tez code that also use the
Path.toUri().getPath() construct that probably also need to be fixed in order
to work correctly on Windows.
The construct Path.toUri().getPath() doesn't handle windows directories
correctly. The Java File class understands how to do this correctly, so this
should really be replaced by
{code}
private boolean createSymlink(Path target, Path link) throws IOException {
LOG.info("Creating symlink: {} <- {}", target, link);
String targetPath = new File(target.toUri()).getCanonicalPath();
String linkPath = new File(link.toUri()).getCanonicalPath();
{code}
{code}
2018-09-19T16:32:53,287 ERROR [LocalContainerLauncher-SubTaskRunner]
org.apache.tez.dag.app.launcher.LocalContainerLauncher - TezSubTaskRunner
failed due to exception
java.nio.file.InvalidPathException: Illegal char <:> at index 2:
/C:/Users/...fullpath
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at java.nio.file.Paths.get(Paths.java:84)
at
org.apache.tez.dag.app.launcher.TezLocalCacheManager.createSymlink(TezLocalCacheManager.java:173)
at
org.apache.tez.dag.app.launcher.TezLocalCacheManager.localize(TezLocalCacheManager.java:126)
at
org.apache.tez.dag.app.launcher.LocalContainerLauncher.launch(LocalContainerLauncher.java:263)
at
org.apache.tez.dag.app.launcher.LocalContainerLauncher.access$300(LocalContainerLauncher.java:82)
at
org.apache.tez.dag.app.launcher.LocalContainerLauncher$TezSubTaskRunner.run(LocalContainerLauncher.java:207)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
