[
https://issues.apache.org/jira/browse/HADOOP-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264577#comment-13264577
]
FKorning commented on HADOOP-8274:
----------------------------------
If the symlinks really are necessary, here's a java.io.File wrapper
that understands cygwin symlinks (and windows shortcuts).
{code}
package net.muxbus.io;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;
/**
*
* @author Francis Korning [email protected]
* @date 2012-04-30
*
* @project muxbus muxbus.net
* @source sourceforge https://sourceforge.net/p/muxbus/
*
* @copyright 2012 F.Korning
* @license LGPL http://www.gnu.org/copyleft/lesser.html
*
* @credit Jesse Hager [email protected] (.lnk shortcut file
format).
*
*
* LinkedFile is a java.io.File wrapper that understands both Cygwin Symbolic
Links
* and Windows Explorer Shorcut Links (.lnk). If a given File is a Link, it
tries
* to resolve the File by following link paths recursively until a final
canonical
* target is found, and if the file exists will act as a wrapper for the link
* target. A LinkedFile always tracks its source and provides link reflection.
*
* A common frustration for POSIX platform developers is that Java on windows
does
* not understand symbolic Links which aren't handled by the windows io native
libs.
* This means that Cygwin or Interix (aka Winterix or Windows Service for Unix)
* integrated java applications that depend on POSIX tools may break when it
comes
* to resolution of paths with linked files and directories.
*
* The first workaround strategy, which works where links are predictably
created,
* is to ensure that cygwin paths and windows paths are equivalent and
identical.
* For example, you create a circular "/cygwin" link that maps back to the root
"/":
*
* <code>
* <pre>
* /cygwin -> / -> \cygwin\ -> C:\cygwin\
* /cygwin/tmp -> /tmp/ -> \cygwin\tmp\ -> C:\cygwin\tmp\
* </pre>
* </code>
*
* But you often have no control over much of the integrated codebase and tools.
* If some of the components create symlinks dynamically where java expects real
* files and directories this workaround will fail.
*
* Now the only OS-native IO alternative would be to use Windows NTFS Junctions.
* These are completely transparent and work in shells, in Explorer, and for
java.
* The best implementation would be the junction provided in the SysInternals
suite.
* You could wrap the cygwin 'ln' command and force it to use a junction
instead,
* but this breaks some POSIX compatibility and requires Administrator superuser
* rights which you may not want to assign to all your running service daemons.
*
* Next Interix symbolic links and the mklink command are built-in since
Windows Vista.
* These fall in between as they act like native symlinks in the shell and in
java,
* look like cygwin links inside the shell, but again they can only be created
by the
* Administrator, and also break POSIX. Unlike cygwin they do appear as
shortcuts in
* a Windows Explorer, with the caveat that these are broken because of
permissions.
*
* Note that both strategies only work if the tools create links via the 'ln'
command.
* But what if the offending integrated tools are in perl or python, or even
better yet in
* compiled c/c++ code making use of libc calls for which we have no source nor
license?
* This means that we will need to adopt a strategy to adapt java.io.File to
Cygwin libc.
* Also, most multi-platform java developers favour Cygwin instead of
Microsoft's Interix.
* All of the POSIX daemons we're looking at expect cygwin shells, and this
compatibility
* is our main focus here; we ideally an a model that works without having to
wrap 'ln'.
*
* Now Both Cygwin Symlinks and Explorer Shortcuts are just plain binary files
underneath
* with some information pointing to their link targets. They have the
advantage that
* we can parse these and thus this allows us to add some link reflection to
java.io.
* (This also creates a strange case where windows would have more
functionality than
* unix, unless one would write a unix java native libc wrapper for link
functions).
* They also require no special Administrator rights for creation and
maintenance.
*
* A side-effect is that shortcuts created under Windows Explorer have the
feature that
* a (.lnk) extension is appended, which is obscured from the explorer. As the
names
* differ, both a Cygwin Symlink and a Windows Shortcut can co-exist in the
same place
* and make for a consistent interface both in the cygwin shell and Windows
Explorer.
*
* This means that one could well mirror every cygwin symlink with a windows
shortcut,
* and thus have symbolic links that work even under non-Administrator user
shells and
* are mirrored by corresponding Windows Shortcuts that work in the Windows
Explorer.
* Cygwin also comes with its own programmatic 'mkshortcut' command to create
Shorcuts
* from a shell (which you can't under windows by default - doesn't Windows
suck?).
*
* <code>
* <pre>
* dir -> /dir/ symlink
* dir.lnk -> /dir/ shortcut
* file -> /dir/file symlink
* file.lnk -> /dir/file shortcut
* </pre>
* </code>
*
* Note that synchronizing of Cygwin symlinks and Windows Shortcuts is left
out-of-scope.
* Our primary concern is to get our POSIX daemons running, not to add sugar to
Explorer.
* However LinkedFile will support and resolve links for both link formats,
with first
* preferrence given to Cygwin Symlinks if found, and then to Windows Shortcuts
of the
* same name (without the .lnk extension), which you may extend for such a
facility.
*
* For those wishing to extend the functionality to support transparent
mirroring between
* Explorer and the shell, note that you will have to map links to their (.lnk)
shortcuts.
* Also note that Windows Shortcuts are never interpreted as links by the
Cygwin shell.
* Most critically note that Windows shortcuts are unpredictable in the shell
if moved,
* as they seem to only recalculate relative paths after a link is accessed in
Explorer.
*
*
* @see cygwin: http://www.cygwin.com
* @see cygwin symlinks: http://cygwin.com/ml/cygwin/2010-11/msg00394.html
* @see interix links:
http://technet.microsoft.com/en-us/library/bb463204.aspx
* @see windows shortcuts:
http://ithreats.files.wordpress.com/2009/05/lnk_the_windows_shortcut_file_format.pdf
* @see windows junctions: http://technet.microsoft.com/en-us/sysinternals
*
*/
public class LinkedFile
extends java.io.File
{
public static final String TOK_SYMLINK = "!<symlink>\u00FF\u00FE";
public static final String TOK_SHORTCUT =
"\u004C\u0000\u0000\u0000\u0001\u0014\u0002";
public static final String TOK_SHORTCUT_SEP = "\u0000";
public static final int SHORTCUT_MAX = 1024; // shortcuts should
probably never be that big
public static final int SHORTCUT_OFFSET = 76; // first shortcut
section length offset at 0x4C
// utilities
/**
* read a link header.
*
* both cygwin symlinks and windows shortcuts are plain binary files.
* cygwin is faily simple, consisting of magic header and the target.
* shortcuts have variable length structures but will not exceed 1kb.
*/
public static String header (File file)
throws IOException
{
if (file == null)
return null;
String header = null;
if (file.exists() && file.canRead())
{
if (file.isFile() || (!file.isDirectory()))
{
char[] buf = new char [SHORTCUT_MAX];
int length = new BufferedReader(new FileReader(file)).read(buf,
0, SHORTCUT_MAX);
header = new String(buf);
//System.err.println ("length: " + length);
//System.err.println ("header: " + header);
if (isSymlinkHeader(header))
{
//System.err.println("isSymlink: true");
}
else if (isShortcutHeader(header))
{
//System.err.println("isShortcut: true");
}
else
{
header = null;
}
}
}
return header;
}
public static boolean isLinkHeader (String header)
throws IOException
{
return ( isSymlinkHeader(header) || isShortcutHeader(header) );
}
public static boolean isSymlinkHeader (String header)
throws IOException
{
if (header == null)
return false;
if (header.startsWith(TOK_SYMLINK))
{
return true;
}
return false;
}
public static boolean isShortcutHeader (String header)
throws IOException
{
if (header == null)
return false;
if (header.startsWith(TOK_SHORTCUT))
{
return true;
}
return false;
}
/**
* parse a 2-char short length offset from a position in a windows shortcut
file.
*/
public static int shortcutHeaderOffset (String header, int pos)
{
if (header == null)
return 0;
char small = header.charAt(pos);
char large = header.charAt(pos +1);
int offset = small + (large * 16 * 16);
//System.err.println ("offset [" + pos + "] '" + (short)small + " " +
(short)large + "' : " + offset);
return offset;
}
/**
* parse the final link target from a cygwin symlink or windows shortcut.
*
* cygwin symlink targets are immediately after the header "<!symlink> 0xFF
xFE".
*
* windows shortcuts start with the header "L 0x00 0x00 0x00 0x01 0x14
0x02".
* the windows shortcut files are organized into sections of variable
lengths,
* starting after the header at 0x4c (76). the lengths are 2-char short
offsets.
* windows shortcut final name is at the end of the second "file location"
section.
*/
public static String headerTarget (String header)
{
if (header == null)
return null;
String target = null;
try
{
if (isSymlinkHeader(header))
{
StringBuffer buf = new StringBuffer("");
for (int i = TOK_SYMLINK.length(); i < header.length(); i+=2)
{
buf.append (header.charAt(i));
}
target = buf.toString();
}
else if (isShortcutHeader(header))
{
// first get the length of the next "item id" section
int a = shortcutHeaderOffset(header, SHORTCUT_OFFSET);
// now get the length of the next "file location" section
int b = shortcutHeaderOffset(header, SHORTCUT_OFFSET + a + 2);
// the final name is the last field of the "file location"
section
int j = SHORTCUT_OFFSET + a + b;
int i = header.lastIndexOf(TOK_SHORTCUT_SEP, j - 2) + 1;
target = header.substring (i,j);
}
}
catch (IOException e) {}
//System.err.println ("target: " + target);
return target;
}
public static boolean isLink (File file)
{
if (file == null)
return false;
try
{
if (file.exists() && file.canRead())
{
if (file.isFile() || (!file.isDirectory()))
{
String header = header(file);
return isLinkHeader(header);
}
}
}
catch (IOException e) {}
return false;
}
public static boolean isSymlink (File file)
{
if (file == null)
return false;
try
{
if (file.exists() && file.canRead())
{
if (file.isFile() || (!file.isDirectory()))
{
String header = header(file);
return isSymlinkHeader(header);
}
}
}
catch (IOException e) {}
return false;
}
public static boolean isShortcut (File file)
{
if (file == null)
return false;
try {
if (file.exists() && file.canRead())
{
if (file.isFile() || (!file.isDirectory()))
{
String header = header(file);
return isShortcutHeader(header);
}
}
}
catch (IOException e) {}
return false;
}
/**
* Resolve a target by recursively traversing links until the canonical
target.
* If the file is not a link, return the file's canonical AbsolutePath.
*/
public static String resolveTarget (File file)
{
String target = file.getAbsolutePath();
//System.err.println("\path: " + target);
try
{
if (file.exists() && file.canRead())
{
if (file.isFile() || (!file.isDirectory()))
{
String header = header(file);
if (isLinkHeader(header))
{
target = headerTarget(header);
boolean absolute =
(target.substring(1)).startsWith(":");
boolean logical = (target.startsWith("/") ||
target.startsWith("\\"));
boolean parent = (target.startsWith("../") ||
target.startsWith("..\\"));
boolean current = (target.startsWith("./") ||
target.startsWith(".\\"));
File link = null;
if (absolute)
link = new File (target.substring(0,2),
target.substring(2));
else if (logical)
link = new File(target.substring(0,1),
target.substring(1));
else if (parent)
link = new File(target.substring(0,2),
target.substring(2));
else if (current)
link = new File(target.substring(0,1),
target.substring(1));
else
link = new File (target);
if (! file.equals(link))
{
target = resolveTarget(link);
}
}
}
}
}
catch (Exception e)
{
e.printStackTrace();
}
//System.err.println("resolve: " + target);
return target;
}
/**
*
* Resolve a file by recursively traversing links until the canonical
target.
* If the file is not a link, return the file itself.
* @param file
* @return
*/
public static File resolveFile (File file)
{
//System.err.println("file: " + file.getAbsolutePath());
try
{
if (file.exists() && file.canRead())
{
if (file.isFile() || (!file.isDirectory()))
{
String header = header(file);
if (isLinkHeader(header))
{
String target = headerTarget(header);
boolean absolute =
(target.substring(1)).startsWith(":");
boolean logical = (target.startsWith("/") ||
target.startsWith("\\"));
boolean parent = (target.startsWith("../") ||
target.startsWith("..\\"));
boolean current = (target.startsWith("./") ||
target.startsWith(".\\"));
File link = null;
if (absolute)
link = new File (target.substring(0,2),
target.substring(2));
else if (logical)
link = new File(target.substring(0,1),
target.substring(1));
else if (parent)
link = new File(target.substring(0,2),
target.substring(2));
else if (current)
link = new File(target.substring(0,1),
target.substring(1));
else
link = new File (target);
if (! file.equals(link))
{
file = resolveFile(link.getAbsoluteFile());
}
}
}
}
}
catch (Exception e)
{
e.printStackTrace();
}
//System.err.println("resolve: " + file.getAbsolutePath());
return file;
}
// commmands
/**
* LinkedFile command resolves a link and prints its final link target name.
*
* @param args
* @throws IOException
*/
public static void main (String[] args)
throws IOException
{
try
{
if (args.length == 0)
{
System.err.println ("usage: LinkedFile <filepath>");
System.exit (1);
}
LinkedFile file = new LinkedFile(args[0]);
System.out.println(file);
}
catch (Exception e)
{
e.printStackTrace();
System.exit (1);
}
}
// fields
protected String source; // the file source
protected String target; // the resolved target path
protected String header; // the link header, if source is a link
protected boolean symlink; // file is a cygwin symlink
protected boolean shortcut; // file is a windows shortcut
protected File link; // the original source file link
// instantiators
// constructors
/**
* Creates a new File instance by converting the given pathname string into
* an abstract pathname.
*/
public LinkedFile(String pathname)
{
super((resolveFile(new File(pathname))).getAbsolutePath());
this.target = getAbsolutePath();
this.source = "" + pathname;
this.link = new File(pathname);
this.symlink = isSymlink(link);
this.shortcut = isShortcut(link);
}
/**
* Creates a new File instance from a parent abstract pathname and a child
* pathname string.
*/
public LinkedFile(File parent, String child)
{
super((resolveFile(new File(parent, child))).getAbsolutePath());
this.target = getAbsolutePath();
this.source = "" + parent.getName() + "/" + child;
this.link = new File(parent, child);
this.symlink = isSymlink(link);
this.shortcut = isShortcut(link);
}
/**
* Creates a new File instance from a parent pathname string and a child
* pathname string.
*/
public LinkedFile(String parent, String child)
{
super((resolveFile(new File(parent, child))).getAbsolutePath());
this.target = getAbsolutePath();
this.source = "" + parent + "/" + child;
this.link = new File(parent, child);
this.symlink = isSymlink(link);
this.shortcut = isShortcut(link);
}
/**
* Creates a new File instance by converting the given file: URI into an
* abstract pathname.
*/
public LinkedFile(URI uri)
{
super((resolveFile(new File(uri))).getAbsolutePath());
this.target = getAbsolutePath();
this.source = "" + uri;
this.link = new File(uri);
this.symlink = isSymlink(link);
this.shortcut = isShortcut(link);
}
// accessors
public boolean isLink()
{
return (symlink || shortcut);
}
public boolean isSymlink()
{
return (symlink);
}
public boolean isShortcut()
{
return (shortcut);
}
public String getSource()
{
return source;
}
public String getTarget()
{
return target;
}
// overrides
public String getName()
{
return super.getName().replace('\\', '/');
}
public String getPath()
{
return super.getPath().replace('\\', '/');
}
public String getAbsolutePath()
{
return super.getAbsolutePath().replace('\\', '/');
}
public String getCanonicalPath()
throws IOException
{
return super.getCanonicalPath().replace('\\', '/');
}
public String toString()
{
// System.err.println("");
// System.err.println("file: " + getAbsolutePath());
// System.err.println("path: " + getPath());
// System.err.println("name: " + getName());
//
// System.err.println("source: " + getSource());
// System.err.println("target: " + getTarget());
//
// System.err.println("exists: " + exists());
// System.err.println("canRead: " + canRead());
// System.err.println("canWrite: " + canWrite());
// System.err.println("canExecute: " + canExecute());
//
// System.err.println("isDirectory: " + isDirectory());
// System.err.println("isSymlink: " + isSymlink());
// System.err.println("isShortcut: " + isShortcut());
StringBuffer s = new StringBuffer("");
if (isLink())
s.append(getSource() + " -> ");
s.append(getTarget());
if (isDirectory())
s.append("/");
return s.toString();
}
}
{code}
> In pseudo or cluster model under Cygwin, tasktracker can not create a new job
> because of symlink problem.
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8274
> URL: https://issues.apache.org/jira/browse/HADOOP-8274
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.205.0, 1.0.0, 1.0.1, 0.22.0
> Environment: windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0
> Reporter: tim.wu
>
> The standalone model is ok. But, in pseudo or cluster model, it always throw
> errors, even I just run wordcount example.
> The HDFS works fine, but tasktracker can not create threads(jvm) for new job.
> It is empty under /logs/userlogs/job-xxxx/attempt-xxxx/.
> The reason looks like that in windows, Java can not recognize a symlink of
> folder as a folder.
> The detail description is as following,
> ======================================================================================================
> First, the error log of tasktracker is like:
> ======================
> 12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_201203280212_0005_m_-1386636958
> 12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner
> jvm_201203280212_0005_m_-1386636958 spawned.
> 12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed
> jvm_201203280212_0005_m_-1386636958 but just removed
> 12/03/28 14:35:17 INFO mapred.JvmManager: JVM :
> jvm_201203280212_0005_m_-1386636958 exited with exit code -1. Number of tasks
> it ran: 0
> 12/03/28 14:35:17 WARN mapred.TaskRunner:
> attempt_201203280212_0005_m_000002_0 : Child Error
> java.io.IOException: Task process exit with nonzero status of -1.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> 12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots :
> 2
> 12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask):
> attempt_201203280212_0005_m_000002_1 task's state:UNASSIGNED
> 12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch :
> attempt_201203280212_0005_m_000002_1 which needs 1 slots
> 12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free
> slots : 2 and trying to launch attempt_201203280212_0005_m_000002_1 which
> needs 1 slots
> 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for
> task: attempt_201203280212_0005_m_000002_0
> java.io.FileNotFoundException:
> D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index
> (The system cannot find the path specified)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at
> org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
> at
> org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
> at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
> at
> org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
> at
> org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stderr log for
> task: attempt_201203280212_0005_m_000002_0
> java.io.FileNotFoundException:
> D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index
> (The system cannot find the path specified)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at
> org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
> at
> org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
> at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
> at
> org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
> at
> org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> =======================================
> I've tried to remote debug tasktracker. In
> org.apache.hadoop.mapredTaskLog.createTaskAttemptLogDir(TaskAttemptID,
> boolean, String[]) line: 97:
> public static void createTaskAttemptLogDir(TaskAttemptID taskID,
> boolean isCleanup, String[] localDirs) throws IOException{
> String cleanupSuffix = isCleanup ? ".cleanup" : "";
> String strAttemptLogDir = getTaskAttemptLogDir(taskID,
> cleanupSuffix, localDirs);
> File attemptLogDir = new File(strAttemptLogDir);
> if (!attemptLogDir.mkdirs()) {
> throw new IOException("Creation of " + attemptLogDir + " failed.");
> }
> String strLinkAttemptLogDir =
> getJobDir(taskID.getJobID()).getAbsolutePath() + File.separatorChar +
> taskID.toString() + cleanupSuffix;
> if (FileUtil.symLink(strAttemptLogDir, strLinkAttemptLogDir) != 0) {
> throw new IOException("Creation of symlink from " +
> strLinkAttemptLogDir + " to " +
> yestrAttemptLogDir +
> " failed.");
> }
> //Set permissions for target attempt log dir
> FsPermission userOnly = new FsPermission((short) 0777); //FsPermission
> userOnly = new FsPermission((short) 0700);
> FileUtil.setPermission(attemptLogDir, userOnly);
> }
> and symlink() function
> public static int symLink(String target, String linkname) throws IOException{
> String cmd = "ln -s " + target + " " + linkname;
> Process p = Runtime.getRuntime().exec(cmd, null);
> int returnVal = -1;
> try{
> returnVal = p.waitFor();
> } catch(InterruptedException e){
> //do nothing as of yet
> }
> if (returnVal != 0) {
> LOG.warn("Command '" + cmd + "' failed " + returnVal +
> " with: " + copyStderr(p));
> }
> return returnVal;
> }
> we know hadoop will create a log folder in ${hadoop.tmp.dir}, and then
> invoke "ln -s " to create its symlink under
> /logs/userlog/job-xxx/attermp-xxxx.
> In my case,
> strLinkAttemptLogDir =
> D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
> strAttemptLogDir=/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
> After a subtrack is created by tasktracker, it runs error in the following
> function:
> in org.apache.hadoop.mapred.java , DefaultTaskController.launchTask(String,
> String, String, List<String>, List<String>, File, String, String) line: 107
>
> ...............
> //mkdir the loglocation
> String logLocation = TaskLog.getAttemptDir(jobId, attemptId).toString();
> if (!localFs.mkdirs(new Path(logLocation))) {
> throw new IOException("Mkdirs failed to create "
> + logLocation);
> }
> ..............
> mkdir() return false, because logLocation is a symlink file. In my case, it
> is
> ogLocation=D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1.
> If I open it from explorer in windows, it is just a file, but not a folder
> or shortcut. And its content is like,
>
> <symlink>/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
>
> Because the mkdir() is
> public boolean mkdirs(Path f) throws IOException {
> Path parent = f.getParent();
> File p2f = pathToFile(f);
> return (parent == null || mkdirs(parent)) &&
> (p2f.mkdir() || p2f.isDirectory());
> }
> So, p2f.isDirectory returns false. And p2f.isFile will return true. So, for
> java, it is a file. Hence, IOException("Mkdirs failed to create
> D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000001_1")
> will be throws in child threads, and return -1. Then, we will get the above
> exception in main thread.
> Is it any way to close this symlink? Or any other way I can follow?
> BTW, in core-site.xml, I set hadoop.tmp.dir = /tmp/hadoop-${user.name}, and
> my $User.name is timwu. So, it should create a tmp folder /tmp/hadoop-timwu
> under cygwin's. However, in deed , it create a folder of
> d:/tmp/hadoop-timwu. If in cygwin, it is /cygdriver/d/tmp/hadoop-timwu. Is
> it correct?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
