Hadoop Pipes do not load custom InputFormats at appropriate time, rendering
them useless in certain scenarios
-------------------------------------------------------------------------------------------------------------
Key: HADOOP-4020
URL: https://issues.apache.org/jira/browse/HADOOP-4020
Project: Hadoop Core
Issue Type: Improvement
Components: pipes
Affects Versions: 0.17.2, 0.17.1, 0.17.0, 0.16.4
Reporter: Mateusz Berezecki
There's a patch provided below.
It was created on the 0.17.0 version,
but with a little tweaking can be applied
to the 0.18.0 as well. It fixes a timing bug,
whereas a custom inputformat was loaded
from the jar file, but the inputformat was being
processed much earlier in time, resulting
in a class not found exception.
--- src/java/org/apache/hadoop/mapred/pipes/Submitter.java.orig 2008-05-15
09:20:16.000000000 +0200
+++ src/java/org/apache/hadoop/mapred/pipes/Submitter.java 2008-07-23
13:01:58.000000000 +0200
@@ -364,6 +364,14 @@
}
if (results.hasOption("-jar")) {
conf.setJar((String) results.getValue("-jar"));
+ // if they gave us a jar file, include it into the class path
+ String jarFile = conf.getJar();
+ if (jarFile != null) {
+ ClassLoader loader =
+ new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
+ pathToFile(new
Path(jarFile)).toURL()});
+ conf.setClassLoader(loader);
+ }
}
if (results.hasOption("-inputformat")) {
setIsJavaRecordReader(conf, true);
@@ -406,14 +414,6 @@
conf.set(keyValSplit[0], keyValSplit[1]);
}
}
- // if they gave us a jar file, include it into the class path
- String jarFile = conf.getJar();
- if (jarFile != null) {
- ClassLoader loader =
- new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
- pathToFile(new
Path(jarFile)).toURL()});
- conf.setClassLoader(loader);
- }
submitJob(conf);
} catch (OptionException oe) {
cli.printUsage();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
