Hi,
You should include files under the conf directory. Since they are not
included in classpath hadoop cannot find the implemetation class for
"file:" FileSystem. Check whether your HADOOP_CONF_DIR is set correct.
Another reason for this exception can be that your HADOOP_CONF_DIR is
pointing to some old hadoop version which donot contain new conf options
like FileSystem implementations.
[EMAIL PROTECTED] wrote:
Hi,
I've got a GUI based program that I'm working on, and I'm trying to
add some funcionality to it where it runs a map reduce job on hadoop.
For the moment I am assuming that anyone who is running the program
will be running it on a machine with a hadoop system running, although
later I would like for them to be able to point it at a different
machine on the network.
So, everything works fine when I run the GUI from eclipse (I also have
a hadoop project in there BTW). However, when I run the GUI from the
command line I get the following exception:
java.io.IOException: No FileSystem for scheme: file
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)
at
pipe.dataLayer.calculations.RTAMapRed.launch(RTAMapRed.java:293)
and so on......
The bit of code that causes it is (the offending line is marked ****):
conf = new Configuration();
jobConf = new JobConf(conf, RTAMapRed.class);
jobConf.setJobName("rta");
// turn off speculative execution, because DFS doesn't handle
// multiple writers to the same file.
jobConf.setSpeculativeExecution(false);
jobConf.setInputFormat(EulerSequenceFileInputFormat.class);
jobConf.setOutputKeyClass(ComplexWritable.class);
jobConf.setOutputValueClass(DoubleWritable.class);
jobConf.setOutputFormat(SequenceFileOutputFormat.class);
jobConf.setMapperClass(RTAMapper.class);
jobConf.setReducerClass(RTAReducer.class);
tmpDir = new Path("rtamapred");
inDir = new Path(tmpDir, "in");
outDir = new Path(tmpDir, "out");
FileSystem fileSys = FileSystem.get(jobConf); //****This line
here!!
fileSys.delete(tmpDir);
if (!fileSys.mkdirs(inDir)) {
throw new IOException("Mkdirs failed to create " +
inDir.toString());
}
I'm running the GUI with the following script to include my hadoop
installation (its a developer one) on the classpath. I was wondering
if I've missed anything obvious off?
CLASSPATH="${HADOOP_CONF_DIR}"
CLASSPATH=${CLASSPATH}:bin
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
CLASSPATH=${CLASSPATH}:/home/ollie/workspace/hadoop/build/classes
CLASSPATH=${CLASSPATH}:/home/ollie/workspace/hadoop/build
CLASSPATH=${CLASSPATH}:$HADOOP_INSTALL/build/test/classes
for f in /home/ollie/workspace/hadoop/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $HADOOP_INSTALL/lib/jetty-ext/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR"
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.file=$HADOOP_LOGFILE"
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.home.dir=$HADOOP_HOME"
HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING"
HADOOP_OPTS="$HADOOP_OPTS
-Dhadoop.root.logger=${HADOOP_ROOT_LOGGER:-INFO,console}"
if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
fi
exec java $HADOOP_OPTS -classpath "$CLASSPATH" RunGui "$@"
(I've tried it with and without the $HADOOP_OPTS set )
Thanks very much in advance for any help offered and apologies for the
information overload!
Cheers,
Ollie
