Thanks - this suggestion is good.

I poked around in Configuration trying to see where things get loaded from and found the "quietmode" flag, which keeps info about where the config is being sourced from being printed. The log statements here are info, but they're wrapped with a global hardcoded constant. Maybe the log statements should be changed to debug and the quietmode constant removed? It'd make it a lot easier to get access to this output.

I've found that my problem seems to be that hadoop-default.xml goes unfound when I'm running my JtestR test suite. For giggles, I created a quick JUnit test just to see if it could access hadoop, and it could. I assume this means something about using JRuby messes with the classloader and interferes with picking up the config files from the classpath.

-Bryan

On May 15, 2008, at 2:21 AM, Steve Loughran wrote:

Bryan Duxbury wrote:
Nobody has any ideas about this?
-Bryan
On May 13, 2008, at 11:27 AM, Bryan Duxbury wrote:
I'm trying to create a java application that writes to HDFS. I have it set up such that hadoop-0.16.3 is on my machine, and the env variables HADOOP_HOME and HADOOP_CONF_DIR point to the correct respective directories. My app lives elsewhere, but generates it's classpath by looking in those environment variables. Here's what my generated classpath looks like:

/Users/bryanduxbury/hadoop-0.16.3/conf:/Users/bryanduxbury/ hadoop-0.16.3/hadoop-0.16.3-core.jar:/Users/bryanduxbury/ hadoop-0.16.3/hadoop-0.16.3-test.jar:/Users/bryanduxbury/ hadoop-0.16.3/lib/commons-cli-2.0-SNAPSHOT.jar:/Users/ bryanduxbury/hadoop-0.16.3/lib/commons-codec-1.3.jar:/Users/ bryanduxbury/hadoop-0.16.3/lib/commons-httpclient-3.0.1.jar:/ Users/bryanduxbury/hadoop-0.16.3/lib/commons-logging-1.0.4.jar:/ Users/bryanduxbury/hadoop-0.16.3/lib/commons-logging- api-1.0.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/ jets3t-0.5.0.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/ jetty-5.1.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/ commons-el.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/ jasper-compiler.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty- ext/jasper-runtime.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/ jetty-ext/jsp-api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/ junit-3.8.1.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/ kfs-0.1.jar:/Users/bryanduxbury/hadoop-0.16.3/li
b/log4j-1.2.13.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/servlet- api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/xmlenc-0.52.jar:/ Users/bryanduxbury/projects/hdfs_collector/lib/jtestr-0.2.jar:/ Users/bryanduxbury/projects/hdfs_collector/lib/jvyaml.jar:/Users/ bryanduxbury/projects/hdfs_collector/lib/libthrift.jar:/Users/ bryanduxbury/projects/hdfs_collector/build/hdfs_collector.jar


The problem I have is that when I go to get a FileSystem object for my file:/// files (for testing locally), I'm getting errors like this:

   [jtestr] java.io.IOException: No FileSystem for scheme: file
[jtestr] org/apache/hadoop/fs/FileSystem.java:1179:in `createFileSystem' [jtestr] org/apache/hadoop/fs/FileSystem.java:55:in `access$300'
   [jtestr]       org/apache/hadoop/fs/FileSystem.java:1193:in `get'
   [jtestr]       org/apache/hadoop/fs/FileSystem.java:150:in `get'
[jtestr] org/apache/hadoop/fs/FileSystem.java:124:in `getNamed'
   [jtestr]       org/apache/hadoop/fs/FileSystem.java:96:in `get'


Filesystems get located by looking up your conf file for the mapping from url schema to classname. if a filesystem isnt found, it means your hadoop-default.xml isnt being picked up (its in the root of hadoop-core), or a different version is being picked up, which contains invalid data.

this could be g good oppportunity for you to write some extra diagnostics for hadoop, something to dump all the job conf information on the client, and even validate a conf to look for important values.

steve

Reply via email to