somehow you have alternate versions of the file earlier in the class path.
Perhaps someone's empty copies are bundled into one of your application
jar files.
Or perhaps the configurationfiles are not distributed to the datanodes
in the expected locations.
Saptarshi Guha wrote:
For some strange reason, neither hadoop-default.xml nor
hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
folder.
HADOOP_CONF_DIR points to thi
In some java code, (not a job), i did
Configuration cf = new Configuration();
String[] xu= new
String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
for(String x : xu){
LOG.info(x+": "+cf.get(x));
}
All the values returned were default values(as noted in
hadoop-default.xml). Changing the value in hadoop-default.xml were not
reflected here. Grepping the source reveal the defaults are
hardcoded. So it seems neither is hadoop-default.xml not
hadooop-site.xml loading.
Also, Configuration.java mentionds that hadoop-site.xml is depecated
(thought still loaded)
Any suggestions?
Regards
Saptarshi
On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <[email protected]> wrote:
Hello,
I have set my HADOOP_CONF_DIR to the conf folder and still not
loading. I have to manually set the options when I create my conf.
Have you resolved this?
Regards
Saptarshi
On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <[email protected]> wrote:
Hey all,
I have a similar issue. I am specifically having problems with the config
option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
working in the previous versions of Hadoop I was using.
I am not explicitly setting HADOOP_CONF_DIR. My site config is in
${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
to print out the ENV values and it has the correct value for HADOOP_HOME,
HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
the JobConf passed to the mapper and it has my specified values for
fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
dfs.data.dir, and mapred.child.java.opts do not have my values.
Any suggestion where to look at next?
Thanks!
On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
[email protected]> wrote:
Saptarshi Guha wrote:
Hello,
I had previously emailed regarding heap size issue and have discovered
that the hadoop-site.xml is not loading completely, i.e
Configuration defaults = new Configuration();
JobConf jobConf = new JobConf(defaults, XYZ.class);
System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
System.out.println("2:"+jobConf.get("mapred.map.tasks"));
System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
hadoop-site.xml are very different.
Is there a way for hadoop to dump the parameters read in from
hadoop-site.xml and hadoop-default.xml?
Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
-Amareshwari
--
Saptarshi Guha - [email protected]
