I would suggest changing the hadoop configuration on the slave(s) You'll have to maintain the delta config amongst the nodes.
-----Original Message----- From: Björn-Elmar Macek [mailto:ma...@cs.uni-kassel.de] Sent: Friday, April 27, 2012 7:28 AM To: hdfs-user@hadoop.apache.org Subject: Re: Hadoop Configuration Issues Hi Alex, thank you for the tip: it pushed me into the right direction. I was using the deb-package to install hadoop, which did not work out cause of the problems i had. Now i use the tarball archive and unpacked it to a subfolder of my home directory. BUT i just ran into another problem: after successfully executing "hadoop namenode -format" the "start-all.sh" script (executed on the master server) runs into errors, when trying to access files on the slave system: it automatically seems to expect the hadoop files lying in the very same directory as on the master: ema@ubuntu:~/Programs/hadoop-1.0.2/bin$ ./start-all.sh Warning: $HADOOP_HOME is deprecated. namenode running as process 19393. Stop it first. slave: bash: line 0: cd: /home/ema/Programs/hadoop-1.0.2/libexec/..: No such file or directory slave: bash: /home/ema/Programs/hadoop-1.0.2/bin/hadoop-daemon.sh: No such file or directory ema@master's password: master: Connection closed by UNKNOWN starting jobtracker, logging to /var/log/hadoop/ema/hadoop-ema-jobtracker-ubuntu.out slave: bash: line 0: cd: /home/ema/Programs/hadoop-1.0.2/libexec/..: No such file or directory slave: bash: /home/ema/Programs/hadoop-1.0.2/bin/hadoop-daemon.sh: No such file or directory How can i tell the slave, that the files lie somewhere else? Best regards, Björn-Elmar Am 27.04.2012 13:11, schrieb alo alt: > Hi, > > yes, sorry - I saw that after I hit the send botton. > Looks like you mixed up some configs with wrong templates. I would suggest > you use the default configs: > http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project > /hadoop-common/src/main/conf/ > > - Alex > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > On Apr 27, 2012, at 12:39 PM, Björn-Elmar Macek wrote: > >> Hi Alex, >> >> as i have written, i already did so! The problem is as already stated in my >> mail before, that all the Variables ${bla} seem to be UNSET - not only >> SECURITY_TYPE. As i dont really understand those parameters, i would like to >> use the default ones, which afaik should be configured in the hadoop-env.sh. >> But obviously they are not. >> >> Best, >> Björn >> >> Am 27.04.2012 12:12, schrieb alo alt: >>> Hi, >>> >>> Invalid attribute value for hadoop.security.authentication of >>> ${SECURITY_TYPE} Set it to simple and it should work (default is kerberos). >>> >>> - Alex >>> >>> -- >>> Alexander Lorenz >>> http://mapredit.blogspot.com >>> >>> On Apr 27, 2012, at 12:01 PM, Björn-Elmar Macek wrote: >>> >>>> Hello, >>>> >>>> i have recently installed Hadoop on my and a second machine in order to >>>> test the setup and develop little programs locally before deploying them >>>> to the cluster. I stumbled over several difficulties, which i could fix >>>> with some internet research. But once again im stuck and i think i can >>>> nail the problem down: >>>> >>>> When Hadoop evaluates the config files in /etc/hadoop it does not have any >>>> default values for all the variables used within: >>>> >>>> \________ First Error: >>>> hadoop namenode -format >>>> Warning: $HADOOP_HOME is deprecated. >>>> >>>> 12/04/27 11:31:41 INFO namenode.NameNode: STARTUP_MSG: >>>> /************************************************************ >>>> STARTUP_MSG: Starting NameNode >>>> STARTUP_MSG: host = ubuntu/127.0.1.1 >>>> STARTUP_MSG: args = [-format] >>>> STARTUP_MSG: version = 1.0.1 >>>> STARTUP_MSG: build = >>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r >>>> 1243785; compiled by 'hortonfo' on Tue Feb 14 08:13:52 UTC 2012 >>>> ************************************************************/ >>>> 12/04/27 11:31:41 INFO util.GSet: VM type = 32-bit >>>> 12/04/27 11:31:41 INFO util.GSet: 2% max memory = 2.475 MB >>>> 12/04/27 11:31:41 INFO util.GSet: capacity = 2^19 = 524288 entries >>>> 12/04/27 11:31:41 INFO util.GSet: recommended=524288, actual=524288 >>>> 12/04/27 11:31:41 ERROR namenode.NameNode: >>>> java.lang.IllegalArgumentException: Invalid attribute value for >>>> hadoop.security.authentication of ${SECURITY_TYPE} >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:202) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:438) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:424) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:473) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:462) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1162) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1271) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: >>>> 1288) >>>> >>>> 12/04/27 11:31:41 INFO namenode.NameNode: SHUTDOWN_MSG: >>>> /************************************************************ >>>> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1 >>>> ************************************************************/ >>>> >>>> >>>> \_________ Solution >>>> I removed the variable and exchanged it with the value "simple". >>>> Then the next error occurred: >>>> >>>> >>>> \_________ Error 2 >>>> hadoop namenode -format >>>> Warning: $HADOOP_HOME is deprecated. >>>> >>>> 12/04/27 11:46:33 INFO namenode.NameNode: STARTUP_MSG: >>>> /************************************************************ >>>> STARTUP_MSG: Starting NameNode >>>> STARTUP_MSG: host = ubuntu/127.0.1.1 >>>> STARTUP_MSG: args = [-format] >>>> STARTUP_MSG: version = 1.0.1 >>>> STARTUP_MSG: build = >>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r >>>> 1243785; compiled by 'hortonfo' on Tue Feb 14 08:13:52 UTC 2012 >>>> ************************************************************/ >>>> 12/04/27 11:46:33 INFO util.GSet: VM type = 32-bit >>>> 12/04/27 11:46:33 INFO util.GSet: 2% max memory = 2.475 MB >>>> 12/04/27 11:46:33 INFO util.GSet: capacity = 2^19 = 524288 entries >>>> 12/04/27 11:46:33 INFO util.GSet: recommended=524288, actual=524288 >>>> 12/04/27 11:46:33 ERROR namenode.NameNode: >>>> java.lang.ExceptionInInitializerError >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:438) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:424) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:473) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:462) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1162) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1271) >>>> at >>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: >>>> 1288) Caused by: java.util.regex.PatternSyntaxException: Illegal >>>> repetition near index 8 [jt]t@.*${KERBEROS_REALM} >>>> ^ >>>> at java.util.regex.Pattern.error(Pattern.java:1730) >>>> at java.util.regex.Pattern.closure(Pattern.java:2792) >>>> at java.util.regex.Pattern.sequence(Pattern.java:1906) >>>> at java.util.regex.Pattern.expr(Pattern.java:1769) >>>> at java.util.regex.Pattern.compile(Pattern.java:1477) >>>> at java.util.regex.Pattern.<init>(Pattern.java:1150) >>>> at java.util.regex.Pattern.compile(Pattern.java:840) >>>> at >>>> org.apache.hadoop.security.KerberosName$Rule.<init>(KerberosName.java:188) >>>> at >>>> org.apache.hadoop.security.KerberosName.parseRules(KerberosName.java:324) >>>> at >>>> org.apache.hadoop.security.KerberosName.setConfiguration(KerberosName.java:343) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239) >>>> at >>>> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83) >>>> ... 10 more >>>> >>>> 12/04/27 11:46:33 INFO namenode.NameNode: SHUTDOWN_MSG: >>>> /************************************************************ >>>> SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1 >>>> ************************************************************/ >>>> >>>> \______ And once again... >>>> ... a variable seems to be undefined. And i guess, if i found a suitable >>>> value for this property the next one will be undefined. I hope there are >>>> any default values present, because i have no idea, what to fill to the >>>> value slots. All books and instructions i read on hadoop never discussed >>>> these issues. >>>> BTW: HADOOP_HOME is defined, although the log tells different. >>>> >>>> I hope you can assist me. >>>> >>>> Best regards, >>>> Björn-Elmar Macek >>>> >
