Hi again, after a bit of work it finally runs the job. The problem was that I always tried to run the MapReduce-jobs directly from eclipse. This worked with Hadoop 1.2.1 but not with Hadoop 2.4.1. Running the job with ./hadoop jar mapredtest.jar works fine with both versions. So here I have an additional question: Is there a way to run MapReduce-Jobs directly from code without creating a jar first?
Best Regards Malte 2014-08-28 17:57 GMT+02:00 Ted Yu <[email protected]>: > Maybe mapred-site.xml was not in classpath ? > It should have something like this: > > <property> > <name>mapreduce.framework.name</name> > <value>yarn</value> > </property> > > Cheers > > > On Thu, Aug 28, 2014 at 5:51 AM, Malte Maltesmann < > [email protected]> wrote: > > > I did not use maven earlier for this, but if I try it with maven and the > > following pom.xml: > > > > <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi=" > > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > > http://maven.apache.org/POM/4.0.0 > > http://maven.apache.org/xsd/maven-4.0.0.xsd";> > > <modelVersion>4.0.0</modelVersion> > > <groupId>de.test.malte</groupId> > > <artifactId>MapRedMaven</artifactId> > > <version>0.0.1-SNAPSHOT</version> > > <dependencies> > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-common</artifactId> > > <version>2.4.1</version> > > </dependency> > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-hdfs</artifactId> > > <version>2.4.1</version> > > </dependency> > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-mapreduce-client-core</artifactId> > > <version>2.4.1</version> > > </dependency> > > </dependencies> > > </project> > > > > I get the following exception on both, Linux and FreeBSD: > > > > 14/08/28 14:45:56 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > Exception in thread "main" java.io.IOException: Cannot initialize > Cluster. > > Please check your configuration for mapreduce.framework.name and the > > correspond server addresses. > > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) > > at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82) > > at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75) > > at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1255) > > at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1251) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1250) > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1279) > > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303) > > at > > NewMaxTemperature$NewMaxTemperatureMapper.main(NewMaxTemperature.java:39) > > > > By the way, I am now trying to run this > > < > > > http://gerrymcnicol.com/index.php/2014/01/02/hadoop-and-cassandra-part-4-writing-your-first-mapreduce-job/ > > > > > example on only one node. It runs fine on Linux without using maven but > > crashes on Linux and FreeBSD with maven with the above Exception and > > without maven on FreeBSD with the NoSuchElementException. > > > > Regards > > Malte > > > > > > 2014-08-27 16:44 GMT+02:00 Artem Ervits <[email protected]>: > > > > > Check your pom.xml, some artifacts changed from earlier releases. > > > > > > > > > Artem Ervits > > > Data Analyst > > > New York Presbyterian Hospital > > > > > > ----- Original Message ----- > > > From: Malte Otten [mailto:[email protected]] > > > Sent: Monday, August 25, 2014 02:12 PM > > > To: [email protected] <[email protected]> > > > Subject: Re: NoSuchElementException while executing MapReduce-job > > > > > > Hi Ted, > > > > > > thanks for your quick answer. > > > I don't think I'm running in a secure setup, because I didn't do any of > > > the things specified in the reference guide chapter from your link and > > > as far as I can see, the secure setup is not the default. My > > > hbase-site.xml looks like the following: > > > > > > <configuration> > > > <property> > > > <name>hbase.cluster.distributed</name> > > > <value>true</value> > > > </property> > > > <property> > > > <name>hbase.master.wait.on.regionservers.mintostart</name> > > > <value>1</value> > > > </property> > > > <property> > > > <name>hbase.rootdir</name> > > > <value>hdfs://my.hbase.server.de:8020/hbase</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.property.dataDir</name> > > > <value>/my/path/hbase-data/zookeeper</value> > > > </property> > > > <property> > > > <name>hbase.zookeeper.quorum</name> > > > <value>my.hbase.server.de</value> > > > </property> > > > </configuration> > > > > > > Do you have any other suggestions? > > > > > > Regards > > > Malte > > > > > > > > > Am 25.08.2014 um 18:01 schrieb Ted Yu: > > > > Looks like you were running in a secure setup. > > > > > > > > If that's the case, you should read: > > > > http://hbase.apache.org/book.html#hbase.secure.configuration > > > > > > > > Cheers > > > > > > > > > > > > On Mon, Aug 25, 2014 at 5:48 AM, Malte Maltesmann < > > > > [email protected]> wrote: > > > > > > > >> Hi all, > > > >> > > > >> when I'm trying to execute a MapReduce-Job over the HBase-Java-API I > > > get an > > > >> exception with the follwoing Stacktrace: > > > >> > > > >> java.util.NoSuchElementException > > > >> at > java.util.StringTokenizer.nextToken(StringTokenizer.java:349) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:565) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:534) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:276) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:240) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389) > > > >> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) > > > >> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) > > > >> at java.security.AccessController.doPrivileged(Native Method) > > > >> at javax.security.auth.Subject.doAs(Subject.java:415) > > > >> at > > > >> > > > >> > > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > > > >> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) > > > >> at > > org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303) > > > >> ... > > > >> > > > >> In the RawLocalFileSystem.java I can see, that its trying to get > some > > > >> Permissions here, but the StringTokenizer gets an empty String, so > the > > > >> first t.nextToken() fails with the Exception above: > > > >> > > > >> /// loads permissions, owner, and group from `ls -ld` > > > >> private void loadPermissionInfo() { > > > >> IOException e = null; > > > >> try { > > > >> String output = FileUtil.execCommand(new > > > File(getPath().toUri()), > > > >> Shell.getGetPermissionCommand()); > > > >> StringTokenizer t = > > > >> new StringTokenizer(output, > Shell.TOKEN_SEPARATOR_REGEX); > > > >> //expected format > > > >> //-rw------- 1 username groupname ... > > > >> String permission = t.nextToken(); > > > >> if (permission.length() > > > FsPermission.MAX_PERMISSION_LENGTH) { > > > >> //files with ACLs might have a '+' > > > >> permission = permission.substring(0, > > > >> FsPermission.MAX_PERMISSION_LENGTH); > > > >> } > > > >> setPermission(FsPermission.valueOf(permission)); > > > >> t.nextToken(); > > > >> > > > >> String owner = t.nextToken(); > > > >> // If on windows domain, token format is DOMAIN\\user and > we > > > want > > > >> to > > > >> // extract only the user name > > > >> if (Shell.WINDOWS) { > > > >> int i = owner.indexOf('\\'); > > > >> if (i != -1) > > > >> owner = owner.substring(i + 1); > > > >> } > > > >> setOwner(owner); > > > >> > > > >> setGroup(t.nextToken()); > > > >> } catch (Shell.ExitCodeException ioe) { > > > >> if (ioe.getExitCode() != 1) { > > > >> e = ioe; > > > >> } else { > > > >> setPermission(null); > > > >> setOwner(null); > > > >> setGroup(null); > > > >> } > > > >> } catch (IOException ioe) { > > > >> e = ioe; > > > >> } finally { > > > >> if (e != null) { > > > >> throw new RuntimeException("Error while running command > to > > > get " > > > >> + > > > >> "file permissions : " + > > > >> > > > StringUtils.stringifyException(e)); > > > >> } > > > >> } > > > >> } > > > >> > > > >> So my question is, what could have caused this Problem. My setup is > > the > > > >> following: > > > >> > > > >> Two nodes running on FreeBSD 9.0 > > > >> Hadoop 2.4.1 > > > >> HBase 0.98.4 > > > >> > > > >> The shell-command "ls -ld" delivers the same output on FreeBSD as on > > any > > > >> other Linux-distribution, so this shouldn't be the problem. > > > >> Also I can use simple operations from the HBase-API like get or put > > > without > > > >> any problems. > > > >> > > > >> I also posted this on stackoverflow > > > >> < > > > >> > > > > > > http://stackoverflow.com/questions/25364802/hbase-mapreduce-nosuchelementexception > > > >> a week ago, but didn't get any answers. > > > >> > > > >> Thanks in advance, > > > >> Malte > > > >> > > > > > > > > > ________________________________ > > > > > > This electronic message is intended to be for the use only of the named > > > recipient, and may contain information that is confidential or > > privileged. > > > If you are not the intended recipient, you are hereby notified that any > > > disclosure, copying, distribution or use of the contents of this > message > > is > > > strictly prohibited. If you have received this message in error or are > > not > > > the named recipient, please notify us immediately by contacting the > > sender > > > at the electronic mail address noted above, and delete and destroy all > > > copies of this message. Thank you. > > > > > > This electronic message is intended to be for the use only of the named > > > recipient, and may contain information that is confidential or > > privileged. > > > If you are not the intended recipient, you are hereby notified that any > > > disclosure, copying, distribution or use of the contents of this > message > > is > > > strictly prohibited. If you have received this message in error or are > > not > > > the named recipient, please notify us immediately by contacting the > > sender > > > at the electronic mail address noted above, and delete and destroy all > > > copies of this message. Thank you. > > > > > >
