Thanks so much for kindly advice. This is my fault. I resolved the problem and the root cause is that I misconfigured HADOOP_CLASSPATH, so sorry for confusing and troubling.
But then I am trying to use hadoop yarn to do terasort benckmark test based on cephfs. New exception message occurs as : Does this mean that I cannot use this ceph-hadoop plugin over the hadoop version? Hadoop version is : 2.7.1 release, Ceph version is : 0.94.3 Thanks again for moving this thread. Best, Sun. 15/10/12 11:08:35 INFO client.RMProxy: Connecting to ResourceManager at /172.16.33.18:8032 15/10/12 11:08:35 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: java.lang.NoSuchMethodException: org.apache.hadoop.fs.ceph.CephFS.<init>(java.net.URI, org.apache.hadoop.conf.Configuration) java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75) at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1260) at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1256) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.connect(Job.java:1255) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1284) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:301) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:305) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) From: Paul Evans Date: 2015-10-12 11:10 To: Fulin Sun Subject: Re: [ceph-users] cephfs replace hdfs problem I don’t think there are many of us that have attempted what you are trying to do… that’s the most likely reason the list is quiet. You may need to be patient, and possibly provide updates (if you have any) to keep the issue in front of people. Best of luck... -- Paul On Oct 11, 2015, at 7:03 PM, Fulin Sun <[email protected]> wrote: sign... I had to say that I have not received any reponse from this mailing list... From: Fulin Sun Date: 2015-10-10 17:27 To: ceph-users Subject: [ceph-users] cephfs replace hdfs problem Hi there, I configured hadoop-cephfs plugin and try to use cephfs as a replacement. I had sucessfully configured hadoop-env.sh with setting the HADOOP_CLASSPATH for hadoop-cephfs.jar But when I run hadoop fs -ls /, I got the following exception. Looks like it cannot find the actual jar for both hadoop-cephfs.jar and libcephfs-java.jar I placed these two in the /usr/local/hadoop/lib directory and edited the hadoop classpath in hadoop-env.sh How could this issue be ? Thanks anyone for kind response. java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.ceph.CephFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:355) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.ceph.CephFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 18 more _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
