[
https://issues.apache.org/jira/browse/HDDS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670800#comment-16670800
]
Soumitra Sulav commented on HDDS-781:
-------------------------------------
Above exceptions were resolved by added mentioned jar containing the class in
*hadoop/share/hadoop/mapreduce/lib/* folder of the mapreduce.tar.gz :
{code:java}
org/apache/hadoop/fs/ozone/OzoneFileSystem -
hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar
org/apache/ratis/proto/RaftProtos$ReplicationLevel -
ratis-proto-0.3.0-9b2d7b6-SNAPSHOT.jar
org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum -
ratis-thirdparty-0.1.0-SNAPSHOT.jar{code}
> Ambari HDP NoClassDefFoundError for MR jobs
> -------------------------------------------
>
> Key: HDDS-781
> URL: https://issues.apache.org/jira/browse/HDDS-781
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
> Affects Versions: 0.3.0
> Reporter: Soumitra Sulav
> Priority: Major
>
> HDP integrated with Ambari has a
> _/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the
> libraries needed for a MR job to run and is copied in the yarn containers at
> time of execution.
> As introducing ozone filesystem, relevant jars need to be packaged as part of
> the tar, also the tar is placed as part of _yum install hadoop_ components
> done by Ambari during cluster setup.
> During an MR Job run, I faced below java.lang.*NoClassDefFoundError*
> exceptions :
>
> {code:java}
> org/apache/hadoop/fs/ozone/OzoneFileSystem
> org/apache/ratis/proto/RaftProtos$ReplicationLevel
> org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum
> {code}
>
> Adding the relevant jar in the mentioned tar file resolves the exception.
>
> Complete stacktrace for one of the *NoClassDefFoundError* exception :
> {code:java}
> 2018-10-31 10:03:05,191 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.NoClassDefFoundError:
> org/apache/ratis/proto/RaftProtos$ReplicationLevel
> at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64)
> at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221)
> at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421)
> at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214)
> at
> org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116)
> at
> org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.ratis.proto.RaftProtos$ReplicationLevel
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 29 more
> 2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.NoClassDefFoundError:
> org/apache/ratis/proto/RaftProtos$ReplicationLevel{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
