[
https://issues.apache.org/jira/browse/HDDS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumitra Sulav updated HDDS-781:
--------------------------------
Description:
HDP integrated with Ambari has a
_/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the
libraries needed for a MR job to run and is copied in the yarn containers at
time of execution.
As introducing ozone filesystem, relevant jars need to be packaged as part of
the tar, also the tar is placed as part of _yum install hadoop_ components done
by Ambari during cluster setup.
During an MR Job run, I faced below java.lang.*NoClassDefFoundError* exceptions
:
{code:java}
org/apache/hadoop/fs/ozone/OzoneFileSystem
org/apache/ratis/proto/RaftProtos$ReplicationLevel
org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum
{code}
Adding the relevant jar in the mentioned tar file resolves the exception.
Complete stacktrace for one of the *NoClassDefFoundError* exception :
{code:java}
2018-10-31 10:03:05,191 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoClassDefFoundError:
org/apache/ratis/proto/RaftProtos$ReplicationLevel
at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64)
at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221)
at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421)
at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214)
at
org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116)
at
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.ClassNotFoundException:
org.apache.ratis.proto.RaftProtos$ReplicationLevel
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 29 more
2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1: java.lang.NoClassDefFoundError:
org/apache/ratis/proto/RaftProtos$ReplicationLevel{code}
was:
HDP integrated with Ambari has a
_/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the
libraries needed for a MR job to run and is copied in the yarn containers at
time of execution.
As introducing ozone filesystem, relevant jars need to be packaged as part of
the tar, also the tar is placed as part of _yum install hadoop_ components done
by Ambari during cluster setup.
During an MR Job run, I faced below java.lang.*NoClassDefFoundError* exceptions
:
{code:java}
org/apache/hadoop/fs/ozone/OzoneFileSystem
org/apache/ratis/proto/RaftProtos$ReplicationLevel
org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum
{code}
Adding the relevant jar in the mentioned tar file resolves the exception.
Complete stacktrace for one of the *NoClassDefFoundError* exception :
{code:java}
2018-10-31 10:03:05,191 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoClassDefFoundError:
org/apache/ratis/proto/RaftProtos$ReplicationLevel
at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64)
at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221)
at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116)
at
org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421)
at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214)
at
org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116)
at
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.ClassNotFoundException:
org.apache.ratis.proto.RaftProtos$ReplicationLevel
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 29 more
2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
with status 1: java.lang.NoClassDefFoundError:
org/apache/ratis/proto/RaftProtos$ReplicationLevel{code}
> Ambari HDP NoClassDefFoundError for MR jobs
> -------------------------------------------
>
> Key: HDDS-781
> URL: https://issues.apache.org/jira/browse/HDDS-781
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
> Affects Versions: 0.3.0
> Reporter: Soumitra Sulav
> Priority: Major
>
> HDP integrated with Ambari has a
> _/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the
> libraries needed for a MR job to run and is copied in the yarn containers at
> time of execution.
> As introducing ozone filesystem, relevant jars need to be packaged as part of
> the tar, also the tar is placed as part of _yum install hadoop_ components
> done by Ambari during cluster setup.
> During an MR Job run, I faced below java.lang.*NoClassDefFoundError*
> exceptions :
> {code:java}
> org/apache/hadoop/fs/ozone/OzoneFileSystem
> org/apache/ratis/proto/RaftProtos$ReplicationLevel
> org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum
> {code}
>
> Adding the relevant jar in the mentioned tar file resolves the exception.
> Complete stacktrace for one of the *NoClassDefFoundError* exception :
> {code:java}
> 2018-10-31 10:03:05,191 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.NoClassDefFoundError:
> org/apache/ratis/proto/RaftProtos$ReplicationLevel
> at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64)
> at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221)
> at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421)
> at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214)
> at
> org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116)
> at
> org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.ratis.proto.RaftProtos$ReplicationLevel
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 29 more
> 2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting
> with status 1: java.lang.NoClassDefFoundError:
> org/apache/ratis/proto/RaftProtos$ReplicationLevel{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
