Duo Zhang created MAPREDUCE-7471:
------------------------------------
Summary: Hadoop mapred minicluster command line fail with class
not found
Key: MAPREDUCE-7471
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7471
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.3.5
Reporter: Duo Zhang
If you run
./bin/mapred minicluster
It will fail with
{noformat}
Exception in thread "Listener at localhost/35325"
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
at
org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2648)
at
org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2662)
at
org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1510)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:989)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:588)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:530)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:160)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 9 more
{noformat}
This line
https://github.com/apache/hadoop/blob/835403d872506c4fa76eb2d721f2d91f413473d5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java#L2648
This is because we rely on mockito in NameNodeAdapter but we do not have
mockito on our classpath, at least in our published hadoop-3.3.5 binary.
And there is another problem that, if we do not run the above command in the
HADOOP_HOME directory, i.e, in another directory by typing the absolute path of
the mapred command, it will fail with
{noformat}
Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
at
org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:336)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:280)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:289)
at
org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:3069)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.<init>(MiniDFSCluster.java:239)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
at
org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 8 mor
{noformat}
This simply because this line
https://github.com/apache/hadoop/blob/835403d872506c4fa76eb2d721f2d91f413473d5/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L601
We should add the $HADOOP_TOOLS_HOME prefix for the default value of
HADOOP_TOOLS_DIR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
