Was wondering if you were using an IDE when you ran the Maven commands?
I ran commands from terminal right after checking out rel/release-3.3.1 tag with no modification on source tree. I used maven-3.4.5-5 and cmake-3.18.2-11 provided by AppStream repo. # You need newer cmake (and gcc9 and boost) if you want to build trunk (3.4.0-SNAPSHOT) with native profile. On 2021/09/14 2:06, Paula Logan wrote:
Was wondering if you were using an IDE when you ran the Maven commands? I am not using an IDE; am running on command line. Looks like maven.test.classpath doesn't include rt.jar; does, however, include tools.jar which seems odd to me. -----Original Message----- From: Paula Logan <pmlin...@verizon.net> To: iwasak...@oss.nttdata.co.jp <iwasak...@oss.nttdata.co.jp> Sent: Mon, Sep 13, 2021 9:08 am Subject: Re: hadoop-hdfs-native-client Help Thank you for taking the time to do a check. RHEL 8.x must have some incompatibilities. Will check with Oracle Java next to see if that Test Case #35 passes or fails. -----Original Message----- From: Masatake Iwasaki <iwasak...@oss.nttdata.co.jp> To: Paula Logan <pmlin...@verizon.net>; user@hadoop.apache.org <user@hadoop.apache.org> Sent: Sun, Sep 12, 2021 11:34 pm Subject: Re: hadoop-hdfs-native-client Help > (2) I am compiling and testing Hadoop 3.3.1 on RHEL 8.4 on the command line not via any IDE inside an AWS instance. I have encountered an issue > with Native Test Case #35 (all other 39 Native Test Cases succeed). I could not reproduce the issue on CentOS 8 by java-1.8.0-openjdk-devel-1.8.0.302.b08-0.el8_4:: $ java -version openjdk version "1.8.0_302" OpenJDK Runtime Environment (build 1.8.0_302-b08) OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode) $ echo $JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64 $ mvn clean install -Pnative -DskipTests -DskipShade $ cd hadoop-hdfs-project/hadoop-hdfs-native-client $ mvn test -Pnative -Dtest=x > (4) What are the following properties? > > require.test.libhadoop - not sure of the purpose of this property or if I need it or is it just for hadoop project developers > bundle.<type> -vs- bundle.<type>.in.bin - What is the difference of bundle vs bundle.in.bin? bundle.*.in.bin seems to be for Windows. https://issues.apache.org/jira/browse/HADOOP-12892 <https://issues.apache.org/jira/browse/HADOOP-12892> Those bundle.* properties are used for creating distribution binary tarball of Hadoop (activated by -Pdist on build). It should not be related to the test failure of hadoop-hdfs-native-client. On 2021/09/10 23:12, Paula Logan wrote: > Hello, > > I am new to building Hadoop locally, and am having some issues. Please let me know if this information should be sent to a different distro. > > > (1) Can Hadoop 3.3.1 be compiled and run with OpenJDK 11 or is OpenJDK 1.8 needed for compile while 1.8 or 11 can be used to run hadoop? > > > (2) I am compiling and testing Hadoop 3.3.1 on RHEL 8.4 on the command line not via any IDE inside an AWS instance. I have encountered an issue > with Native Test Case #35 (all other 39 Native Test Cases succeed). > > First here is my maven command: > > mvn -e -X test -Pnative,parallel-tests,shelltest,yarn-ui -Dtest=allNative -Dparallel-tests=true -Drequire.bzip2=true -Drequire.fuse=true -Drequire.isal=true -Disal.prefix=/usr/local -Disal.lib=/usr/local/lib64 -Dbundle.isal=true -Drequire.openssl=true -Dopenssl.prefix=/usr -Dopenssl.include=/usr/include -Dopenssl.lib=/usr/lib64 -Dbundle.openssl=true -Dbundle.openssl.in.bin=true -Drequire.pmdk=true -Dpmdk.lib=/usr/lib64 -Dbundle.pmdk=true -Drequire.snappy=true -Dsnappy.prefix=/usr -Dsnappy.include=/usr/include -Dsnappy.lib=/usr/lib64 -Dbundle.snappy=true -Drequire.valgrind=true -Dhbase.profile=2.0 -Drequire.zstd=true -Dzstd.prefix=/usr -Dzstd.include=/usr/include -Dzstd.lib=/usr/lib64 -Dbundle.zstd=true -Dbundle.zstd.in.bin=true -Drequire.test.libhadoop=true > > This is what I get for Test Case #35: > > [exec] 35/40 Test #35: test_libhdfs_threaded_hdfspp_test_shim_static ..............***Failed 31.58 sec > [exec] testRecursiveJvmMutex error: > [exec] ClassNotFoundException: RuntimeExceptionjava.lang.NoClassDefFoundError: RuntimeException > [exec] Caused by: java.lang.ClassNotFoundException: RuntimeException > [exec] at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > [exec] at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > [exec] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > [exec] at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > [exec] 2021-09-02 22:31:09,706 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(529)) - starting cluster: numNameNodes=1, numDataNodes=1 > [exec] 2021-09-02 22:31:10,134 INFO namenode.NameNode (NameNode.java:format(1249)) - Formatting using clusterid: testClusterID > [exec] 2021-09-02 22:31:10,156 INFO namenode.FSEditLog (FSEditLog.java:newInstance(229)) - Edit logging is async:true > [exec] 2021-09-02 22:31:10,182 INFO namenode.FSNamesystem (FSNamesystem.java:<init>(814)) - KeyProvider: null > [exec] 2021-09-02 22:31:10,184 INFO namenode.FSNamesystem (FSNamesystemLock.java:<init>(141)) - fsLock is fair: true > [exec] 2021-09-02 22:31:10,185 INFO namenode.FSNamesystem (FSNamesystemLock.java:<init>(159)) - Detailed lock hold time metrics enabled: false > [exec] 2021-09-02 22:31:10,185 INFO namenode.FSNamesystem (FSNamesystem.java:<init>(847)) - fsOwner = ec2-user (auth:SIMPLE) > [exec] 2021-09-02 22:31:10,185 INFO namenode.FSNamesystem (FSNamesystem.java:<init>(848)) - supergroup > ... > [exec] 2021-09-02 22:31:13,204 INFO ipc.Server (Server.java:logException(3020)) - IPC Server handler 7 on default port 44945, call Call#6 Retry#-1 org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 127.0.0.1:37362: java.io.FileNotFoundException: File does not exist: /tlhData0001/file1 > ... > [exec] 98% tests passed, 1 tests failed out of 40 > [exec] > [exec] Total Test time (real) = 270.30 sec > [exec] > [exec] The following tests FAILED: > [exec] 35 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] Errors while running CTest > [INFO] ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main 3.3.1 ........................... SUCCESS [ 0.707 s] > [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.743 s] > [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.692 s] > [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.955 s] > [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.106 s] > [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.101 s] > [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.194 s] > [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.806 s] > [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.192 s] > [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 0.452 s] > [INFO] Apache Hadoop Common ............................... SUCCESS [ 54.493 s] > [INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.123 s] > [INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.087 s] > [INFO] Apache Hadoop Registry ............................. SUCCESS [ 2.538 s] > [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.055 s] > [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.283 s] > [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 24.263 s] > [INFO] Apache Hadoop HDFS Native Client ................... FAILURE [04:49 min] > [INFO] Apache Hadoop HttpFS ............................... SKIPPED > ... > [INFO] ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] ------------------------------------------------------------------------ > [INFO] Total time: 06:49 min > [INFO] Finished at: 2021-09-02T22:34:18Z > [INFO] ------------------------------------------------------------------------ > [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (native_tests) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 8 > [ERROR] around Ant part ...<exec failonerror="true" dir="/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/" executable="ctest">... @ 6:150 in /home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml > [ERROR] -> [Help 1] > > > This RuntimeException error also appears for Native Test Case 2 but that test case doesn't fail. > > Also, see a lot of "File not found" messages. Assume at this point that the NoClassDefFoundError causes the code that creates the files to be skipped > and once the NoClassDefFoundError is fixed, these files will be generated. > > The compile of Hadoop 3.3.1 on RHEL 8.4 succeeded without issues. > I have JAVA_HOME and JRE_HOME set in .bashrc to OpenJDK 1.8 and have added these into the $PATH. > > export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64 > export JAVA_OPTS="-Xms2048m -Xmx4096m -XX:+UseZGC" > export JRE_HOME=${JAVA_HOME}/jre > export LIBHDFS_OPTS="-Xms2048m -Xmx4096m" > export MAVEN_HOME=/usr/share/maven > export MAVEN_OPTS="-Xms256m -Xmx1536m" > export PROTOBUF_HOME=/usr/local > export PATH=/home/ec2-user/.local/bin:/home/ec2-user/bin:${JAVA_HOME}/bin:${JRE_HOME}/bin:${MAVEN_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > It appears the $JRE_HOME/lib/rt.jar isn't being included in the maven.test.classpath or in the native module tests. > > I tried setting CLASSPATH and JAVA_LIBRARY_PATH in .bashrc and tried passing via 'mvn' command, but still no success. > > I followed the procedures in the BUILDING.txt file for the CentOS 8 as that was the closest to RHEL 8.4. > > ${HADOOP_SRC_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml > > <properties> > <native_cmake_args></native_cmake_args> > <native_ctest_args></native_ctest_args> > <native_make_args></native_make_args> > > Do I need to supply anything for any of these three properties? > > > (3) Here is some more snippets of information from the maven log that I captured. > > [exec] 98% tests passed, 1 tests failed out of 40 > [exec] > [exec] Total Test time (real) = 270.30 sec > [exec] > [exec] The following tests FAILED: > [exec] 35 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] Errors while running CTest > [INFO] ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main 3.3.1 ........................... SUCCESS [ 0.707 s] > [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.743 s] > [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.692 s] > [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.955 s] > [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.106 s] > [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.101 s] > [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.194 s] > [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.806 s] > [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.192 s] > [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 0.452 s] > [INFO] Apache Hadoop Common ............................... SUCCESS [ 54.493 s] > [INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.123 s] > [INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.087 s] > [INFO] Apache Hadoop Registry ............................. SUCCESS [ 2.538 s] > [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.055 s] > [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.283 s] > [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 24.263 s] > [INFO] Apache Hadoop HDFS Native Client ................... FAILURE [04:49 min] > [INFO] Apache Hadoop HttpFS ............................... SKIPPED > ... > [INFO] ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] ------------------------------------------------------------------------ > [INFO] Total time: 06:49 min > [INFO] Finished at: 2021-09-02T22:34:18Z > [INFO] ------------------------------------------------------------------------ > [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (native_tests) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 8 > [ERROR] around Ant part ...<exec failonerror="true" dir="/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/" executable="ctest">... @ 6:150 in /home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml > [ERROR] -> [Help 1] > > > (4) What are the following properties? > > require.test.libhadoop - not sure of the purpose of this property or if I need it or is it just for hadoop project developers > bundle.<type> -vs- bundle.<type>.in.bin - What is the difference of bundle vs bundle.in.bin? > > > Please let me know what I might be missing or if a (or some) native files need to be modified to have the rt.jar (RuntimeException class contained within) be included. > Wasn't sure if this was an OpenJDK 1.8 vs OpenJDK 11 issue as the JRE binary is located in a different directory in OpenJDK 11. I am not using OpenJDK 11 at all > nor is it installed in my RHEL 8.4 AWS instance. > > I didn't submit a ticket as I assume there is something that I am not doing correctly or forgetting to include/do. > > Any help you can give me would be very much appreciated. > > Paula > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org <mailto:user-unsubscr...@hadoop.apache.org> For additional commands, e-mail: user-h...@hadoop.apache.org <mailto:user-h...@hadoop.apache.org>
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
