Jonathan,
My company has their own server suite. We will not be using Amazon for the
final solution. I am only using AWS EC2 instance to compile, test, and package
Hadoop 3.3.1using RHEL 8.4 while our SAs load RHEL 8.4 onto the servers.
Mostly concerned with the *Native* functions, modules, and methods. I removed
the -Dtest=allNative tosee which other test cases fail. Several test cases
fail with errors that I would have thought would have been fixed before
releasing. I assume at this point that these maybe specific to RHEL 8.4 and
wouldn't have been caught with Ubuntu and CentOS 7. Not sure. I did sent
e-mail to hdfs-...@hadoop.apache.org. Waiting to see whatthe developers
recommend.
Appreciate your feedback.
Paula
-----Original Message-----
From: Jonathan Aquilina <jaquil...@eagleeyet.net.INVALID>
To: Paula Logan <pmlin...@verizon.net>; user@hadoop.apache.org
<user@hadoop.apache.org>
Sent: Fri, Sep 10, 2021 5:36 pm
Subject: RE: hadoop-hdfs-native-client Help
<!--#yiv8393330214 _filtered {} _filtered {}#yiv8393330214 #yiv8393330214
p.yiv8393330214MsoNormal, #yiv8393330214 li.yiv8393330214MsoNormal,
#yiv8393330214 div.yiv8393330214MsoNormal
{margin:0cm;font-size:11.0pt;font-family:"Calibri", sans-serif;}#yiv8393330214
span.yiv8393330214EmailStyle18 {font-family:"Calibri",
sans-serif;color:windowtext;}#yiv8393330214 .yiv8393330214MsoChpDefault
{font-family:"Calibri", sans-serif;} _filtered {}#yiv8393330214
div.yiv8393330214WordSection1 {}-->Hi Paula, I am not sure how to answer
your questions but is there a reason why you are using an EC2 instance instead
of amazonz EMR (elastic Map reduce) Hadoop cluster. As far as I know you can
set that up to work with HDFS setup as well as S3 buckets if you don’t need a
long term cluster to stay online. Regards, Jonathan From: Paula Logan
<pmlin...@verizon.net.INVALID>
Sent: 10 September 2021 16:13
To: user@hadoop.apache.org
Subject: hadoop-hdfs-native-client Help Hello, I am new to building
Hadoop locally, and am having some issues. Please let me know if this
information should be sent to a different distro. (1) Can Hadoop 3.3.1 be
compiled and run with OpenJDK 11 or is OpenJDK 1.8 needed for compile while 1.8
or 11 can be used to run hadoop? (2) I am compiling and testing Hadoop
3.3.1 on RHEL 8.4 on the command line not via any IDE inside an AWS instance.
I have encountered an issue with Native Test Case #35 (all other 39
Native Test Cases succeed). First here is my maven command: mvn -e -X
test -Pnative,parallel-tests,shelltest,yarn-ui -Dtest=allNative
-Dparallel-tests=true -Drequire.bzip2=true -Drequire.fuse=true
-Drequire.isal=true -Disal.prefix=/usr/local -Disal.lib=/usr/local/lib64
-Dbundle.isal=true -Drequire.openssl=true -Dopenssl.prefix=/usr
-Dopenssl.include=/usr/include -Dopenssl.lib=/usr/lib64 -Dbundle.openssl=true
-Dbundle.openssl.in.bin=true -Drequire.pmdk=true -Dpmdk.lib=/usr/lib64
-Dbundle.pmdk=true -Drequire.snappy=true -Dsnappy.prefix=/usr
-Dsnappy.include=/usr/include -Dsnappy.lib=/usr/lib64 -Dbundle.snappy=true
-Drequire.valgrind=true -Dhbase.profile=2.0 -Drequire.zstd=true
-Dzstd.prefix=/usr -Dzstd.include=/usr/include -Dzstd.lib=/usr/lib64
-Dbundle.zstd=true -Dbundle.zstd.in.bin=true -Drequire.test.libhadoop=true
This is what I get for Test Case #35: [exec] 35/40 Test #35:
test_libhdfs_threaded_hdfspp_test_shim_static ..............***Failed 31.58
sec [exec] testRecursiveJvmMutex error: [exec]
ClassNotFoundException: RuntimeExceptionjava.lang.NoClassDefFoundError:
RuntimeException [exec] Caused by: java.lang.ClassNotFoundException:
RuntimeException [exec] at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) [exec] at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) [exec] at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [exec]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) [exec] 2021-09-02
22:31:09,706 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(529)) -
starting cluster: numNameNodes=1, numDataNodes=1 [exec] 2021-09-02
22:31:10,134 INFO namenode.NameNode (NameNode.java:format(1249)) - Formatting
using clusterid: testClusterID [exec] 2021-09-02 22:31:10,156 INFO
namenode.FSEditLog (FSEditLog.java:newInstance(229)) - Edit logging is
async:true [exec] 2021-09-02 22:31:10,182 INFO namenode.FSNamesystem
(FSNamesystem.java:<init>(814)) - KeyProvider: null [exec] 2021-09-02
22:31:10,184 INFO namenode.FSNamesystem (FSNamesystemLock.java:<init>(141)) -
fsLock is fair: true [exec] 2021-09-02 22:31:10,185 INFO
namenode.FSNamesystem (FSNamesystemLock.java:<init>(159)) - Detailed lock hold
time metrics enabled: false [exec] 2021-09-02 22:31:10,185 INFO
namenode.FSNamesystem (FSNamesystem.java:<init>(847)) - fsOwner
= ec2-user (auth:SIMPLE) [exec] 2021-09-02 22:31:10,185 INFO
namenode.FSNamesystem (FSNamesystem.java:<init>(848)) - supergroup ...
[exec] 2021-09-02 22:31:13,204 INFO ipc.Server
(Server.java:logException(3020)) - IPC Server handler 7 on default port 44945,
call Call#6 Retry#-1
org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from
127.0.0.1:37362: java.io.FileNotFoundException: File does not exist:
/tlhData0001/file1 ... [exec] 98% tests passed, 1 tests
failed out of 40 [exec] [exec] Total Test time (real) = 270.30 sec
[exec] [exec] The following tests FAILED: [exec] 35 -
test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] Errors while
running CTest [INFO]
------------------------------------------------------------------------ [INFO]
Reactor Summary: [INFO] [INFO] Apache Hadoop Main 3.3.1
........................... SUCCESS [ 0.707 s] [INFO] Apache Hadoop Build
Tools .......................... SUCCESS [ 2.743 s] [INFO] Apache Hadoop
Project POM .......................... SUCCESS [ 0.692 s] [INFO] Apache Hadoop
Annotations .......................... SUCCESS [ 1.955 s] [INFO] Apache Hadoop
Project Dist POM ..................... SUCCESS [ 0.106 s] [INFO] Apache Hadoop
Assemblies ........................... SUCCESS [ 0.101 s] [INFO] Apache Hadoop
Maven Plugins ........................ SUCCESS [ 3.194 s] [INFO] Apache Hadoop
MiniKDC .............................. SUCCESS [ 0.806 s] [INFO] Apache Hadoop
Auth ................................. SUCCESS [ 4.192 s] [INFO] Apache Hadoop
Auth Examples ........................ SUCCESS [ 0.452 s] [INFO] Apache Hadoop
Common ............................... SUCCESS [ 54.493 s] [INFO] Apache Hadoop
NFS .................................. SUCCESS [ 2.123 s] [INFO] Apache Hadoop
KMS .................................. SUCCESS [ 2.087 s] [INFO] Apache Hadoop
Registry ............................. SUCCESS [ 2.538 s] [INFO] Apache Hadoop
Common Project ....................... SUCCESS [ 0.055 s] [INFO] Apache Hadoop
HDFS Client .......................... SUCCESS [ 16.283 s] [INFO] Apache Hadoop
HDFS ................................. SUCCESS [ 24.263 s] [INFO] Apache Hadoop
HDFS Native Client ................... FAILURE [04:49 min] [INFO] Apache Hadoop
HttpFS ............................... SKIPPED ... [INFO]
------------------------------------------------------------------------ [INFO]
BUILD FAILURE [INFO]
------------------------------------------------------------------------ [INFO]
Total time: 06:49 min [INFO] Finished at: 2021-09-02T22:34:18Z [INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (native_tests) on project
hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 8
[ERROR] around Ant part ...<exec failonerror="true"
dir="/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/"
executable="ctest">... @ 6:150 in
/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
[ERROR] -> [Help 1] This RuntimeException error also appears for Native
Test Case 2 but that test case doesn't fail. Also, see a lot of "File not
found" messages. Assume at this point that the NoClassDefFoundError causes the
code that creates the files to be skipped and once the NoClassDefFoundError is
fixed, these files will be generated. The compile of Hadoop 3.3.1 on RHEL
8.4 succeeded without issues. I have JAVA_HOME and JRE_HOME set in .bashrc to
OpenJDK 1.8 and have added these into the $PATH. export
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64 export
JAVA_OPTS="-Xms2048m -Xmx4096m -XX:+UseZGC" export JRE_HOME=${JAVA_HOME}/jre
export LIBHDFS_OPTS="-Xms2048m -Xmx4096m" export MAVEN_HOME=/usr/share/maven
export MAVEN_OPTS="-Xms256m -Xmx1536m" export PROTOBUF_HOME=/usr/local
export
PATH=/home/ec2-user/.local/bin:/home/ec2-user/bin:${JAVA_HOME}/bin:${JRE_HOME}/bin:${MAVEN_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
It appears the $JRE_HOME/lib/rt.jar isn't being included in the
maven.test.classpath or in the native module tests. I tried setting
CLASSPATH and JAVA_LIBRARY_PATH in .bashrc and tried passing via 'mvn' command,
but still no success. I followed the procedures in the BUILDING.txt file for
the CentOS 8 as that was the closest to RHEL 8.4.
${HADOOP_SRC_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml
<properties> <native_cmake_args></native_cmake_args>
<native_ctest_args></native_ctest_args>
<native_make_args></native_make_args> Do I need to supply anything for any
of these three properties? (3) Here is some more snippets of information
from the maven log that I captured. [exec] 98% tests passed, 1 tests
failed out of 40 [exec] [exec] Total Test time (real) = 270.30 sec
[exec] [exec] The following tests FAILED: [exec] 35 -
test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] Errors while
running CTest [INFO]
------------------------------------------------------------------------ [INFO]
Reactor Summary: [INFO] [INFO] Apache Hadoop Main 3.3.1
........................... SUCCESS [ 0.707 s] [INFO] Apache Hadoop Build
Tools .......................... SUCCESS [ 2.743 s] [INFO] Apache Hadoop
Project POM .......................... SUCCESS [ 0.692 s] [INFO] Apache Hadoop
Annotations .......................... SUCCESS [ 1.955 s] [INFO] Apache Hadoop
Project Dist POM ..................... SUCCESS [ 0.106 s] [INFO] Apache Hadoop
Assemblies ........................... SUCCESS [ 0.101 s] [INFO] Apache Hadoop
Maven Plugins ........................ SUCCESS [ 3.194 s] [INFO] Apache Hadoop
MiniKDC .............................. SUCCESS [ 0.806 s] [INFO] Apache Hadoop
Auth ................................. SUCCESS [ 4.192 s] [INFO] Apache Hadoop
Auth Examples ........................ SUCCESS [ 0.452 s] [INFO] Apache Hadoop
Common ............................... SUCCESS [ 54.493 s] [INFO] Apache Hadoop
NFS .................................. SUCCESS [ 2.123 s] [INFO] Apache Hadoop
KMS .................................. SUCCESS [ 2.087 s] [INFO] Apache Hadoop
Registry ............................. SUCCESS [ 2.538 s] [INFO] Apache Hadoop
Common Project ....................... SUCCESS [ 0.055 s] [INFO] Apache Hadoop
HDFS Client .......................... SUCCESS [ 16.283 s] [INFO] Apache Hadoop
HDFS ................................. SUCCESS [ 24.263 s] [INFO] Apache Hadoop
HDFS Native Client ................... FAILURE [04:49 min] [INFO] Apache Hadoop
HttpFS ............................... SKIPPED ... [INFO]
------------------------------------------------------------------------ [INFO]
BUILD FAILURE [INFO]
------------------------------------------------------------------------ [INFO]
Total time: 06:49 min [INFO] Finished at: 2021-09-02T22:34:18Z [INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (native_tests) on project
hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 8
[ERROR] around Ant part ...<exec failonerror="true"
dir="/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/"
executable="ctest">... @ 6:150 in
/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
[ERROR] -> [Help 1] (4) What are the following properties?
require.test.libhadoop - not sure of the purpose of this property or if I need
it or is it just for hadoop project developers bundle.<type> -vs-
bundle.<type>.in.bin - What is the difference of bundle vs bundle.in.bin?
Please let me know what I might be missing or if a (or some) native files need
to be modified to have the rt.jar (RuntimeException class contained within) be
included. Wasn't sure if this was an OpenJDK 1.8 vs OpenJDK 11 issue as the JRE
binary is located in a different directory in OpenJDK 11. I am not using
OpenJDK 11 at all nor is it installed in my RHEL 8.4 AWS instance. I didn't
submit a ticket as I assume there is something that I am not doing correctly or
forgetting to include/do. Any help you can give me would be very much
appreciated. Paula
