Joe McDonnell created IMPALA-12975:
--------------------------------------
Summary: Rework organization of Hadoop dependency on ARM builds
Key: IMPALA-12975
URL: https://issues.apache.org/jira/browse/IMPALA-12975
Project: IMPALA
Issue Type: Task
Components: Infrastructure
Reporter: Joe McDonnell
The hadoop binaries that we download from the CDP build number are built for
x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP
hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces
(including the C++ build) use these environment variables to find the native
libraries.
On ARM, we leave those environment variables pointed to that same location. We
fix things up by downloading a separate hadoop-client built for ARM, then
copying the contents into the usual location in the CDP hadoop directory,
overwriting the x86_64 contents. The code to overwrite the libraries runs on
each invocation of buildall.sh
On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded
hadoop-client (which is built for ARM). With a bit of work on the
hadoop-client, we could get it to also have the header files and also point
HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during
buildall.sh. Any build that wants to pass in a custom hadoop can then use
HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
