[
https://issues.apache.org/jira/browse/IMPALA-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108565#comment-18108565
]
ASF subversion and git services commented on IMPALA-12975:
----------------------------------------------------------
Commit 81733770f0fde939dadf93d0f8d5afcfc1954d81 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=81733770f ]
IMPALA-12795: (addendum) Fix symlink command for ARM Hadoop Client
The fix for IMPALA-12975 changed the logic buildall.sh to
symlink $HADOOP_HOME/lib to point to the ARM hadoop-client
library directory. If the ARM hadoop-client version changes,
the symlink needs to be updated. The current symlink command
doesn't use the -T argument, so it is interpreting the
second argument as a directory rather than the name for the
link. It is trying to create $HADOOP_CLIENT/lib/lib rather
than $HADOOP_CLIENT/lib. This adds the -T argument to ln to
fix this.
Testing:
- Ran a perf-AB-test job that uses two different hadoop-client
versions (and requires updating the symlink)
Change-Id: Id84641158d02f844da011661049e613e97e6a9c5
Reviewed-on: http://gerrit.cloudera.org:8080/24757
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> Rework organization of Hadoop dependency on ARM builds
> ------------------------------------------------------
>
> Key: IMPALA-12975
> URL: https://issues.apache.org/jira/browse/IMPALA-12975
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> The hadoop binaries that we download from the CDP build number are built for
> x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP
> hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces
> (including the C++ build) use these environment variables to find the native
> libraries.
> On ARM, we leave those environment variables pointed to that same location.
> We fix things up by downloading a separate hadoop-client built for ARM, then
> copying the contents into the usual location in the CDP hadoop directory,
> overwriting the x86_64 contents. The code to overwrite the libraries runs on
> each invocation of buildall.sh
> On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded
> hadoop-client (which is built for ARM). With a bit of work on the
> hadoop-client, we could get it to also have the header files and also point
> HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during
> buildall.sh. Any build that wants to pass in a custom hadoop can then use
> HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
