Alexey Serbin has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24640
Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts
......................................................................
KUDU-3788 an option to use pre-built thirdparty artifacts
With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.
The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
* name of the component (cmake, protobuf, etc.)
* version of the component as in $KUDU_HOME/thirdparty/vars.sh
* Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
* OS name/flavor (redhat, ubuntu, macos, etc.)
* OS major version (9, 24, 26, etc.) -- this assumes the newer minor
versions of the same major release are backwards compatible
* CPU architecture (x86_64, aarch64, etc.)
* build toolchain info:
** compiler family (gcc, clang, etc.)
** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)
The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature. If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache. After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt. I didn't implement
automatic uploading into the bucket because:
* it makes sense to limit upload and refresh of the pre-built
archives, making sure we distribute only vetted and verified ones
* as for CI, our current upstream CI pipeline isn't supposed to keep
valid S3 credentials in the build environment
To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:
aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive
The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component. This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach. This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed. When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.
Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available. Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.
Below a list of TODOs to address in follow-up changelists:
* address the issue with trace-viewer: it should be packed into
pre-built archive as well, and links should be established
from $KUDU_HOME/www into somewhere under thirdparty/installed
instead of copying files directly into $KUDU_HOME
* avoid unconditionally downloading and unpacking source tarballs:
as of this version, the build script is still downloading and
unpacking the source archives for all the 3rd-party components
when doesn't find corresponding sub-directory in
$KUDU_HOME/thirdparty/src
* add CPU features that are auto-detected and enabled for building
particular 3rd-party components, e.g., AVX2 instructions support
when building bitshuffle
* introduce matching of any CPU architecture and any OS: this
should help with de-duplication of packed JAR content and similar
content (e.g., interpreted scripts, etc.).
Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
more
Change-Id: I46b020c84afcb785463195493dc704acc2a2bebb
---
M CMakeLists.txt
M build-support/run_dist_test.py
M thirdparty/.gitignore
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
A thirdparty/fetch-utils.sh
A thirdparty/prebuilt-utils.sh
M thirdparty/vars.sh
9 files changed, 1,025 insertions(+), 393 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/40/24640/1
--
To view, visit http://gerrit.cloudera.org:8080/24640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I46b020c84afcb785463195493dc704acc2a2bebb
Gerrit-Change-Number: 24640
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <[email protected]>
