Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )
Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts ...................................................................... Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@30 PS5, Line 30: After that, it's : possible to upload the result into a sub-directory of the designated : S3 bucket s3://cloudera-thirdparty-libs/prebuilt > Question: To leverage this gain, how do we manage the manual upload of pre- As of current state of this, I logged into the jenkins slave nodes and uploaded the newly built archives from there. It's all done in one command line once proper creds are put into the environment. In the long run, I'm planning to take care of upload in a separate changelist -- I don't want to mix it in with this initial update that focuses on providing the base for reducing 3rd-party build times. http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@35 PS5, Line 35: distrute > distribute Done http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45 PS5, Line 45: /opt/kudu/ The 'home' directory approach wouldn't help: with pre-built artifacts, it's necessary to guarantee same path at any machine, while PREFIX-related paths are hard-coded into the binaries and pkgconfig files. If not using same layout for the 3rd-party artifacts everywhere, it'd necessary to always override location of the libraries with LD_LIBRARY_PATH for every invocation of Kudu binaries and tests in runtime, add extra compilation flags, and also do 'manual' update of pkgconfig and other 3rd-party files upon installation. I tried that and it turned to be too involved and very error-prone. > I am guessing single user CI/dev system is the general use case here and > concurrent build by multiple users is not something that needs to be > considered here? That's the only relevant and the most prevalent use case as of time of writing this, correct. Concurrent builds by multiple users is OK if all those users utilize the same version of Kudu 3rd-party. Concurrent builds might end up with links in /opt/kudu/thirdparty pointing to different target directories, but with same versions everywhere it's not a problem. If it's desired to use different 3rd-party versions for different workspaces, the solution is simple: add `export USE_PREBUILT_THIRDPARTY=1` into .bashrc or .profile, and build everything from source in the local workspace. http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@45 PS5, Line 45: uninstrumeted > uninstrumented Done http://gerrit.cloudera.org:8080/#/c/24583/5//COMMIT_MSG@62 PS5, Line 62: Only CMake/GNUmake combination is now used for building : 3rd-party components that rely on CMake to bootstrap build environment > Is there a significant increase in build time if not using ninja, especiall No, I haven't seen any significant increase in total build times for 3rd-party if not using ninja. While using ninja provides a marginal improvement in build times, it's main appeal is a more compact output. I guess a measured reduction in build times using ninja vs make might be seen if we weren't explicitly setting '-j' option for GNU make or having multiple build-recipe-heavy projects like LLVM, but that's not the case here. -- To view, visit http://gerrit.cloudera.org:8080/24583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143 Gerrit-Change-Number: 24583 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Comment-Date: Tue, 28 Jul 2026 18:08:20 +0000 Gerrit-HasComments: Yes
