Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24583 )
Change subject: KUDU-3788 an option to use pre-built thirdparty artifacts ...................................................................... Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24583/4//COMMIT_MSG@16 PS4, Line 16: * name of the component (cmake, protobuf, etc.) : * version of the component as in $KUDU_HOME/thirdparty/vars.sh : * OS name/flavor (redhat, ubuntu, macos, etc.) : * OS major and minor version (9.2, 24.04, 26.5, etc.) : * CPU architecture (x86_64, aarch64, etc.) : * build toolchain info: : ** compiler family (gcc, clang, etc.) : ** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.) There can be dependencies within the toolchain. For example, curl depends on zlib. If you have an identifier for curl that doesn't know about zlib, then you can end up in a situation where you're using a curl built against zlib X when you're now using zlib Y. With zlib, that probably isn't a problem, but it's a more general theme: Do the identifiers know about transitive dependencies? In Impala's toolchain, we have a variety of these dependencies. Thrift depends on zlib and boost. googletcmalloc and re2 depend on abseil-cpp. orc depends on lz4, protobuf, snappy, zlib, zstd. Etc. A lot of these interfaces are stable, but some are not (e.g. abseil-cpp). We generate the toolchain using a Jenkins job and use the build number as an additional piece of the key. For developers, we have logic to avoid redownloading a package if the version is the same (i.e. if both the old and new build numbers have foo-x.y.z, we don't redownload it). If something goes wrong, someone can delete their toolchain and redownload a coherent copy. The cost is that each build is maintaining its own copy, so our toolchain footprint in the bucket is very large. Separately, matching on a major+minor version can make it harder to populate the cache and get cache hits. For some cases, there aren't other minor versions (e.g. Ubuntu 22.04), but others have several (e.g. Redhat). For Redhat, Impala uses only the major version and builds on a single configuration for all versions (Rocky 9.2). That relies on the newer minor versions being backwards compatible. http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh File thirdparty/prebuilt-utils.sh: http://gerrit.cloudera.org:8080/#/c/24583/4/thirdparty/prebuilt-utils.sh@163 PS4, Line 163: # Building a variable in the xxx_NAME form: vars.sh contains corresponding : # versioned strings for all the 3rd-party components. : local version_var="${name}_VERSION" : local version="${!version_var}" : if [ -z "$version" ]; then : echo "ERROR: $version_var isn't defined: check vars.sh" >&2 : exit 1 : fi : echo ${component_name}-${version} Does this handle applying an additional patch to a component (or changing its build definition slightly)? For Impala, we have a patch level in the version number. -- To view, visit http://gerrit.cloudera.org:8080/24583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143 Gerrit-Change-Number: 24583 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Comment-Date: Mon, 20 Jul 2026 20:45:39 +0000 Gerrit-HasComments: Yes
