Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/18018 )
Change subject: [build] Introduce an env variable to indicate the thirdparty path ...................................................................... Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/18018/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18018/2//COMMIT_MSG@15 PS2, Line 15: - Build an docker image with thirdparty libraries well built, then we > The Kudu docker doc [1] says images like 'apache/kudu:thirdparty-[OS]-[VERS I'd discussed this with Grant when he contributed the Docker deployments. IIRC, he explicitly opted not to package thirdparty and store it in DockerHub because of space concerns. I.e. the thirdparty Docker build should be functional today, we just don't publish to DockerHub because in general, the DockerHub images are very stripped down and are the bare minimum to get Kudu running. That isn't to say we shouldn't host a thirdparty build, but it does come with some other questions. Right now, DockerHub only hosts released versions. If we upload a thirdparty image, should that change? Or should we upload a new thirdparty image every time thirdparty changes? If the goal is developer velocity, or faster pre-commits, it'd be important that the thirdparty image be as up-to-date as possible. But that could also mean it's more confusing to reason about what versions are hosted on DockerHub. Specifically regarding faster pre-commits in a CI/CD pipeline, one approach that we've taken is to have a couple of long-lived machines that keep a long-lived workspace when building Kudu. Upon building a patched version of Kudu in this workspace, Kudu's build detects whether anything has changed in thirdparty compared to the cached version, and potentially doesn't build anything. http://gerrit.cloudera.org:8080/#/c/18018/2/build-support/jenkins/build-and-test.sh File build-support/jenkins/build-and-test.sh: http://gerrit.cloudera.org:8080/#/c/18018/2/build-support/jenkins/build-and-test.sh@314 PS2, Line 314: : if [ ! -n "${THIRDPARTY_DIR}" ]; then : THIRDPARTY_DIR=$SOURCE_ROOT/thirdparty : fi nit: consider using the same syntax used elsewhere? Same in other files, if you want. E.g. similar to export KUDU_FLAKY_TEST_ATTEMPTS=${KUDU_FLAKY_TEST_ATTEMPTS:-1} export KUDU_ALLOW_SLOW_TESTS=${KUDU_ALLOW_SLOW_TESTS:-$DEFAULT_ALLOW_SLOW_TESTS} export KUDU_COMPRESS_TEST_OUTPUT=${KUDU_COMPRESS_TEST_OUTPUT:-1} export TEST_TMPDIR=${TEST_TMPDIR:-/tmp/kudutest-$UID} export PARALLEL=${PARALLEL:-$(getconf _NPROCESSORS_ONLN)} export PARALLEL_TESTS=${PARALLEL_TESTS:-$PARALLEL} -- To view, visit http://gerrit.cloudera.org:8080/18018 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5e0bdf4faa44322622c48cacadf8e1165eccd38a Gerrit-Change-Number: 18018 Gerrit-PatchSet: 3 Gerrit-Owner: Yingchun Lai <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Attila Bukor <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai <[email protected]> Gerrit-Comment-Date: Fri, 12 Nov 2021 19:33:43 +0000 Gerrit-HasComments: Yes
