lhotari commented on a change in pull request #14819:
URL: https://github.com/apache/pulsar/pull/14819#discussion_r834103121
##########
File path: build/pulsar_ci_tool.sh
##########
@@ -46,6 +47,123 @@ function ci_dependency_check() {
_ci_mvn -Pmain,skip-all,skipDocker,owasp-dependency-check initialize verify
-pl '!pulsar-client-tools-test' "$@"
}
+# installs a tool executable if it's not found on the PATH
+function ci_install_tool() {
+ local tool_executable=$1
+ local tool_package=${2:-$1}
+ if ! command -v $tool_executable &>/dev/null; then
+ echo "::group::Installing ${tool_package}"
+ sudo apt-get -y install ${tool_package} >/dev/null
+ echo '::endgroup::'
+ fi
+}
+
+# outputs the given message to stderr and exits the shell script
+function fail() {
+ echo "$*" >&2
+ exit 1
+}
+
+# function to retry a given commend 3 times with a backoff of 10 seconds in
between
+function ci_retry() {
+ local n=1
+ local max=3
+ local delay=10
+ while true; do
+ "$@" && break || {
+ if [[ $n -lt $max ]]; then
+ ((n++))
+ echo "::warning::Command failed. Attempt $n/$max:"
+ sleep $delay
+ else
+ fail "::error::The command has failed after $n attempts."
+ fi
+ }
+ done
+}
+
+# saves a given image (1st parameter) to the GitHub Actions cache with the
given key (2nd parameter)
+function ci_docker_save_image_to_github_actions_cache() {
+ local image=$1
+ local cachekey=$2
+ ci_install_tool pv
+ echo "::group::Saving docker image ${image} with key ${cachekey} in GitHub
Actions Cache"
+ ci_retry bash -c "docker save ${image} | zstd | pv -ft -i 5 | pv -Wbaf -i 5
| curl -s -H 'Content-Type: application/octet-stream' -X PUT --data-binary @-
http://localhost:12321/${cachekey}";
+ echo "::endgroup::"
+}
+
+# loads a docker image from the GitHub Actions cache with the given key (1st
parameter)
+function ci_docker_load_image_from_github_actions_cache() {
+ local cachekey=$1
+ ci_install_tool pv
+ echo "::group::Loading docker image from key ${cachekey} in GitHub Actions
Cache"
+ ci_retry bash -c "curl -s http://localhost:12321/${cachekey} | pv -batf -i 5
| unzstd | docker load"
Review comment:
In apache/pulsar builds, there will be the 10GB limit for GitHub Actions
cache. The cache is a LRU cache based on the [description in actions/cache
README](https://github.com/actions/cache#cache-limits):
> A repository can have up to 10GB of caches. **Once the 10GB limit is
reached, older caches will be evicted based on when the cache was last
accessed.** Caches that are not accessed within the last week will also be
evicted.
This behavior is optimal. Each build will temporarily create about 2.5GB of
cache entries. Since the needed cache entries are accessed frequently, this
will ensure that important caches (the maven dependencies caches) aren't likely
to be dropped. maven dependencies caches are updated with a scheduled job once
a day:
https://github.com/apache/pulsar/blob/master/.github/workflows/ci-maven-cache-update.yaml
The 10GB limit could cause issues if there are too many concurrent build
jobs for apache/pulsar. That problem can be addressed by falling back to
building the binaries if there's a cache miss. That solution could be added
later if problems occur. GitHub actions has also concurrency limits which are
added in this PR. A more recent build will cancel a build that started earlier
for a given branch. In the context of apache/pulsar, the builds will be pushes
to master and branch-* branches. It would be a very rare case where there's
more than 4 branches that are built at the same time. It's possible, but if
that happens, the build can be improved to handle cache misses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
