This is an automated email from the ASF dual-hosted git repository. sjanc pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/mynewt-nimble.git
commit 7e69d46f3147c9dff067bab12338cce6f64b762b Author: Szymon Janc <[email protected]> AuthorDate: Fri Oct 16 16:36:01 2026 +0200 ci: Cache external repositories between runs Each job using newt downloaded all external repositories with newt upgrade. Add composite action for setting up newt project which restores repositories from cache before upgrade, so only changes since cache was created are fetched. Only .git directories are cached and working trees (including initialized submodules) are checked out after restore, since git packs are smaller than compressed working trees. Index is removed before checkout, as it contains file stats from system that created it. This also lets single cache created on Linux be used on Windows and macOS, each system checks out files with its own git settings. Repositories still appearing modified after checkout are removed, as newt silently skips upgrade of modified repositories. apache-mynewt-nimble downloaded by newt is replaced with tested commit, as before, and is removed before saving cache so it is never cached. apache-mynewt-core is cached but excluded from cache key, as it is tracked at master and changes often. Single cache is shared by all projects as newt skips repositories not used by project. Cache is saved only from master branch by apps job, which uses project with all repositories, and only when external repositories versions changed. Caches from default branch are also available to pull requests. mcuboot main used by unit tests is now checked out as detached HEAD, as local main branch restored from cache may point to old upstream main and, since repository is shallow, git would reject its update as non-fast-forward. Assisted-by: Claude:claude-opus-5-5 --- .github/actions/newt-project/action.yml | 125 +++++++++++++++++++++++++++++++ .github/scripts/checkout_repos.sh | 54 +++++++++++++ .github/workflows/build_all_apps.yml | 16 ++-- .github/workflows/build_cc_target.yml | 12 +-- .github/workflows/build_targets.yml | 14 +--- .github/workflows/newt_test_all.yml | 18 ++--- .github/workflows/ports_syscfg_check.yml | 11 +-- 7 files changed, 199 insertions(+), 51 deletions(-) diff --git a/.github/actions/newt-project/action.yml b/.github/actions/newt-project/action.yml new file mode 100644 index 000000000..11c746dcd --- /dev/null +++ b/.github/actions/newt-project/action.yml @@ -0,0 +1,125 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +name: Set up newt project +description: > + Create newt project for testing current commit. External repositories + are restored from cache and then upgraded by newt, so only changes + since cache was created are downloaded. apache-mynewt-nimble is + replaced with tested commit and is not cached. Only .git directories + are cached and working trees are checked out after restore, which + keeps cache small and lets each system check out files with its own + git settings. Cache is created on Linux and shared with other + systems. Single cache is shared by all projects since newt skips + repositories not used by project. Cache should be saved only by one + job using project with all repositories and only from master branch, + since only caches from default branch are available to pull requests. + +inputs: + project: + description: Project file to use (relative to repository root) + required: false + default: .github/project.yml + path: + description: Directory to create project in + required: false + default: build + save-cache: + description: Save repositories to cache (Linux and master branch only) + required: false + default: 'false' + +runs: + using: composite + steps: + - name: Create project + id: project + shell: bash + run: | + newt new "${{ inputs.path }}" + cp -f "${{ inputs.project }}" "${{ inputs.path }}/project.yml" + echo "cache-prefix=newt-repos-git-" >> "$GITHUB_OUTPUT" + + - name: Restore repositories from cache + id: restore + uses: actions/cache/restore@v6 + with: + path: ${{ inputs.path }}/repos/*/.git + key: ${{ steps.project.outputs.cache-prefix }}${{ github.sha }} + restore-keys: ${{ steps.project.outputs.cache-prefix }} + # Cache is created on Linux, Windows needs this to restore it + enableCrossOsArchive: true + + # newt silently skips upgrade of modified repositories, so remove + # those still appearing modified after checkout (e.g. due to line + # endings on Windows), so that they are cloned again. + - name: Check out restored repositories + if: steps.restore.outputs.cache-matched-key != '' + shell: bash + working-directory: ${{ inputs.path }} + run: | + bash "$GITHUB_WORKSPACE/.github/scripts/checkout_repos.sh" . + for d in repos/*/; do + if [ -n "$(git -C "$d" status --porcelain --untracked-files=no)" ]; then + echo "::warning::$d modified after restore from cache, removing" + rm -rf "$d" + fi + done + + # apache-mynewt-nimble version downloaded by newt is not used, it is + # replaced with tested commit later. Upgrade is not repeated after + # that, as newt would switch apache-mynewt-nimble back to version + # required by apache-mynewt-core. + - name: Upgrade repositories + shell: bash + working-directory: ${{ inputs.path }} + run: | + newt upgrade --shallow=1 + rm -rf repos/apache-mynewt-nimble + + # Cache key is based on external repositories versions only, so new + # cache is created only when dependencies change and not on every + # commit to apache-mynewt-core. + - name: Compute repositories cache key + id: key + shell: bash + working-directory: ${{ inputs.path }} + run: | + hash=$(for d in repos/*/; do + [ "$d" = "repos/apache-mynewt-core/" ] && continue + echo "$d $(git -C "$d" rev-parse HEAD)" + done | git hash-object --stdin | cut -c1-16) + echo "key=${{ steps.project.outputs.cache-prefix }}$hash" >> "$GITHUB_OUTPUT" + + - name: Save repositories to cache + if: >- + inputs.save-cache == 'true' && runner.os == 'Linux' && + github.ref == 'refs/heads/master' && + github.event_name != 'pull_request' && + steps.restore.outputs.cache-matched-key != steps.key.outputs.key + uses: actions/cache/save@v6 + with: + path: ${{ inputs.path }}/repos/*/.git + key: ${{ steps.key.outputs.key }} + enableCrossOsArchive: true + + - name: Use tested apache-mynewt-nimble + shell: bash + run: | + git clone -q . "${{ inputs.path }}/repos/apache-mynewt-nimble" diff --git a/.github/scripts/checkout_repos.sh b/.github/scripts/checkout_repos.sh new file mode 100755 index 000000000..395e199f6 --- /dev/null +++ b/.github/scripts/checkout_repos.sh @@ -0,0 +1,54 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Recreate working trees of project repositories restored from cache or +# artifact. Only .git directories are stored there, since working trees +# can be checked out from those and are bigger after compression. +# Submodules initialized when repository was cloned are checked out +# from repositories stored in .git/modules. +# +# Usage: checkout_repos.sh <project directory> + +set -e + +cd "$1" + +for d in repos/*/; do + [ -d "$d.git" ] || continue + + # .git may come from other system, use settings of this one + opts=() + if [ "$RUNNER_OS" = "Windows" ]; then + opts=(-c core.symlinks=false -c core.filemode=false) + git -C "$d" config core.symlinks false + git -C "$d" config core.filemode false + fi + + # Index contains file stats from system that created it + rm -f "$d.git/index" + git -C "$d" reset -q --hard + + if [ -f "$d.gitmodules" ]; then + git "${opts[@]}" -C "$d" submodule update -q --recursive + if [ "$RUNNER_OS" = "Windows" ]; then + git -C "$d" submodule foreach -q --recursive \ + "git config core.symlinks false; git config core.filemode false" + fi + fi +done diff --git a/.github/workflows/build_all_apps.yml b/.github/workflows/build_all_apps.yml index b15519fad..c619501d9 100644 --- a/.github/workflows/build_all_apps.yml +++ b/.github/workflows/build_all_apps.yml @@ -45,17 +45,13 @@ jobs: sudo apt-get install -y gcc-multilib - name: Install newt uses: ./.github/actions/newt-install - - name: Setup project + - name: Set up project + uses: ./.github/actions/newt-project + with: + save-cache: true + - name: Remove project targets shell: bash - run: | - newt new build - cp -f .github/project.yml build/project.yml - cd build - newt upgrade --shallow=1 - rm -rf targets - rm -rf repos/apache-mynewt-nimble - git clone .. repos/apache-mynewt-nimble - cd .. + run: rm -rf build/targets - name: Build applications shell: bash run: | diff --git a/.github/workflows/build_cc_target.yml b/.github/workflows/build_cc_target.yml index 2fd069355..3e0c7cabd 100644 --- a/.github/workflows/build_cc_target.yml +++ b/.github/workflows/build_cc_target.yml @@ -44,16 +44,8 @@ jobs: release: ${{ matrix.gcc }} - name: Install newt uses: ./.github/actions/newt-install - - name: Setup project - shell: bash - run: | - newt new build - cp -f .github/project.yml build/project.yml - cd build - newt upgrade --shallow=1 - rm -rf repos/apache-mynewt-nimble - git clone .. repos/apache-mynewt-nimble - cd .. + - name: Set up project + uses: ./.github/actions/newt-project - name: Build test target shell: bash run: | diff --git a/.github/workflows/build_targets.yml b/.github/workflows/build_targets.yml index df595a03f..97de444c4 100644 --- a/.github/workflows/build_targets.yml +++ b/.github/workflows/build_targets.yml @@ -50,17 +50,11 @@ jobs: sudo apt-get install -y gcc-multilib - name: Install newt uses: ./.github/actions/newt-install - - name: Setup project + - name: Set up project + uses: ./.github/actions/newt-project + - name: Remove project targets shell: bash - run: | - newt new build - cp -f .github/project.yml build/project.yml - cd build - newt upgrade --shallow=1 - rm -rf targets - rm -rf repos/apache-mynewt-nimble - git clone .. repos/apache-mynewt-nimble - cd .. + run: rm -rf build/targets - name: Build targets shell: bash run: | diff --git a/.github/workflows/newt_test_all.yml b/.github/workflows/newt_test_all.yml index 03797238c..20d14114a 100644 --- a/.github/workflows/newt_test_all.yml +++ b/.github/workflows/newt_test_all.yml @@ -42,23 +42,17 @@ jobs: sudo apt-get install -y gcc-multilib - name: Install newt uses: ./.github/actions/newt-install - - name: Setup project - shell: bash - run: | - newt new build - cp -f .github/project_newt_test_all.yml build/project.yml - cd build - newt upgrade --shallow=1 - rm -rf repos/apache-mynewt-nimble - git clone .. repos/apache-mynewt-nimble - cd .. + - name: Set up project + uses: ./.github/actions/newt-project + with: + project: .github/project_newt_test_all.yml - name: Switch to main mcuboot # This is temporary until next mcuboot release is available shell: bash run: | cd build/repos/mcuboot - git fetch origin main:main - git checkout main + git fetch --depth=1 origin main + git checkout --detach FETCH_HEAD - name: newt test all shell: bash run: | diff --git a/.github/workflows/ports_syscfg_check.yml b/.github/workflows/ports_syscfg_check.yml index 01a4476de..e24ed9e7d 100644 --- a/.github/workflows/ports_syscfg_check.yml +++ b/.github/workflows/ports_syscfg_check.yml @@ -46,15 +46,8 @@ jobs: sudo apt-get install -y gcc-multilib - name: Install newt uses: ./.github/actions/newt-install - - name: Setup project - shell: bash - run: | - newt new build - cp -f .github/project.yml build/project.yml - cd build - newt upgrade --shallow=1 - rm -rf repos/apache-mynewt-nimble - git clone .. repos/apache-mynewt-nimble + - name: Set up project + uses: ./.github/actions/newt-project - name: Build ports tests targets shell: bash run: |
