Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs
On Mon, Jan 06, 2020 at 02:34:02PM +, Anthony PERARD wrote:
> On Fri, Jan 03, 2020 at 02:29:07PM +, Wei Liu wrote:
> > On Thu, Dec 19, 2019 at 02:42:17PM +, Anthony PERARD wrote:
> > > GitLab have a caching capability, see [1]. Let's use it to avoid using
> > > Internet too often.
> > >
> > > The cache is setup so that when xen.git/Config.mk is changed, the
> > > cache will need to be recreated. This has been chosen because that is
> > > where the information about how to clone sub-project trees is encoded
> > > (revisions). That may not work for qemu-xen tree which usually is
> > > `master', but that should be fine for now.
> > >
> > > The cache is populated of "git bundle" which will contain a mirror of
> > > the original repo, and can be cloned from. If the bundle exist, the
> > > script have the Xen makefiles clone from it, otherwise it will clone
> > > from the original URL and the bundles will be created just after.
> > >
> > > We have more than one runner in GitLab, and no shared cache between
> > > them, so every build jobs will be responsible to create the cache.
> > >
> > > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> > >
> > > Signed-off-by: Anthony PERARD
> >
> > This is a good improvement.
> >
> > Have you run this in Gitlab CI? Can you point me to a run?
>
> I have use the CI to develop the patch, so yes I have a run of it. But
> it is a run made with my wip branch, still it should be the same result
> if it was done with the final patch:
> https://gitlab.com/xen-project/people/anthonyper/xen/pipelines/104343621
This looks good to me.
>
> > > diff --git a/automation/scripts/prepare-cache.sh
> > > b/automation/scripts/prepare-cache.sh
> > > new file mode 100755
> > > index ..017f1b8f0672
> > > --- /dev/null
> > > +++ b/automation/scripts/prepare-cache.sh
> > > @@ -0,0 +1,52 @@
> > > +#!/bin/bash
> > > +
> > > +set -ex
> > > +
> > > +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> > > +mkdir -p "$cachedir"
> > > +
> > > +declare -A r
> > > +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> > > +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> > > +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> > > +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> > > +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
> >
> > Does this mean if in the future we add or remove trees we will need to
> > modify this part in the same commit?
>
> We would need to modify the script when trees are removed, because I
> haven't thought of that. But when trees are added, the script can be
> changed in a follow-up.
>
> Ideally, we would use the Makefiles to discovers the git clones that can
> be cached, but that's not possible just yet.
>
> In the mean time, I think I should make the script more robust against
> removal of trees, so it doesn't have to be modified in the same commit.
OK. I'm expecting a new version then.
Wei.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs
On Fri, Jan 03, 2020 at 02:29:07PM +, Wei Liu wrote:
> On Thu, Dec 19, 2019 at 02:42:17PM +, Anthony PERARD wrote:
> > GitLab have a caching capability, see [1]. Let's use it to avoid using
> > Internet too often.
> >
> > The cache is setup so that when xen.git/Config.mk is changed, the
> > cache will need to be recreated. This has been chosen because that is
> > where the information about how to clone sub-project trees is encoded
> > (revisions). That may not work for qemu-xen tree which usually is
> > `master', but that should be fine for now.
> >
> > The cache is populated of "git bundle" which will contain a mirror of
> > the original repo, and can be cloned from. If the bundle exist, the
> > script have the Xen makefiles clone from it, otherwise it will clone
> > from the original URL and the bundles will be created just after.
> >
> > We have more than one runner in GitLab, and no shared cache between
> > them, so every build jobs will be responsible to create the cache.
> >
> > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> >
> > Signed-off-by: Anthony PERARD
>
> This is a good improvement.
>
> Have you run this in Gitlab CI? Can you point me to a run?
I have use the CI to develop the patch, so yes I have a run of it. But
it is a run made with my wip branch, still it should be the same result
if it was done with the final patch:
https://gitlab.com/xen-project/people/anthonyper/xen/pipelines/104343621
> > diff --git a/automation/scripts/prepare-cache.sh
> > b/automation/scripts/prepare-cache.sh
> > new file mode 100755
> > index ..017f1b8f0672
> > --- /dev/null
> > +++ b/automation/scripts/prepare-cache.sh
> > @@ -0,0 +1,52 @@
> > +#!/bin/bash
> > +
> > +set -ex
> > +
> > +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> > +mkdir -p "$cachedir"
> > +
> > +declare -A r
> > +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> > +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> > +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> > +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> > +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
>
> Does this mean if in the future we add or remove trees we will need to
> modify this part in the same commit?
We would need to modify the script when trees are removed, because I
haven't thought of that. But when trees are added, the script can be
changed in a follow-up.
Ideally, we would use the Makefiles to discovers the git clones that can
be cached, but that's not possible just yet.
In the mean time, I think I should make the script more robust against
removal of trees, so it doesn't have to be modified in the same commit.
Thanks,
--
Anthony PERARD
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs
On Thu, Dec 19, 2019 at 02:42:17PM +, Anthony PERARD wrote:
> GitLab have a caching capability, see [1]. Let's use it to avoid using
> Internet too often.
>
> The cache is setup so that when xen.git/Config.mk is changed, the
> cache will need to be recreated. This has been chosen because that is
> where the information about how to clone sub-project trees is encoded
> (revisions). That may not work for qemu-xen tree which usually is
> `master', but that should be fine for now.
>
> The cache is populated of "git bundle" which will contain a mirror of
> the original repo, and can be cloned from. If the bundle exist, the
> script have the Xen makefiles clone from it, otherwise it will clone
> from the original URL and the bundles will be created just after.
>
> We have more than one runner in GitLab, and no shared cache between
> them, so every build jobs will be responsible to create the cache.
>
> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
>
> Signed-off-by: Anthony PERARD
This is a good improvement.
Have you run this in Gitlab CI? Can you point me to a run?
> ---
> automation/gitlab-ci/build.yaml | 8 +
> automation/scripts/prepare-cache.sh | 52 +
> 2 files changed, 60 insertions(+)
> create mode 100755 automation/scripts/prepare-cache.sh
>
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 1e61d30c8545..8f9f53a4222f 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -1,6 +1,14 @@
> .build-tmpl:
>stage: build
>image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> + cache:
> +key:
> + files:
> +- Config.mk
> +paths:
> + - ci_cache
> + before_script:
> +- ./automation/scripts/prepare-cache.sh
>script:
> - ./automation/scripts/build 2>&1 | tee build.log
>artifacts:
> diff --git a/automation/scripts/prepare-cache.sh
> b/automation/scripts/prepare-cache.sh
> new file mode 100755
> index ..017f1b8f0672
> --- /dev/null
> +++ b/automation/scripts/prepare-cache.sh
> @@ -0,0 +1,52 @@
> +#!/bin/bash
> +
> +set -ex
> +
> +cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
> +mkdir -p "$cachedir"
> +
> +declare -A r
> +r[extras/mini-os]=MINIOS_UPSTREAM_URL
> +r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
> +r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
> +r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
> +r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
Does this mean if in the future we add or remove trees we will need to
modify this part in the same commit?
Wei.
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
[Xen-devel] [XEN PATCH 2/2] automation: Cache sub-project git tree in build jobs
GitLab have a caching capability, see [1]. Let's use it to avoid using
Internet too often.
The cache is setup so that when xen.git/Config.mk is changed, the
cache will need to be recreated. This has been chosen because that is
where the information about how to clone sub-project trees is encoded
(revisions). That may not work for qemu-xen tree which usually is
`master', but that should be fine for now.
The cache is populated of "git bundle" which will contain a mirror of
the original repo, and can be cloned from. If the bundle exist, the
script have the Xen makefiles clone from it, otherwise it will clone
from the original URL and the bundles will be created just after.
We have more than one runner in GitLab, and no shared cache between
them, so every build jobs will be responsible to create the cache.
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#cache
Signed-off-by: Anthony PERARD
---
automation/gitlab-ci/build.yaml | 8 +
automation/scripts/prepare-cache.sh | 52 +
2 files changed, 60 insertions(+)
create mode 100755 automation/scripts/prepare-cache.sh
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 1e61d30c8545..8f9f53a4222f 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -1,6 +1,14 @@
.build-tmpl:
stage: build
image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+ cache:
+key:
+ files:
+- Config.mk
+paths:
+ - ci_cache
+ before_script:
+- ./automation/scripts/prepare-cache.sh
script:
- ./automation/scripts/build 2>&1 | tee build.log
artifacts:
diff --git a/automation/scripts/prepare-cache.sh
b/automation/scripts/prepare-cache.sh
new file mode 100755
index ..017f1b8f0672
--- /dev/null
+++ b/automation/scripts/prepare-cache.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+
+set -ex
+
+cachedir="${CI_PROJECT_DIR:=`pwd`}/ci_cache"
+mkdir -p "$cachedir"
+
+declare -A r
+r[extras/mini-os]=MINIOS_UPSTREAM_URL
+r[tools/qemu-xen-dir]=QEMU_UPSTREAM_URL
+r[tools/qemu-xen-traditional-dir]=QEMU_TRADITIONAL_URL
+r[tools/firmware/ovmf-dir]=OVMF_UPSTREAM_URL
+r[tools/firmware/seabios-dir]=SEABIOS_UPSTREAM_URL
+
+bundle_loc() {
+echo "$cachedir/${1//\//_}.git.bundle"
+}
+for d in ${!r[@]}; do
+if [ -e $(bundle_loc $d) ]; then
+export ${r[$d]}=$(bundle_loc $d)
+fi
+done
+
+if ! make subtree-force-update-all; then
+# There's maybe an issue with one of the git bundle, just clear the cache
+# and allow it to be rebuilt by a different jobs.
+# Make will reclone missing clones from original URLs instead of from the
+# bundle.
+for d in ${!r[@]}; do
+rm -f "$(bundle_loc $d)"
+done
+exit
+fi
+
+
+tmpdir=$(mktemp -d "$CI_PROJECT_DIR/ci-tmp.XXX")
+for d in ${!r[@]}; do
+bundle=$(bundle_loc $d)
+if [ -e $bundle ]; then
+# We didn't download anything new
+continue
+fi
+# We create a mirror to be able to create a bundle that is a mirror of
+# upstream. Otherwise, the bundle may not have refs that the build system
+# will want, i.e. refs/heads/master would be missing from the bundle.
+url=$(git --git-dir=$d/.git config remote.origin.url)
+repo_mirrored="$tmpdir/${d//\//_}"
+git clone --bare --mirror --reference "$d" "$url" "$repo_mirrored"
+git --git-dir="$repo_mirrored" bundle create $bundle --all
+rm -rf "$repo_mirrored"
+done
+rmdir "$tmpdir"
--
Anthony PERARD
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
