Hi,
based on Sammy's work I've set up another Azure build matrix behind a nice url:
https://dev.azure.com/gentoo-prefix/ci-builds/_build
This does bootstrap Gentoo Prefix (using self hosted Windows Server
build agents for the long running Cygwin builds) in all these setups:
---------------------+---------------+---------------+
| Prefix | RAP |
64bit OS | 64bit | 32bit | 64bit | 32bit |
---------------------+-------+-------+-------+-------+
Fedora current | yes | yes | yes | yes |
Fedora 28 | yes | yes | yes | yes |
OpenSUSE current | yes | yes | yes | yes |
Ubuntu current | - | - | yes | yes |
Ubuntu 16.04 | - | - | yes | yes |
MacOS current | yes | - | - | - |
Cygwin Server 2012R2 | yes | - | - | - |
Cygwin Server 2016 | yes | - | - | - |
Cygwin Server 2019 | yes | - | - | - |
---------------------+-------+-------+-------+-------+
One Cygwin agent on Windows Server 2019 also does submit results to
http://bootstrap.prefix.bitzolder.nl/results/
Uploading docker images with bootstrapped /tmp/gentoo Prefix I'm happy
to leave up to Sammy's jobs still.
Thanks!
/haubi/
On 11/26/18 5:02 AM, Sam Pfeiffer wrote:
> Hello,
>
> First, let me apologise if I have any wrong preexisting ideas/guesses about
> Gentoo Prefix and how it is developed. Secondly, sorry if this email is a bit
> too long.
>
> I'll present myself, explain my use-case and my troubles and tricks, and then
> ask for feedback.
>
> Hello, I'm Sammy Pfeiffer. I am a PhD student at University of Technology
> Sydney. I'm a (software) robotics engineer pursuing my PhD in a lab with a
> bunch of different robots and machine learning setups. I'm using Gentoo
> Prefix for deploying a big bunch of software into robots where the OS is old
> and frozen (and also to deploy in computation farms experiments... once again
> with no option to touch the OS). (The bunch of software is ROS, the 'Robotics
> Operating System' which has a ton of dependencies, and there is an existing
> overlay which I'm helping on maintaining and improving
> (https://github.com/ros/ros-overlay)).
>
> I'd like to share the trick I found to overcome a few annoying bits of my
> platforms:
> * I have no root access in the machines I need to deploy.
> * The machines don't have Docker installed (too old kernel anyways).
> * I have 32 bit and 64 bit machines (the OS running in them is).
> * I have a different username (and home folder) on each machine.
> * Each machine has a different disk(s)/partition(s) structure (main issue
> here is that I may need to use a different disk to store my data, cause of
> size constrains).
>
> The trick to be able to use Gentoo Prefix with all these constraints
> bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.
> Then you can deploy the full bootstrapped system in any folder/disk and just
> do a softlink to /tmp/gentoo. This works nicely (I was scared of the softlink
> breaking stuff somewhere, but it was alright).
> The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run in
> any 32b or 64b box (it's not ideal but simplifies my deployment currently).
>
> Also, having all these machines use the same Gentoo Prefix in the same place,
> with the same 32bit compilation, I can setup a binary package server, which
> all the deployments can point to and just get all the packages skipping the
> long compilation times (specially in very old and low powered machines).
>
> The final trick I'm experimenting with is to use a set of Docker images (and
> soon in a continuous integration environment) to bootstrap all the system.
> With that I can save snapshot of successfully built systems & packages to
> serve as an easy deployment (and easy installation of extra packages thru the
> binary package server).
>
> Given my particular annoyance of the 32bit system (and noting that my hosts
> are 64bits, as is the standard nowadays) I found that I can use either:
> * https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
> * https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
> Docker image
>
> And then execute the build step of Docker with:
>
> setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
>
> Which will trick any program trying to do uname -m to assume 32bit machine.
> (Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping,
> but I found some problems, which didn't appear with this method).
>
> Once I have all this setup working nicely, I'd like to trigger rebuilds every
> X time, and on changes on main players (or all dependencies actually) of my
> setup, like the bootstrapping of Gentoo Prefix (Ideally, on every change of a
> part of the system just trigger a rebuild from that point on, Docker layers
> make this possible).
>
> With Azure announcing unlimited minutes on CI/CD for open source projects:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
>
> Even bootstrapping Gentoo prefix, with pieces of software like gcc taking
> very long to compile, is possible.
>
> The point is: I have been trying to build Gentoo Prefix during the last days
> after a few months of break since the last time I touched the system. And
> it's failing. I haven't managed yet to bootstrap it completely. I feel there
> is no CI/CD setup to catch these issues and be able to offer a working
> version of Gentoo Prefix at any time.
>
> I was going to build it for myself (cause I need it), but instead, I'd like
> to offer my help to build it for the community. At least offer as an option
> ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
> just copy the full folder structure, do the softlink, and you are ready to
> play with Gentoo. If you mess up anything, just re-deploy.
>
> To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to
> get blocked on little issues that take a while to google or debug (specially
> with the long compilation times of the bootstrap and some big packages).
>
> I've posted a short issue in the Docker repo of Gentoo images about this
> (https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
> maintainers probably don't usually work with Gentoo Prefix.
>
> Thank you very much for your time, and for your open source efforts.
>
> --
> *
> *
> *Sammy Pfeiffer*
> PhD Candidate at The Magic Lab within UTS.
