Hello, I just wanted to say I'm very happy to see this project advancing and that you found my work useful :) I've been offline for quite a while and it has been great to find all this activity!
On Mon, Mar 18, 2019 at 11:53 PM Michael Haubenwallner <[email protected]> wrote: > Hi, > > based on Sammy's work I've set up another Azure build matrix behind a nice > url: > https://dev.azure.com/gentoo-prefix/ci-builds/_build > > This does bootstrap Gentoo Prefix (using self hosted Windows Server > build agents for the long running Cygwin builds) in all these setups: > > ---------------------+---------------+---------------+ > | Prefix | RAP | > 64bit OS | 64bit | 32bit | 64bit | 32bit | > ---------------------+-------+-------+-------+-------+ > Fedora current | yes | yes | yes | yes | > Fedora 28 | yes | yes | yes | yes | > OpenSUSE current | yes | yes | yes | yes | > Ubuntu current | - | - | yes | yes | > Ubuntu 16.04 | - | - | yes | yes | > MacOS current | yes | - | - | - | > Cygwin Server 2012R2 | yes | - | - | - | > Cygwin Server 2016 | yes | - | - | - | > Cygwin Server 2019 | yes | - | - | - | > ---------------------+-------+-------+-------+-------+ > > One Cygwin agent on Windows Server 2019 also does submit results to > http://bootstrap.prefix.bitzolder.nl/results/ > > Uploading docker images with bootstrapped /tmp/gentoo Prefix I'm happy > to leave up to Sammy's jobs still. > > Thanks! > /haubi/ > > On 11/26/18 5:02 AM, Sam Pfeiffer wrote: > > Hello, > > > > First, let me apologise if I have any wrong preexisting ideas/guesses > about Gentoo Prefix and how it is developed. Secondly, sorry if this email > is a bit too long. > > > > I'll present myself, explain my use-case and my troubles and tricks, and > then ask for feedback. > > > > Hello, I'm Sammy Pfeiffer. I am a PhD student at University of > Technology Sydney. I'm a (software) robotics engineer pursuing my PhD in a > lab with a bunch of different robots and machine learning setups. I'm using > Gentoo Prefix for deploying a big bunch of software into robots where the > OS is old and frozen (and also to deploy in computation farms > experiments... once again with no option to touch the OS). (The bunch of > software is ROS, the 'Robotics Operating System' which has a ton of > dependencies, and there is an existing overlay which I'm helping on > maintaining and improving (https://github.com/ros/ros-overlay)). > > > > I'd like to share the trick I found to overcome a few annoying bits of > my platforms: > > * I have no root access in the machines I need to deploy. > > * The machines don't have Docker installed (too old kernel anyways). > > * I have 32 bit and 64 bit machines (the OS running in them is). > > * I have a different username (and home folder) on each machine. > > * Each machine has a different disk(s)/partition(s) structure (main > issue here is that I may need to use a different disk to store my data, > cause of size constrains). > > > > The trick to be able to use Gentoo Prefix with all these constraints > bootstrapping it only once is to set the EPREFIX to /tmp/gentoo. > > Then you can deploy the full bootstrapped system in any folder/disk and > just do a softlink to /tmp/gentoo. This works nicely (I was scared of the > softlink breaking stuff somewhere, but it was alright). > > The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run > in any 32b or 64b box (it's not ideal but simplifies my deployment > currently). > > > > Also, having all these machines use the same Gentoo Prefix in the same > place, with the same 32bit compilation, I can setup a binary package > server, which all the deployments can point to and just get all the > packages skipping the long compilation times (specially in very old and low > powered machines). > > > > The final trick I'm experimenting with is to use a set of Docker images > (and soon in a continuous integration environment) to bootstrap all the > system. With that I can save snapshot of successfully built systems & > packages to serve as an easy deployment (and easy installation of extra > packages thru the binary package server). > > > > Given my particular annoyance of the 32bit system (and noting that my > hosts are 64bits, as is the standard nowadays) I found that I can use > either: > > * https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image > > * https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit > Gentoo Docker image > > > > And then execute the build step of Docker with: > > > > setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image . > > > > Which will trick any program trying to do uname -m to assume 32bit > machine. (Previously I used the variable CHOST=i686-pc-linux-gnu for > bootstrapping, but I found some problems, which didn't appear with this > method). > > > > Once I have all this setup working nicely, I'd like to trigger rebuilds > every X time, and on changes on main players (or all dependencies actually) > of my setup, like the bootstrapping of Gentoo Prefix (Ideally, on every > change of a part of the system just trigger a rebuild from that point on, > Docker layers make this possible). > > > > With Azure announcing unlimited minutes on CI/CD for open source > projects: > > > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/ > > > > Even bootstrapping Gentoo prefix, with pieces of software like gcc > taking very long to compile, is possible. > > > > The point is: I have been trying to build Gentoo Prefix during the last > days after a few months of break since the last time I touched the system. > And it's failing. I haven't managed yet to bootstrap it completely. I feel > there is no CI/CD setup to catch these issues and be able to offer a > working version of Gentoo Prefix at any time. > > > > I was going to build it for myself (cause I need it), but instead, I'd > like to offer my help to build it for the community. At least offer as an > option ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a > Docker image, just copy the full folder structure, do the softlink, and you > are ready to play with Gentoo. If you mess up anything, just re-deploy. > > > > To do this I'd need a bit of help as I'm quite new to Gentoo and I tend > to get blocked on little issues that take a while to google or debug > (specially with the long compilation times of the bootstrap and some big > packages). > > > > I've posted a short issue in the Docker repo of Gentoo images about this > (https://github.com/gentoo/gentoo-docker-images/issues/62) but I think > the maintainers probably don't usually work with Gentoo Prefix. > > > > Thank you very much for your time, and for your open source efforts. > > > > -- > > * > > * > > *Sammy Pfeiffer* > > PhD Candidate at The Magic Lab within UTS. > > -- *Sammy Pfeiffer* PhD Candidate at The Magic Lab within UTS.
