Re: [gentoo-dev] Re: Managing updates on many identical Gentoo systems
On 01/20/2018 07:34 AM, Anthony G. Basile wrote: > On 1/19/18 10:03 AM, Anthony G. Basile wrote: >> >> Zac pretty much nailed the requirements in bug #644990. You should not >> need the portage tree at all, neither locally nor via any network >> filesystem. He mentions there that it is currently possible via "a >> dummy profile", but I'm not sure what he means by that yet or how to set >> one up. I'll read his bug #640318 and try to figure it out. >> >> Thanks guys, I'm glad people at least recognized the usefulness of such >> a possibility. >> > > Okay, I have a workable solution to my question. I was able to get > binhost working with a portage tree containing ONLY /profiles and > /eclass. That's 12MB and 2.8MB in size, respectively, and I can > probably dump a bunch of the unused profile directories slimming that > down. With just those two directories in PORTDIR, emerge -K pulls down > the update packages from BINHOST and installs them. > > @zac any comments about this approach? Is it likely to break? It's desirable to rely exclusively on the BINHOST as a single source of truth, since otherwise you have to keep multiple data sources in consistent states. You should not need the eclasses, since portage uses the eclass code from environment.bz2 that is embedded in each binary package. Using /profiles can cause problems because things like package.mask and package.keywords have to be consistent with the BINHOST. For the above reasons, I use a dummy profile. I also sync /profiles/updates so that emerge can apply package moves, but I intend to eliminate that as part of bug #644990 since keeping /profiles/updates consistent with the binhost is not practical. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Managing updates on many identical Gentoo systems
On 1/19/18 10:03 AM, Anthony G. Basile wrote: > On 1/19/18 9:45 AM, Alec Warner wrote: >> On Thu, Jan 18, 2018 at 5:13 PM, Bill Kenworthy wrote: >> >>> On 18/01/18 23:36, Duncan wrote: Anthony G. Basile posted on Thu, 18 Jan 2018 06:46:53 -0500 as excerpted: > I'm trying to design an update system for many identical Gentoo systems. > Using a binhost is obvious, but there are still problems with this > approach. > >>> >>> I'd suggest go for a semi diskless OS - boot them from one central image >>> with an individual overlay filesystem with local customisations. NFS >>> mount the common directories. >>> >>> you just have a one central host to build for and don't need to worry >>> about portage everywhere. >>> >>> Worked ok with a small number of mythtv frontends. >>> >> >> It doesn't work if you have a WAN; NFS needs low latencies between the NFS >> server and the client or you will have a bad time. >> >> > > Zac pretty much nailed the requirements in bug #644990. You should not > need the portage tree at all, neither locally nor via any network > filesystem. He mentions there that it is currently possible via "a > dummy profile", but I'm not sure what he means by that yet or how to set > one up. I'll read his bug #640318 and try to figure it out. > > Thanks guys, I'm glad people at least recognized the usefulness of such > a possibility. > Okay, I have a workable solution to my question. I was able to get binhost working with a portage tree containing ONLY /profiles and /eclass. That's 12MB and 2.8MB in size, respectively, and I can probably dump a bunch of the unused profile directories slimming that down. With just those two directories in PORTDIR, emerge -K pulls down the update packages from BINHOST and installs them. @zac any comments about this approach? Is it likely to break? -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: Managing updates on many identical Gentoo systems
On 01/19/2018 07:03 AM, Anthony G. Basile wrote:
> On 1/19/18 9:45 AM, Alec Warner wrote:
>> On Thu, Jan 18, 2018 at 5:13 PM, Bill Kenworthy wrote:
>>
>>> On 18/01/18 23:36, Duncan wrote:
Anthony G. Basile posted on Thu, 18 Jan 2018 06:46:53 -0500 as excerpted:
> I'm trying to design an update system for many identical Gentoo systems.
> Using a binhost is obvious, but there are still problems with this
> approach.
>
>>>
>>> I'd suggest go for a semi diskless OS - boot them from one central image
>>> with an individual overlay filesystem with local customisations. NFS
>>> mount the common directories.
>>>
>>> you just have a one central host to build for and don't need to worry
>>> about portage everywhere.
>>>
>>> Worked ok with a small number of mythtv frontends.
>>>
>>
>> It doesn't work if you have a WAN; NFS needs low latencies between the NFS
>> server and the client or you will have a bad time.
>>
>>
>
> Zac pretty much nailed the requirements in bug #644990. You should not
> need the portage tree at all, neither locally nor via any network
> filesystem. He mentions there that it is currently possible via "a
> dummy profile", but I'm not sure what he means by that yet or how to set
> one up.
The dummy profile consists of making a /etc/portage/make.profile/
directory that is nearly empty, containing only a make.defaults file
with minimal settings like these:
ARCH="amd64"
ACCEPT_KEYWORDS="**"
ACCEPT_LICENSE="*"
You also need a big IUSE_IMPLICIT setting in there if you don't have
this patch that's in portage-2.3.19:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=2382082dcfce5e6c2be7c4dd6aed7c32e1d20616
With the dummy profile, you can install all of the packages available in
the binhost. Currently, there's not a really good way to tell emerge to
install all packages from the binhost, but you can parse them all from
${PORTAGE_BINHOST}/Packages and write the list to /var/lib/portage/world
or something similar. Then `emerge -vuGD @world`.
I've described my currently planned sequence of steps here:
https://bugs.gentoo.org/644990#c3
If you try the dummy profile approach, then if there are any in-place
package moves in the binhost then you may have problems with file
collisions, but I've described how I plan to handle that in the above
bug comment.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
Re: [gentoo-dev] Re: Managing updates on many identical Gentoo systems
On 1/19/18 9:45 AM, Alec Warner wrote: > On Thu, Jan 18, 2018 at 5:13 PM, Bill Kenworthy wrote: > >> On 18/01/18 23:36, Duncan wrote: >>> Anthony G. Basile posted on Thu, 18 Jan 2018 06:46:53 -0500 as excerpted: >>> I'm trying to design an update system for many identical Gentoo systems. Using a binhost is obvious, but there are still problems with this approach. >> >> I'd suggest go for a semi diskless OS - boot them from one central image >> with an individual overlay filesystem with local customisations. NFS >> mount the common directories. >> >> you just have a one central host to build for and don't need to worry >> about portage everywhere. >> >> Worked ok with a small number of mythtv frontends. >> > > It doesn't work if you have a WAN; NFS needs low latencies between the NFS > server and the client or you will have a bad time. > > Zac pretty much nailed the requirements in bug #644990. You should not need the portage tree at all, neither locally nor via any network filesystem. He mentions there that it is currently possible via "a dummy profile", but I'm not sure what he means by that yet or how to set one up. I'll read his bug #640318 and try to figure it out. Thanks guys, I'm glad people at least recognized the usefulness of such a possibility. -- Anthony G. Basile, Ph.D. Gentoo Linux Developer [Hardened] E-Mail: bluen...@gentoo.org GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA GnuPG ID : F52D4BBA
Re: [gentoo-dev] Re: Managing updates on many identical Gentoo systems
On Thu, Jan 18, 2018 at 5:13 PM, Bill Kenworthy wrote: > On 18/01/18 23:36, Duncan wrote: > > Anthony G. Basile posted on Thu, 18 Jan 2018 06:46:53 -0500 as excerpted: > > > >> I'm trying to design an update system for many identical Gentoo systems. > >> Using a binhost is obvious, but there are still problems with this > >> approach. > >> > > I'd suggest go for a semi diskless OS - boot them from one central image > with an individual overlay filesystem with local customisations. NFS > mount the common directories. > > you just have a one central host to build for and don't need to worry > about portage everywhere. > > Worked ok with a small number of mythtv frontends. > It doesn't work if you have a WAN; NFS needs low latencies between the NFS server and the client or you will have a bad time. > > BillK > >
Re: [gentoo-dev] Re: Managing updates on many identical Gentoo systems
On 18/01/18 23:36, Duncan wrote: > Anthony G. Basile posted on Thu, 18 Jan 2018 06:46:53 -0500 as excerpted: > >> I'm trying to design an update system for many identical Gentoo systems. >> Using a binhost is obvious, but there are still problems with this >> approach. >> I'd suggest go for a semi diskless OS - boot them from one central image with an individual overlay filesystem with local customisations. NFS mount the common directories. you just have a one central host to build for and don't need to worry about portage everywhere. Worked ok with a small number of mythtv frontends. BillK
[gentoo-dev] Re: Managing updates on many identical Gentoo systems
Anthony G. Basile posted on Thu, 18 Jan 2018 06:46:53 -0500 as excerpted: > I'm trying to design an update system for many identical Gentoo systems. > Using a binhost is obvious, but there are still problems with this > approach. > > Unless there's some magic I don't know about (and this is why I'm > sending this email) each machine still needs to have the portage tree > installed locally (1.5 GB) or somehow mounted by a network filesystem > (which is not practical if the machines are not on a local network). > Furthermore, each machine would have to run emerge locally to do the > calculation of what packages need updating. > > This procedure is redundant because each machine is housing the same > data and doing the same dependence-tree calculation. I had a gen-1.5 32-bit netbook for a number of years, that I updated using rsync from a 32-bit chroot on my main machine, no portage tree on the target netbook, tho I didn't worry about separating out build deps from run deps. That was a single machine config, but it should be even easier if you're running nearly identical machines and thus don't need the separate chroot build image. If you have temporary networking you can rsync directly machine to machine, as I did after I was fully setup, but at first I was sneaker- netting it, rsyncing to a thumb drive from the build machine, that I would then plug into the target and rsync thumb drive to target. The thumb drive was bootable, and I used it to do the first gentoo boots on the target as well, testing my config and updating as necessary as I went. When I got everything I initially wanted booting from the thumb drive, I booted the thumb drive, wiped the initial Pingus Linux on the netbook and setup the partitioning, etc, then rsynced selected bits into the appropriate place on the target. For multiple nearly identical machines you can exclude the non-identical bits from the primary rsync image, keeping the specific bits in individual images synced on top of the primary. Of course you can sync in reverse as well to keep the non-identical bits updated, giving you a nice backup of each one as well. =:^) Alternatives would include simply creating the thumb drive once and then cloning it enough times to give every machine a bootable thumb drive copy (using symlinking and/or mounts to handle the non-identical stuff, so simply toggling a symlink lets you switch machine layouts), or if the machines have enough memory, setting up a single thumb drive to boot and put everything in a tmpfs for the machine to run from, so you can use the same thumb drive to boot them all, effectively the sneakernet version of net-boot. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
