Re: Current best practices for system configuration management?
On 6/22/24 11:33, Dmitrii Odintcov wrote: Hi all, Sorry to resurrect an old-ish thread, but I am facing the exact same task, minus the know-how. Basically I am looking to pre-configure a number of Debian setups - let's say, "server", "laptop" and "PC" - that would contain sets of packages to install (or uninstall), configuration files (including but not limited to /etc/), and possibly arbitrary scripts to execute upon installation (e.g. to make more precise edits to configs). I would like to store these in some central, git-controlled location where I could pull them from to my target machines. This kind of setup could be rather low-level, such as configuring APT sources and preferences, installing and configuring systemd-networkd, etc. Ideally, I would also be able to incorporate these into a Debian installer so I could get a new machine "up and running" with my defaults in one go. It is worth adding that I am not looking to build an "infrastructure", and no setup *between* these machines (networking, file sharing, etc.) is necessary. Generally, I would rather avoid complicated tools like Ansible and those with large dependencies such as interpreted languages (except Perl). So far, equivs and config-package-dev appear most relevant, but the former is somewhat lacking in documentation (or I am in documentation-finding ability), and the latter seems to be focused on config editing/deployment rather than package installation. Would be grateful for some advice! I think the "best" answer depends upon the scale of your installation. I have a SOHO network with a dozen or so Debian, Windows, macOS, and iOS clients, a FreeBSD/ZFS CVS, SSH, and Samba server, and a FreeBSD/ZFS backup server. For system administration, including configuration management, I have gone down the do-it-yourself (DIY) path using lowest-common-denominator FOSS command line tools. After the network, ssh(1), and rsync(1), the most valuable tool for system administration (including configuration management) has been a version control system. I prefer CVS over Git because CVS provides monotonically-increasing MAJOR.MINOR version numbers via keywords (e.g. $Revision$) that can be included inside managed plaintext files. I create a CVS project for everything I want to manage. For OS's, I check in a sysadmin log file with my notes and console sessions, a list of packages installed, various reports that I have run, and any system configuration files that I have modified, added, or deleted. After version control, the next most valuable tool has been scripting. Over the years, I have written numerous scripts to automate repetitive chores. When the needs are simple, I write Bourne shell scripts. When I want more power, I upgrade to Perl. All of the scripts are checked in to CVS. I expect Ansible, Puppet, etc., would work at my scale, but are designed for large installations. David
Re: Current best practices for system configuration management?
Hi all, Sorry to resurrect an old-ish thread, but I am facing the exact same task, minus the know-how. Basically I am looking to pre-configure a number of Debian setups - let's say, "server", "laptop" and "PC" - that would contain sets of packages to install (or uninstall), configuration files (including but not limited to /etc/), and possibly arbitrary scripts to execute upon installation (e.g. to make more precise edits to configs). I would like to store these in some central, git-controlled location where I could pull them from to my target machines. This kind of setup could be rather low-level, such as configuring APT sources and preferences, installing and configuring systemd-networkd, etc. Ideally, I would also be able to incorporate these into a Debian installer so I could get a new machine "up and running" with my defaults in one go. It is worth adding that I am not looking to build an "infrastructure", and no setup *between* these machines (networking, file sharing, etc.) is necessary. Generally, I would rather avoid complicated tools like Ansible and those with large dependencies such as interpreted languages (except Perl). So far, equivs and config-package-dev appear most relevant, but the former is somewhat lacking in documentation (or I am in documentation-finding ability), and the latter seems to be focused on config editing/deployment rather than package installation. Would be grateful for some advice!
Re: Current best practices for system configuration management?
On Sat, Apr 20, 2024 at 4:40 PM Mike Castle wrote: > Thanks for all of the commentary so far. > > Once I get something working, I will *try* to remember to follow up > here with what I've managed to cobble together. I have done quite a bit of research and experimentation and finally settled on a solution that seems like it will work for me: Plain old debian/* control files along with config-package-dev. Effectively, I've abandoned `equivs` and now just using plain old `debhelper` with a small wrapper script. One thing I've learned along the way is that debian/control files can build multiple .deb files. But the "control" files that `equivs` uses are not really the same as regular control files, so need one per package. Since I created a hierarchy of packages, I needed several configs for equivs. Now, I can use just one, and it is MUCH faster (about 8 seconds total). I'm still working on replacing my old equivs stuff, but, I do have some things I'm installing now. Essentially my setup looks like this: debian/control: Maintainer: Mike Castle Source: mrc Build-Depends: debhelper-compat (= 12), config-package-dev Rules-Requires-Root: no Standards-Version: 4.5.1 Section: metapackages Priority: optional Package: mrc-base Architecture: all Depends: ${misc:Depends}, bc, cron, ... zip, Package: mrc-mozilla Architecture: all Description: MRC's Mozilla apt configuration Lots of files under /etc/apt. Depends: ${misc:Depends}, Package: mrc-desktop Architecture: all Description: MRC's desktop installation Graphical stuff. Depends: ${misc:Depends}, mrc-base, mrc-games, mrc-python, mrc-mozilla, ... more meta packages ... Package: mrc-mars Architecture: all Description: MRC's host mars Host specific deps. Provides: ${diverted-files} Conflicts: ${diverted-files} Depends: ${misc:Depends}, mrc-desktop, mrc-development, mrc-virtual-machines, ... most host packages ... A few other debian control files: $ find debian/ -type f debian/source/format debian/mrc-mars.install debian/mrc-mars.displace debian/mrc-mozilla.install debian/copyright debian/changelog debian/rules debian/control $ cat debian/rules #!/usr/bin/make -f %: dh $@ --with=config-package The files that match .* are the magic. For example: $ cat debian/mrc-mozilla.install files/mozilla/* / $ find files/mozilla -type f files/mozilla/usr/share/lintian/overrides/mrc-mozilla files/mozilla/etc/apt/sources.list.d/mozilla.list files/mozilla/etc/apt/preferences.d/mozilla files/mozilla/etc/apt/keyrings/packages.mozilla.org.asc The etc/apt files come from https://support.mozilla.org/en-US/kb/install-firefox-linux#w_install-firefox-deb-package-for-debian-based-distributions and the lintian overrides is: $ cat files/mozilla/usr/share/lintian/overrides/mrc-mozilla mrc-mozilla: package-installs-apt-sources mrc-mozilla: package-installs-apt-preferences For the host packages, it is slightly more complicated, but not much once I figured it out. Again, a basic "copy in everything under this directory" install file: $ cat debian/mrc-mars.install files/mars/* / which is currently this: $ find files/mars -type f files/mars/etc/hostname.mrc Then the interesting bit: $ cat debian/mrc-mars.displace /etc/hostname.mrc This "dispace" file is something that the `config-package` addon takes care of. It sets up backing up the existing file name (/etc/hostname) and symlinks in my version: $ ls -l /etc/hostname* lrwxrwxrwx 1 root root 12 May 27 08:57 /etc/hostname -> hostname.mrc -rw-r--r-- 1 root root 5 May 27 06:52 /etc/hostname.mrc -rw-r--r-- 1 root root 6 May 27 07:37 /etc/hostname.mrc-orig If I purge mrc-mars, then /etc/hostname.mrc-orig is moved back to /etc/hostname. (The way config-package-dev works is, it takes the first portion of the package names, in this case "mrc", to identify what file names to use.) To drive it all, I currently use a simple wrapper script: $ cat doit.sh #!/bin/bash set -e work_dir=$(mktemp -d) find -depth | cpio -pdm ${work_dir}/work cd ${work_dir}/work debuild --no-conf --no-sign --lintian-opts --info OUTPUT=/srv/deb/packages rm -rf $OUTPUT mkdir -p $OUTPUT cp ${work_dir}/*.deb $OUTPUT cd $OUTPUT dpkg-scanpackages . > Packages echo "work_dir was ${work_dir}" Not the fanciest, but "works for me" (so far). Essentially it builds things in /tmp because the debian packaging tool chain always drops the files into the parent directory and leaves things that *I* currently do not care about littering my disk. Also, it drops intermediate files into the debian/* directory, making it more difficult to know what needs to go under SCM. With this approach, "git status" will be easier to read. The only drawback so far, over my previous approach of one `equivs` config per package, is that now, all built debs share the same version number. With the individual approach, I could add one package to a metapackage, and when I do "apt update && apt upgrade", only the single package (and its new deps) are processed. Now, every
Re: Current best practices for system configuration management?
Mike Castle writes: Hah! https://lists.debian.org/debian-user/2013/08/msg00042.html Yes, that was me > 10a ago. Transitioning from these scripts to ant allowed came with a few improvements: * I switched all package building to `debuild` in favor of using more low-level tools for `raw` packages before. * I was able to establish some sort of “buld upon new commit” poor man's local CI -- I experimented with some fully-fledged systems like Concurse CI and GitLab, too, but they were all too much for my local-first activities * When resources were required from external locations (e.g. source code downloads), that set of scripts did not provide for any automatism in this regard. One of my continuous goals is to split between “my settings” and some re-downloadable files. It was only achieved partially back then and the ant-approach takes it much further :) Still, in many ways the ant does not do much more than the scripts linked at the message above. If you are about to create your own scripts (which is probably a better idea than trying to use my scripts anyways :) ) the shellscripts may be easier to understand. HTH Linux-Fan öö pgpEtXWCbBgkx.pgp Description: PGP signature
Mesh VPN on Debian (Was Re: Current best practices for system configuration management?)
Hi, On Sat, Apr 20, 2024 at 04:40:24PM -0700, Mike Castle wrote: > Like Alex, one of my physical machines is a laptop that is not always > on the home network. Though I'm usually connected to *something*. > I'm still debating whether to bother with a VPN or trying something > like a tailnet. For mesh VPN I really like Yggdrasil (packaged in Debian, but widely available). It does quite a lot of the things that people use Tailscale for, but has the advantages of: - Completely FOSS - No need to contact a central authority - your nodes all self-organise - Thus no limit on how many nodes you can have for free (though Tailscale's limit is very generous) Like Tailscale it will detect other instances of itself on your LAN so local traffic remains local (avoid a VPN hairpin) while you still use the same Yggdrasil IP addresses to talk to things. Downsides compared to Tailscale are things like: - Not as polished a product so no hand-holding; you need to read the docs - Not available on as many platforms. It is a single static Go binary so it's not hard to deploy if you can compile it, but I don't know what the story is on things like mobile platforms, whereas there's Tailscale apps for everything. - I don't have personal experience but possibly it's more energy intensive than Tailscale which would matter a lot on mobile devices There is a good introduction and comparison with some other solutions here: https://www.complete.org/easily-accessing-all-your-stuff-with-a-zero-trust-mesh-vpn/ I still wouldn't want to automated a config push/pull to a laptop over a mesh VPN I think, but others have mentioned that you can do Ansible in a pull mode. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Current best practices for system configuration management?
On Thu, 18 Apr 2024, Mike Castle wrote: Now, I would like to expand that into also setting up various config files that I currently do manually, for example, the `/etc/apt/*` configs I need to make the above work. For a single set of files, manual isn't bad, but as I want to get into setting up LDAP, autofs, and so on, it is time to explore solutions. I only have four systems at the moment (two physical and two virtual), so I don't think I need something too fancy. I do this - but it needs a bit of thought as to how you want to do it. I'm assuming you don't care about debian policy ... I do one of four things in packages depending on what I want to achieve and what the package I'm configuring supports. 1.(easiest) - just drop a config file into a conf.d directory. I use this for maintaining sources.list. Back in the buster days I was using openssh from backports that supported this as maintaining the config is so much easier than any of the other ways and so much easier to fix bugs in your packages - a failing postinst script can be a pain to resolve. apt-mirror@aptmirror19:~ (none)$ apt-cache search bookworm-sources bookworm-sources - Meta-package to pull in list of apt-sources for debian (I generate these packages automatically from what is available on deb.debian.org.) FWIW, I also have a dev repo and a fast repo - the dev repo is a clone where I can test out substantial changes and the fast repo only has a couple of packages but can rebuild "the world" very quickly when I'm testing. So I have a number of extra levels of indirection to support this: $ apt-cache depends bookworm-sources bookworm-sources Depends: bookworm-local-sources That is the normal package that I would install. apt-mirror@aptmirror19:/mnt/mirror/local/main/o (master)$ apt-cache depends bookworm-dev-sources bookworm-dev-sources Depends: bookworm-dev-main-sources That lets me use the dev repo - and thanks to pinning, the packages in it will replace (even downgrade) packages from local. Ditto for my fast repo. 2. (next easiest) - add a file in /usr/share/ and then overwrite the file in /etc by copying in postinst. 3. edit the config file without diverting it using a sed script. 4. divert the config and edit/replace it in the package. 3. and 4. in particular can make upgrades more difficult if/when you find bugs in the way you're handling diversions and uninstalls.
Re: Current best practices for system configuration management?
Hah! https://lists.debian.org/debian-user/2013/08/msg00042.html
Re: Current best practices for system configuration management?
On 21/04/2024 08:40, Mike Castle wrote: One thing Linux-Fan mentioned was `config-package-dev`. In my OP, I commented about ``slightly old to really old tools'', and that was one I was thinking of. It looks like it hasn't been touched in seven years, and I wasn't sure if it still worked. But that drive by comments lends some hope. Using it would help address Alex's concern about modifying existing config files. That debhelper extension is designed precisely for that situation. But, its age is pretty much what inspired me to start this thread. I've been using config-package-dev for some years now - to add some customized configs to existing installed packages - and it still seems to be working perfectly. It's just a wrapper on top of dpkg diverts so there's not all that much to go wrong. -- John
Re: Current best practices for system configuration management?
Thanks for all the suggestions so far. Like Alex, one of my physical machines is a laptop that is not always on the home network. Though I'm usually connected to *something*. I'm still debating whether to bother with a VPN or trying something like a tailnet. Heck, before I adopted Debian and ran my own Linux-from-Scratch-started-before-LFS-or-Debian-existed "distribution", I used to run $HOME over NFS over WiFi. Sharing configs that way would be acceptable for me. I also used to keep my configs in CVS and pushed them using rsync-over-ssh-as-root, hence "current best practices". :-> Linux-Fan's MDVL stuff looks interesting. Everything can always be solved by one more layer of redirection. :-> I just have to suss out the Java and Ant stuff to understand the basics. Personally, I'm a big fan of NIH-Syndrome (otherwise I wouldn't have run a LFS type system for ~15 years). So it is likely I would reinvent what is happening there. One thing Linux-Fan mentioned was `config-package-dev`. In my OP, I commented about ``slightly old to really old tools'', and that was one I was thinking of. It looks like it hasn't been touched in seven years, and I wasn't sure if it still worked. But that drive by comments lends some hope. Using it would help address Alex's concern about modifying existing config files. That debhelper extension is designed precisely for that situation. But, its age is pretty much what inspired me to start this thread. Interestingly enough, I sent the OP just before I went to a BayLISA talk about Apple's PKL. Since I tend to only use whatever comes out of the Debian repos (my only exceptions being Firefox and an emacs package the maintainers won't fix for bookworm), I'm unlikely to do more than look at it, But, between it and MDVL, I think I'm definitely going to try to make sure I don't configure myself into a corner. :-> Thanks for all of the commentary so far. Once I get something working, I will *try* to remember to follow up here with what I've managed to cobble together. Cheers! mrc PS: Actually, I used to share $HOME (and /usr) over PLIP, so, it is probably obvious that FS speed is not always a concern for me.
Re: Current best practices for system configuration management?
Hi, On Thu, Apr 18, 2024 at 05:05:14PM -0700, Mike Castle wrote: > Anyway, suggestions based upon actually experience would be appreciated. As others have mentioned, Ansible can be a good choice for as little as one machine as long as you don't object to installing Python and a bunch of Python apps. Probably best to do it in a Python virtualenv rather than try to use OS packages as it does still change quite often, but I've found backwards compatibility to be decent. I used Puppet for quite a few years. It was a nicer, more expressive and powerful language. It's also a lot faster than Ansible when you have tens of hosts. It probably does not make sense for only a couple of hosts though. I stopped using it because the rate of change (requiring changes to its server and client on all hosts) was too much for me. These days I think the original developers consider it "done" and have moved on, selling the lot to Perforce who I suspect will just coast along milking the community for what it can. I would not consider Puppet for a new setup at this time, mainly for that reason. Chef, never used it but did consider it when wanting to move away from Puppet. Again probably not make sense for small number of hosts. Likely has more of a future than Puppet considering Cinc. We all hear great things about Nix, but I want something that manages Debian packages, not replaces Debian packages. Ansible is still viable for me but I fully expect Red Hat to ruin it one day in search of profit. Which is their right. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Current best practices for system configuration management?
Mike Castle writes: For a while now, I've been using `equivs-build` for maintaining a hierarchy of metapackages to control what is installed on my various machines. Generally, I can do `apt install mrc-$(hostname -s)` and I'm golden. Now, I would like to expand that into also setting up various config files that I currently do manually, for example, the `/etc/apt/*` configs I need to make the above work. For a single set of files, [...] My first thought was to simply add a `Files:` section to *.control files I use for my metapackages. After all, for configs going into *.d directories, they are usually easy to just drop in and remove, no editing in place required. But, that is when I discovered that all files under `/etc` are treated specially. [...] Hello, I can confirm from experience that Ansible can indeed scale down to as little as the one local machine that it is running from. It has a learning curve and at least to me it always felt a little clumsy to learn a YAML based scripting language for this purpose, but its a solid choice. Continuing the package-based approach is what I do because once some wrapper around the `debuild` commands was established, it became acceptably easy to use. I even maintain my “dotfiles” (not under $HOME but under /etc, but to a similar effect) this way: https://masysma.net/32/conf-cli.xhtml. With `config-package-dev` there are some tricks to even allow changing (config) files supplied by other packages. The disadvantage with the package-based approach is that it is heavily distribution-specific and also if you mess anything up, a core component of the OS (package management) can become broken - I luckily never broke it to the extent that recovery was impossible, but in the beginning ran a dedicated test VM to validate all package changes prior to installing them on my main system I have also heard good things about Nix and if I had to start again from scratch today, I'd probably invest time into learning that technology. Right now I am sufficiently satisfied with the package-based approach to not look into it yet. HTH Linux-Fan öö pgplLIRpEGPqu.pgp Description: PGP signature
Re: Current best practices for system configuration management?
Hi, > > and so on, it is time to explore solutions. I only have four systems > > at the moment (two physical and two virtual), so I don't think I need > > something too fancy. I am in the same situation with an extra constraint: some are laptops and not always connected. > > My first thought was to simply add a `Files:` section to *.control > > files I use for my metapackages. After all, for configs going into > > *.d directories, they are usually easy to just drop in and remove, no > > editing in place required. But, that is when I discovered that all > > files under `/etc` are treated specially. The limitation of this is that you cannot modify existing configuration files, which is required sometimes. > > Anyway, suggestions based upon actually experience would be appreciated. > > The easy end of single-machine is etckeeper, which just checks > your /etc (and whatever else you specify) into a local git. The > high end of single machine is Nix, which has a complete language > designed to capture the complete configuration of a system (and > has spawned NixOS, a complete distribution). > > The easy end of multi-machine systems is cdist and itamae. You might be > quite happy with those, and itamae is reputedly very Chef-like. I can also mention ansible which is ubiquitous nowadays and is relevant even for a single machine. I've worked around my laptop deployment requirement with a wrapper script around ansible-pull and a systemd timer to regularly pull the conf from a git repository. It works well but the complete configuration is known to all machines. The Nix mention is highly relevant, but I did not get the chance to play with it yet. The big advantage over ansible and probably many others is that if you remove a package installation from your configuration, it will get removed from the host upon configuration deployment. Whereas with ansible, you must add a explicit uninstallation rule. This is fine for cloud host deployment where you always start from scratch, but for physical machines and user laptops, Nix enforces consistency (but then you need to learn something that is not Debian...). Cheers, Alex
Re: Current best practices for system configuration management?
Mike Castle wrote: > and so on, it is time to explore solutions. I only have four systems > at the moment (two physical and two virtual), so I don't think I need > something too fancy. > > My first thought was to simply add a `Files:` section to *.control > files I use for my metapackages. After all, for configs going into > *.d directories, they are usually easy to just drop in and remove, no > editing in place required. But, that is when I discovered that all > files under `/etc` are treated specially. > > I've found a lot of documentation out there, but, of course, much of > it is out of date. https://wiki.debian.org/ConfigPackages , for > example, seems to recommend slightly old to really old tools. Tools > like `ansible`, `puppet` and so on seem, at first blush, aimed at > larger installations than mine. But maybe other's experience with > them will show they scale down fine? > > Anyway, suggestions based upon actually experience would be appreciated. Chef or Puppet are good, but probably not worth the bother for 4-5 systems. Definitely worthwhile for 10+ systems, and scalable to thousands. https://cinc.sh is producing a fully open release of Chef (with the new name, Cinc is not chef). I use Chef at work. The easy end of single-machine is etckeeper, which just checks your /etc (and whatever else you specify) into a local git. The high end of single machine is Nix, which has a complete language designed to capture the complete configuration of a system (and has spawned NixOS, a complete distribution). The easy end of multi-machine systems is cdist and itamae. You might be quite happy with those, and itamae is reputedly very Chef-like. -dsr-
Current best practices for system configuration management?
For a while now, I've been using `equivs-build` for maintaining a hierarchy of metapackages to control what is installed on my various machines. Generally, I can do `apt install mrc-$(hostname -s)` and I'm golden. Now, I would like to expand that into also setting up various config files that I currently do manually, for example, the `/etc/apt/*` configs I need to make the above work. For a single set of files, manual isn't bad, but as I want to get into setting up LDAP, autofs, and so on, it is time to explore solutions. I only have four systems at the moment (two physical and two virtual), so I don't think I need something too fancy. My first thought was to simply add a `Files:` section to *.control files I use for my metapackages. After all, for configs going into *.d directories, they are usually easy to just drop in and remove, no editing in place required. But, that is when I discovered that all files under `/etc` are treated specially. I've found a lot of documentation out there, but, of course, much of it is out of date. https://wiki.debian.org/ConfigPackages , for example, seems to recommend slightly old to really old tools. Tools like `ansible`, `puppet` and so on seem, at first blush, aimed at larger installations than mine. But maybe other's experience with them will show they scale down fine? Anyway, suggestions based upon actually experience would be appreciated. Cheers, mrc