Consider this a +1 for pretty much every point that Matthew made, and articulated much better than I would have. I wholly understand the goal with making ceph more approachable for people learning it, and I think that is a great cause and deserves attention. I also think Tim Serong's Linux.conf.au talk "Containers are hideously undebuggable black boxes and we never should have invented them <https://www.youtube.com/watch?v=pPZsN_urpqw>" was spot on with this problem scenario, in that debugging is a very different path with multiple layers of abstraction. He obviously details the solution(s), but I think it highlights a lot of the friction with moving to container based ceph as compared to package based ceph.
Reed > On Jun 2, 2021, at 4:36 AM, Matthew Vernon <m...@sanger.ac.uk> wrote: > > Hi, > > In the discussion after the Ceph Month talks yesterday, there was a bit of > chat about cephadm / containers / packages. IIRC, Sage observed that a common > reason in the recent user survey for not using cephadm was that it only > worked on containerised deployments. I think he then went on to say that he > hadn't heard any compelling reasons why not to use containers, and suggested > that resistance was essentially a user education question[0]. > > I'd like to suggest, briefly, that: > > * containerised deployments are more complex to manage, and this is not > simply a matter of familiarity > * reducing the complexity of systems makes admins' lives easier > * the trade-off of the pros and cons of containers vs packages is not > obvious, and will depend on deployment needs > * Ceph users will benefit from both approaches being supported into the future > > We make extensive use of containers at Sanger, particularly for scientific > workflows, and also for bundling some web apps (e.g. Grafana). We've also > looked at a number of container runtimes (Docker, singularity, charliecloud). > They do have advantages - it's easy to distribute a complex userland in a way > that will run on (almost) any target distribution; rapid "cloud" deployment; > some separation (via namespaces) of network/users/processes. > > For what I think of as a 'boring' Ceph deploy (i.e. install on a set of > dedicated hardware and then run for a long time), I'm not sure any of these > benefits are particularly relevant and/or compelling - Ceph upstream produce > Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of > a couple of different Ceph releases per Ubuntu LTS - meaning we can easily > separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages > _doesn't_ restart the daemons[1], meaning that we maintain control over > restart order during an upgrade. And while we might briefly install packages > from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, > rather than trying to run a mixed set of versions on a single cluster - and I > understand this single-version approach is best practice. > > Deployment via containers does bring complexity; some examples we've found at > Sanger (not all Ceph-related, which we run from packages): > > * you now have 2 process supervision points - dockerd and systemd > * docker updates (via distribution unattended-upgrades) have an unfortunate > habit of rudely restarting everything > * docker squats on a chunk of RFC 1918 space (and telling it not to can be a > bore), which coincides with our internal network... > * there is more friction if you need to look inside containers (particularly > if you have a lot running on a host and are trying to find out what's going > on) > * you typically need to be root to build docker containers (unlike packages) > * we already have package deployment infrastructure (which we'll need > regardless of deployment choice) > > We also currently use systemd overrides to tweak some of the Ceph units (e.g. > to do some network sanity checks before bringing up an OSD), and have some > tools to pair OSD / journal / LVM / disk device up; I think these would be > more fiddly in a containerised deployment. I'd accept that fixing these might > just be a SMOP[2] on our part. > > Now none of this is show-stopping, and I am most definitely not saying "don't > ship containers". But I think there is added complexity to your deployment > from going the containers route, and that is not simply a "learn how to use > containers" learning curve. I do think it is reasonable for an admin to want > to reduce the complexity of what they're dealing with - after all, much of my > job is trying to automate or simplify the management of complex systems! > > I can see from a software maintainer's point of view that just building one > container and shipping it everywhere is easier than building packages for a > number of different distributions (one of my other hats is a Debian > developer, and I have a bunch of machinery for doing this sort of thing). But > it would be a bit unfortunate if the general thrust of "let's make Ceph > easier to set up and manage" was somewhat derailed with "you must use > containers, even if they make your life harder". > > I'm not going to criticise anyone who decides to use a container-based > deployment (and I'm sure there are plenty of setups where it's an obvious > win), but if I were advising someone who wanted to set up and use a 'boring' > Ceph cluster for the medium term, I'd still advise on using packages. I don't > think this makes me a luddite :) > > Regards, and apologies for the wall of text, > > Matthew > > [0] I think that's a fair summary! > [1] This hasn't always been true... > [2] Simple (sic.) Matter of Programming > > > -- > The Wellcome Sanger Institute is operated by Genome Research Limited, a > charity registered in England with number 1021457 and a company registered in > England with number 2742969, whose registered office is 215 Euston Road, > London, NW1 2BE. _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
