Re: Init systems and docker
On Fri, Oct 11 2019, Scott Kitterman wrote: > I have been told by docker users (I'm not one) that systemd as provided on > Debian can't be used in docker. I have no idea if that's true or not. I try > really hard to know as little about init systems as possible and trust our > maintainers who work on such things. That is a popular misconception. Actually it is easier to make systemd work well under Docker than sysvinit, and it doesn't require --privileged. I have a whole series of Debian Docker images that work that way; see https://github.com/jgoerzen/docker-debian-base systemd nowadays even has explicit code to help with this. I made sysvint work in the stretch and jessie days, but it was a real pain and not at all friendly. Have a look at my debian-base-minimal code for how to make it work. - John
Re: Init systems and docker
On Fri, Oct 11, 2019 at 06:49:37PM -0400, Scott Kitterman wrote: > I have been told by docker users (I'm not one) that systemd as provided on > Debian can't be used in docker. I have no idea if that's true or not. I try > really hard to know as little about init systems as possible and trust our > maintainers who work on such things. Hey Scott, this can be a bit of an opinionated topic and the following may not directly apply to your issue at hand, but here's my two cents from prior experience. To keep the container small and to make sure you do only one thing and one thing well in the context of the container, what I would do is to keep the OS as lightweight as possible, and only start the process needed when the container starts up using Docker's entrypoint. It can invoke a helper script to assist with starting up the application. In order to get some supervisory capability on the process I would use use supervisord which I could trigger directly from the entrypoint as this is lot easier to manage and you don't need to use the --privilege parameter, not to mention not having to deal with init/systemd related complexity (IMHO). See this for an example: https://advancedweb.hu/2018/07/03/supervisor_docker/ Some could argue that even the use of supervisord is an anti-pattern since orchestration systems provide that capability, in which case it becomes even more simpler and you can just spin up the process directly through a helper, init-like script and let the orchestrator supervise the container through health and readiness hooks that you expose in the application. All logs go to standard out so that it can be aggregated and logged in an orchestrated environment, and so fits in with the paradigm where you don't keep the logs in the container. As mentioned by others on this trail, I would also suggest against treating the container like a traditional VM and instead use the new primitives and best practices available to compose the solution, and err on the side of simplicity and keeping things self-contained and lightweight. Hope this helps. -- Regards Anuradha
Re: Init systems and docker
On 10/14/19 05:49, Marc Haber wrote: > > So, +1 to be able to run a default systemd-based Debian with systemd > as pid 1 in a docker container. Hi Marc, you can do this already. As explained somewhere else in the thread, just run docker run with the --privilege flag. Cheers. -- __.https://wiki.debian.org/JoseLuisRivas ___'/___ rsa4096/D278F9C15E5461AA3C1E2FCD13EC43EEB9AC8C43
Re: Init systems and docker
On Fri, 11 Oct 2019 19:25:54 -0400, Jose-Luis Rivas wrote: >There's not much sense in using systemd inside a docker container, to be >honest. Generally you want to launch your service as custom as possible >and the ENTRYPOINT allows you to do just that. Docker already sends the >SIGKILL to the PID 1, which should be the service you're shipping in >that container. Generally, I concur. But there are other uses for containers than running a production service. For example; I'd love to have the possibility to just fire up a Debian that is as close to a standard installation to try "something". Docker is the natual way to do this when you already have docker to run services. Then it's only the work of creating (or using) a Debian image to instantiate the container and to get rid of it quickly after the tests have been completed. LXC might be the more natural tool to use for this, but this is a different tool that needs different knowledge, and it's tooling is way behind what we have with docker. So, +1 to be able to run a default systemd-based Debian with systemd as pid 1 in a docker container. Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber | " Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom " | Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834
Re: Init systems and docker
Hi Tomas!
On 10/12/19 7:00 PM, Tomas Pospisek wrote:
> I have to say that I disagree with you and many others on this thread.
> Maybe Docker was *meant* for single application containers, I do not know.
In theory, yes.
> However running a service ("a single application") often implies
> surrounding services. F.ex. you want logs to be saved? Maybe you need to
> run cron or at? Maybe you want to get notified about problems, stats,
> whatever via email?
There are several options:
- use more than one container. Your mailserver happily runs in a
different container and can still be used by your application. Same for
cronjobs (with k8s, you'd want to look into k8s based cronjobs) - in a
lot of cases there is no need to have cron running in the same container
as your application.
- start more than one application from your entrypoint.sh
- if you really, really need it: use supervisord.
>[...]
> Now the next problem is how to start those. Easiest way is to hook the
> provided Debian init scripts into whatever mini-init system one chooses.
> And so forth.
please don't use init-scripts in containers... see above.
Containers are not meant to be full-blown Linux stacks with everything
you can imagine in one container.
Bernd
--
Bernd ZeimetzDebian GNU/Linux Developer
http://bzed.dehttp://www.debian.org
GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Re: Init systems and docker
Hello Tomas!
On 10/12/19 13:00, Tomas Pospisek wrote:
> I have to say that I disagree with you and many others on this thread.
> Maybe Docker was *meant* for single application containers, I do not know.
>
> However running a service ("a single application") often implies
> surrounding services. F.ex. you want logs to be saved? Maybe you need to
> run cron or at? Maybe you want to get notified about problems, stats,
> whatever via email?
In all the professional instances of containerization I have seen, these
are provided by external containers/services, so you don't have to
maintain several configuration in several places. Logs go to stdout and
from there you collect them into whatever service you have for this,
process them and use that processing to send notifications or make them
searchable.
>
> Now you can start re-implementing all the existing "surrounding" service
> solutions on the outside of the container. Which is a *lot* of complex
> work in my experience. The quick fix to those "surrounding" problems is
> often enough to stand onto proven-to-work shoulders and to install the
> "surrounding services" *inside* the container:
>
> apt-get install cron at rsyslogd etc. etc.
>
> Now the next problem is how to start those. Easiest way is to hook the
> provided Debian init scripts into whatever mini-init system one chooses.
> And so forth.
>
> So I imagine people that want to run stuff in containers are usually
> very glad if init scripts are available, work and can be re-used.
>
> John Goerzen is maintaining Debian docker images (thanks!) that contain
> a useful set of the mentioned "surrounding" services, that are quite
> popular AFAIK.
>
> Being you, I'd ask for patches. I think running stuff in
> Docker/Kubernetes is a good solution for a lot of problems. I think
> Debian should grow to accommodate those architectures. And I think
> Debian *will* accommodate them (see John's work and a lot of other nice
> efforts in that direction not least by Ubuntu), it just takes time to
> find the sweet spot solutions, to spread the knowledge and knowhow etc.
> A lot of container loads are Debian based, not the least a lot of
> Kubernetes' own people's
Debian already does for a part of container users. You don't need
patches, support is already there. The ones complaining right now
haven't read the right part in Docker's manual to get it running.
Cheers.
--
__.https://wiki.debian.org/JoseLuisRivas
___'/___ rsa4096/D278F9C15E5461AA3C1E2FCD13EC43EEB9AC8C43
Re: Init systems and docker
Hi,
On 12.10.19 19:00, Tomas Pospisek wrote:
> However running a service ("a single application") often implies
> surrounding services. F.ex. you want logs to be saved? Maybe you need to
> run cron or at? Maybe you want to get notified about problems, stats,
> whatever via email?
That's the point of containerization though: a site-wide service
orchestrator keeps track of services and their dependencies. You likely
only need a single mailer instance for however many of your application
containers you run, and your application should only talk to some
abstract service provider through a well-defined interface, so the
actual implementation behind it can be scaled and updated independently.
Containers and systemd solve the same problem at different scales
(site-wide vs per-host).
Stacking two systems that implement the same feature on top of each
other usually gives you bad side effects. Linux knows that virtio disks
should have an arbitrarily long command queue and no scheduler, because
I/O schedulers do not stack. RAID controllers rely on harddisks to
report read errors immediately instead of retrying. TCP-over-TCP behaves
erratically.
The only place where I could see a potential use would be exporting dbus
services between containers, similar to DCOM in Windows, but everyone
seems to be happy with socket connections and per-application protocol
definitions.
Simon
signature.asc
Description: OpenPGP digital signature
Re: Init systems and docker
Hi,
Am 12.10.19 um 19:00 schrieb Tomas Pospisek:
> However running a service ("a single application") often implies
> surrounding services. F.ex. you want logs to be saved? Maybe you need to
> run cron or at? Maybe you want to get notified about problems, stats,
> whatever via email?
>
> Now you can start re-implementing all the existing "surrounding" service
> solutions on the outside of the container. Which is a *lot* of complex
> work in my experience. The quick fix to those "surrounding" problems is
> often enough to stand onto proven-to-work shoulders and to install the
> "surrounding services" *inside* the container:
>
> apt-get install cron at rsyslogd etc. etc.
I don't see why you say, that you would have to *re-implement* the
surrounding solutions on the outside of the container. From my
experience, the most appropriate solution for having cron, at, and
rsyslogd run alongside you main service would be to put each of them
into their own container and link the containers together as needed.
This might sound more complicated than just installing everything you
need into one container, but container based service management (such as
kubernetes or docker swarm) are (at least in my experience) a lot easier
to manage and more effective, when you consistently separate each
service into its own container.
Regards
Sven
signature.asc
Description: OpenPGP digital signature
Re: Init systems and docker
Hi Debianistas!
Am 12.10.19 um 01:06 schrieb Ben Hutchings:
> On Fri, 2019-10-11 at 18:49 -0400, Scott Kitterman wrote:
>> I have had bugs filed against more than one package I maintain regarding
>> issues
>> with sysv init scripts when used in docker.
>>
>> I have been told by docker users (I'm not one) that systemd as provided on
>> Debian can't be used in docker. I have no idea if that's true or not. I
>> try
>> really hard to know as little about init systems as possible and trust our
>> maintainers who work on such things.
>>
>> If it is true, then to the extent we want Debian to be useful for docker
>> does
>> that mean we ought to maintain sysv init scripts? If it's not true, can
>> someone point me to documentation that explains using systemd on Debian in
>> docker?
> [...]
>
> As I understand it, Docker is meant for application containers, and
> application containers should have a trivial init process that directly
> launches a single application process. No init script, or indeed any
> shell script, should be used at all.
I have to say that I disagree with you and many others on this thread.
Maybe Docker was *meant* for single application containers, I do not know.
However running a service ("a single application") often implies
surrounding services. F.ex. you want logs to be saved? Maybe you need to
run cron or at? Maybe you want to get notified about problems, stats,
whatever via email?
Now you can start re-implementing all the existing "surrounding" service
solutions on the outside of the container. Which is a *lot* of complex
work in my experience. The quick fix to those "surrounding" problems is
often enough to stand onto proven-to-work shoulders and to install the
"surrounding services" *inside* the container:
apt-get install cron at rsyslogd etc. etc.
Now the next problem is how to start those. Easiest way is to hook the
provided Debian init scripts into whatever mini-init system one chooses.
And so forth.
So I imagine people that want to run stuff in containers are usually
very glad if init scripts are available, work and can be re-used.
John Goerzen is maintaining Debian docker images (thanks!) that contain
a useful set of the mentioned "surrounding" services, that are quite
popular AFAIK.
Being you, I'd ask for patches. I think running stuff in
Docker/Kubernetes is a good solution for a lot of problems. I think
Debian should grow to accommodate those architectures. And I think
Debian *will* accommodate them (see John's work and a lot of other nice
efforts in that direction not least by Ubuntu), it just takes time to
find the sweet spot solutions, to spread the knowledge and knowhow etc.
A lot of container loads are Debian based, not the least a lot of
Kubernetes' own people's.
This is my view of course, which *might* be herethic to the narrower
Docker and Kubernetes orthodoxies.
Re: Init systems and docker
On Fri, 11 Oct 2019 at 18:49:37 -0400, Scott Kitterman wrote: > I have been told by docker users (I'm not one) that systemd as provided on > Debian can't be used in docker. I have no idea if that's true or not. It isn't, as noted elsewhere on this thread. However, my understanding is that they are sort of correct, because systemd can't be used *in a Docker container that isn't specially configured*. It used to be the case that systemd needed `docker run --privileged`, which removes the security boundary between host and container (aka "containers don't contain"). systemd has improved since then, and if given some specific bind-mounts from the host system (which can be read-only) it will adapt its behaviour to be more container-friendly - so it still needs special configuration from outside the container, but the special configuration is now simpler and "more contained". > If it is true, then to the extent we want Debian to be useful for docker does > that mean we ought to maintain sysv init scripts? I think it's necessary to distinguish between two aspects of init here: * pid 1 (process reaper of last resort) * service manager In systemd-world, both of these are the systemd manager, /lib/systemd/systemd. In sysvinit-world, sysvinit itself is a process reaper with some mostly-vestigial service management (configured in /etc/inittab), but 95% of the service management is delegated to sysv-rc, OpenRC or some similar framework (for services that start during boot), or to service(8) and friends (for services that are restarted or otherwise controlled after the system is booted). With Debian's historical (non-systemd) defaults, a small minority of services (mostly just getty) are managed by sysvinit itself, but if it involves init.d or rc.d, then it's really sysv-rc's job, not sysvinit's; and The Thread about systemd vs. sysvinit was in fact mostly about systemd vs. sysv-rc. The fact that we call that configuration "sysvinit" is not really very helpful, because it makes sysvinit's pid 1 seem more important than it actually is. If you will ever have more than one process in your container, you need a pid 1 that reaps processes, but it doesn't have to be as big as systemd or even sysvinit: tini will do fine. `docker run --init` provides a miniature init process for this purpose. Booting a full operating system in a Docker container is not what it is designed for (perhaps you *can*, but that doesn't necessarily mean you *should*). The way I tend think about it is that Docker is a heavyweight chroot, rather than a lightweight virtual machine. Some other container frameworks, like lxc/lxd, aim to behave more like a virtual machine, and for those you do need a full service management framework like systemd or sysv-rc. smcv
Re: Init systems and docker
Hello, Le sam. 12 oct. 2019 à 01:26, Jose-Luis Rivas a écrit : > > Hello Scott, > > On 10/11/19 18:49, Scott Kitterman wrote: [...] > In short: they should not be using systemd inside the container and if > they want to the issue is not on how Debian ships systemd but that they > are not using the --privilege parameter for launching their container. While I agree to not recommend using systemd inside docker, it is very useful in CI. We use this at work to test our salt states (salt is like puppet/ansible/chef/...). For this, we use Gitlab CI running test-kitchen with ruby-kitchen-salt and ruby-kitchen-docker (all in buster). Another solution would either mean hacking gitlab-runner [1] or spanning a lot of VMs. [1]: https://gitlab.com/gitlab-org/gitlab-runner/issues/1585 Also systemd can be run inside Docker without --privileged but requires careful configuration [2]. [2]: https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-non-privileged-container/ Regards -- Mathieu Parent Parent
Re: Init systems and docker
On Fri, Oct 11, 2019 at 18:49, Scott Kitterman wrote: If it is true, then to the extent we want Debian to be useful for docker does that mean we ought to maintain sysv init scripts? If it's not true, can someone point me to documentation that explains using systemd on Debian in docker? https://wiki.debian.org/Packaging/Pre-Requisites#Docker_with_non-Debian-based_host
Re: Init systems and docker
On Fri, 2019-10-11 at 19:25 -0400, Jose-Luis Rivas wrote: > There's not much sense in using systemd inside a docker container, to > be honest. To put it another way, in the container world the init system belongs outside of the container. That is because the closest thing equivalent to a container is not a box running Debian doing multiple things, it's a single process. In fact if you are Google it's actually simpler - it's a process whose executable is a single statically linked file. It pulls it's configuration from something like etcd (something that lives in the cloud), it uses the file system as an extension of it's RAM - temporary storage that can be blown away when the process exits, if it needs persistent storage comes in the form of a database it connects to using TCP/IP. It's utility doesn't arise because it's more powerful and flexible than the environment Debian provides. On the contrary, it is fare less rich, and flexible than the environment Debian provides. It's power arises because reduces the dependencies to an absolute minimum. It is one file whose main configuration is the TCP endpoints its receives commands from and sends results to, and TCP endpoints of services, log storage and databases it depends on. It's only dependency outside of that list of endpoints is an amd64 machine with some amount of RAM and CPU cycles and the Linux syscall API - something that can be replicated 1000's or millions of times automatically and on demand. A major part of it's utility that unlike Debian dependencies (like say a Web Server or a .so of the right version) that programmers tend to assume is there or their program can break in weird and wonderful ways, programmers tend to assume this one dependency, a TCP connection, is unreliable and ephemeral. So you can create these things and destroy them to handle varying loads, and outages in any one data centre can be handled by shifting the load to another one. The consistency of persistent data is not a problem because it's stored in a full ACID database. An init system of any sort in this is just unwanted complexity. What replaces the init system is something starts up these containers and connects them together on an as needed basis. Kubernetes is one such system, but there alternatives. Docker Swarm is a much simpler thing that creates something is closer to a Debian box (a collection of containers running on one physical box). In the most evolved container systems inside a container it's not just the init system that's considered an anti-pattern, a distribution Debian is also out of place. None of the services Debian provides (like packaging, dependencies, security upgraded) are helpful. The host box will have an OS - but it's just a support system for Kubernetes and you don't really need an entire distribution just to run one program. However, that is the extreme edge. It's where you have a team of programmers creating a static app that includes all of the services it needs, like a web server. Us smaller guys will use pre-packaged software provided by a distribution simply because we are familiar with nginx or apache, and don't want to compile stuff to do off the shelf functions. The rest remains the same - a container is a single process, configuration is primarily connecting TCP end points, things are written to retry when a TCP connection dies, security patches are "installed" by rebuilding the container and restarting it, shells and what not in containers are there merely to facilitate development, debugging and fault analysis. The most popular container distribution is I think Alpine Linux which has an impressively small base install. If Debian wanted to compete in this area it would have to start by shrinking it's minbase install by a factor or 10 or so. Anybody arguing about what sort of init system that install includes hasn't groked containers yet. signature.asc Description: This is a digitally signed message part
Re: Init systems and docker
Forgot to add that docker has a recommended way of running multiple processes per container and it actually avoids init systems launching its own micro init: https://docs.docker.com/config/containers/multi-service_container/ On 10/11/19 19:25, Jose-Luis Rivas wrote > Here's a little bit of history on trying to make systemd work into > containers and the reason why it was tried (IMO, a silly one): > https://lwn.net/Articles/676831/ > > In short: they should not be using systemd inside the container and if > they want to the issue is not on how Debian ships systemd but that they > are not using the --privilege parameter for launching their container. > > The extended version: > > There's not much sense in using systemd inside a docker container, to be > honest. Generally you want to launch your service as custom as possible > and the ENTRYPOINT allows you to do just that. Docker already sends the > SIGKILL to the PID 1, which should be the service you're shipping in > that container. > > Ideally (and this is one the best practices I've seen people using) > containers should be built following something like the UNIX paradigm, > so: they ship one service and only one which then communicates with > others via some network protocol. People trying to use systemd inside a > container is probably trying to ship several services in just one image > which is essentially an anti-pattern on containerization. > > On the other hand, not all people is using a container to launch a > service installed via the package manager, so very few would benefit > from having systemd doing the init of the service they installed. > > In my opinion, there's no requirement at all to maintain sysv scripts, > at least not under the assumption that they are required by docker > containers because they are not. Actually, no init script at all is > required by docker containers. > -- __.https://wiki.debian.org/JoseLuisRivas ___'/___ rsa4096/D278F9C15E5461AA3C1E2FCD13EC43EEB9AC8C43
Re: Init systems and docker
Hello Scott, On 10/11/19 18:49, Scott Kitterman wrote: > I have had bugs filed against more than one package I maintain regarding > issues > with sysv init scripts when used in docker. > > I have been told by docker users (I'm not one) that systemd as provided on > Debian can't be used in docker. I have no idea if that's true or not. I try > really hard to know as little about init systems as possible and trust our > maintainers who work on such things. > > If it is true, then to the extent we want Debian to be useful for docker does > that mean we ought to maintain sysv init scripts? If it's not true, can > someone point me to documentation that explains using systemd on Debian in > docker? > > Thanks. > > I'm sure I'll fail, but I'm really not trying to start another email list > flamefest. I'd just like to understand this a little better so I can > properly > maintain my packages. > > Scott K > > Here's a little bit of history on trying to make systemd work into containers and the reason why it was tried (IMO, a silly one): https://lwn.net/Articles/676831/ In short: they should not be using systemd inside the container and if they want to the issue is not on how Debian ships systemd but that they are not using the --privilege parameter for launching their container. The extended version: There's not much sense in using systemd inside a docker container, to be honest. Generally you want to launch your service as custom as possible and the ENTRYPOINT allows you to do just that. Docker already sends the SIGKILL to the PID 1, which should be the service you're shipping in that container. Ideally (and this is one the best practices I've seen people using) containers should be built following something like the UNIX paradigm, so: they ship one service and only one which then communicates with others via some network protocol. People trying to use systemd inside a container is probably trying to ship several services in just one image which is essentially an anti-pattern on containerization. On the other hand, not all people is using a container to launch a service installed via the package manager, so very few would benefit from having systemd doing the init of the service they installed. In my opinion, there's no requirement at all to maintain sysv scripts, at least not under the assumption that they are required by docker containers because they are not. Actually, no init script at all is required by docker containers. -- __.https://wiki.debian.org/JoseLuisRivas ___'/___ rsa4096/D278F9C15E5461AA3C1E2FCD13EC43EEB9AC8C43
Re: Init systems and docker
On Fri, 2019-10-11 at 18:49 -0400, Scott Kitterman wrote: > I have had bugs filed against more than one package I maintain regarding > issues > with sysv init scripts when used in docker. > > I have been told by docker users (I'm not one) that systemd as provided on > Debian can't be used in docker. I have no idea if that's true or not. I try > really hard to know as little about init systems as possible and trust our > maintainers who work on such things. > > If it is true, then to the extent we want Debian to be useful for docker does > that mean we ought to maintain sysv init scripts? If it's not true, can > someone point me to documentation that explains using systemd on Debian in > docker? [...] As I understand it, Docker is meant for application containers, and application containers should have a trivial init process that directly launches a single application process. No init script, or indeed any shell script, should be used at all. When people want to put Debian in a container, rather than a single application, they would probably do better to use LXC, which I think allows running pretty much any init system. All that said, Docker has name recognition to the point that's is almost synonymous with containers, so people are going to keep trying to use it for things it doesn't do well. Ben. -- Ben Hutchings It is easier to change the specification to fit the program than vice versa. signature.asc Description: This is a digitally signed message part
Init systems and docker
I have had bugs filed against more than one package I maintain regarding issues with sysv init scripts when used in docker. I have been told by docker users (I'm not one) that systemd as provided on Debian can't be used in docker. I have no idea if that's true or not. I try really hard to know as little about init systems as possible and trust our maintainers who work on such things. If it is true, then to the extent we want Debian to be useful for docker does that mean we ought to maintain sysv init scripts? If it's not true, can someone point me to documentation that explains using systemd on Debian in docker? Thanks. I'm sure I'll fail, but I'm really not trying to start another email list flamefest. I'd just like to understand this a little better so I can properly maintain my packages. Scott K
