subject:"Re\: \[openstack\-dev\] \[kolla\] on Dockerfile patterns"


On 10/14/2014 06:10 PM, Clint Byrum wrote:

Excerpts from Fox, Kevin M's message of 2014-10-14 17:40:16 -0700:

I'm not arguing that everything should be managed by one systemd, I'm
just saying, for certain types of containers, a single docker container
with systemd in it might be preferable to trying to slice it unnaturally
into several containers.


Can you be more concrete? Most of the time things that need to be in
the same machine tend to have some kind of controller already. Meanwhile
it is worth noting that you can have one _image_, but several containers
running from that one image. So if you're trying to run a few pieces of
Neutron, for instance, you can have multiple containers each from that
one neutron image.


Systemd has invested a lot of time/effort to be able to relaunch failed
services, support spawning and maintaining unix sockets and services
across them, etc, that you'd have to push out of and across docker
containers. All of that can be done, but why reinvent the wheel? Like you
said, pacemaker can be made to make it all work, but I have yet to see
a way to deploy pacemaker services anywhere near as easy as systemd+yum
makes it. (Thanks be to redhat. :)


There are some of us who are rather annoyed that systemd tries to do
this in such a naive way and assumes everyone will want that kind of
management. It's the same naiveté that leads people to think if they
make their app server systemd service depend on their mysql systemd
service that this will eliminate startup problems. Once you have more
than one server, it doesn't work.

Kubernetes adds a distributed awareness of the containers that makes it
uniquely positioned to do most of those jobs much better than systemd
can.


The answer seems to be, its not "dockerish". Thats ok. I just wanted to
understand the issue for what it is. If there is a really good reason for
not wanting to do it, or that its just "not the way things are done". I've
had kind of the opposite feeling regarding docker containers. Docker use
to do very bad things when killing the container. nasty if you wanted
your database not to go corrupt. killing pid 1 is a bit sketchy then
forcing the container down after 10 seconds was particularly bad. having
something like systemd in place allows the database to be notified, then
shutdown properly. Sure you can script up enough shell to make this work,
but you have to do some difficult code, over and over again... Docker
has gotten better more recently but it still makes me a bit nervous
using it for statefull things.


What I think David was saying was that the process you want to run under
systemd is the pid 1 of the container. So if killing that would be bad,
it would also be bad to stop the systemd service, which would do the
same thing: send it SIGTERM. If that causes all hell to break loose, the
stateful thing isn't worth a dime, because it isn't crash safe.


As for recovery, systemd can do the recovery too. I'd argue at this
point in time, I'd expect systemd recovery to probably work better
then some custom shell scripts when it comes to doing the right thing
recovering at bring up. The other thing is, recovery is not just about
pid1 going away. often it sticks around and other badness is going
on. Its A way to know things are bad, but you can't necessarily rely on
it to know the container's healty. You need more robust checks for that.

I think one thing people like about Kubernetes is that when a container
crashes, and needs to be brought back up, it may actually be brought
up on a different, less busy, more healthy host. I could be wrong, or
that might be in the "FUTURE" section. But the point is, recovery and
start-up are not things that always want to happen on the same box.
yes FUTURE as  t his doesn't exist today.  There is a significant runway 
to enhancing the availability management of kubernetes natively.


Regards
-steve


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

On 10/14/2014 05:44 PM, Angus Lees wrote:

On Tue, 14 Oct 2014 07:51:54 AM Steven Dake wrote:

Angus,

On 10/13/2014 08:51 PM, Angus Lees wrote:

I've been reading a bunch of the existing Dockerfiles, and I have two
humble requests:

1. It would be good if the "interesting" code came from python
sdist/bdists
rather than rpms.

This will make it possible to rebuild the containers using code from a
private branch or even unsubmitted code, without having to go through a
redhat/rpm release process first.

I care much less about where the python dependencies come from. Pulling
them from rpms rather than pip/pypi seems like a very good idea, given
the relative difficulty of caching pypi content and we also pull in the
required C, etc libraries for free.

With this in place, I think I could drop my own containers and switch to
reusing kolla's for building virtual testing environments. This would
make me happy.

I've captured this requirement here:
https://blueprints.launchpad.net/kolla/+spec/run-from-master

I also believe it would be interesting to run from master or a stable
branch for CD. Unfortunately I'm still working on the nova-compute
docker code, but if someone comes along and picks up that blueprint, i
expect it will get implemented :) Maybe that could be you.

Yeah I've already got a bunch of working containers that pull from master[1],
but I've been thinking I should change that to use an externally supplied
bdist. The downside is you quickly end up wanting a docker container to build
your deployment docker container. I gather this is quite a common thing to
do, but I haven't found the time to script it up yet.

[1] https://github.com/anguslees/kube-openstack/tree/master/docker

I could indeed work on this, and I guess I was gauging the level of enthusiasm
within kolla for such a change. I don't want to take time away from the
alternative I have that already does what I need only to push uphill to get it
integrated :/
There would be no uphill push. For milestone #2, I am already going to
reorganize the docker directory to support centos+rdo as an alternative
to fedora+rdo. Fedora+master is just another directory in this model
(or Ubuntu + master if you want that choice as well). IMO the more
choice about deployment platforms the better, especially a master model
(or more likely a stable branch model).

Regards
-steve

2. I think we should separate out "run the server" from "do once-off
setup".

Currently the containers run a start.sh that typically sets up the
database, runs the servers, creates keystone users and sets up the
keystone catalog. In something like k8s, the container will almost
certainly be run multiple times in parallel and restarted numerous times,
so all those other steps go against the service-oriented k8s ideal and
are at-best wasted.

The server may not start before the configuration of the server is
complete. I guess I don't quite understand what you indicate here when
you say we have separate pods that do initial database setup/migrate.
Do you mean have dependencies in some way, or for eg:

glance-registry-setup-pod.yaml - the glance registry pod descriptor
which sets up the db and keystone
glance-registry-pod.yaml - the glance registry pod descriptor which
starts the application and waits for db/keystone setup

and start these two pods as part of the same selector (glance-registry)?

That idea sounds pretty appealing although probably won't be ready to go
for milestone #1.

So the way I do it now, I have a replicationController that starts/manages
(eg) nova-api pods[2]. I separately have a nova-db-sync pod[3] that basically
just runs "nova-manage db sync".

I then have a simple shell script[4] that starts them all at the same time.
The nova-api pods crash and get restarted a few times until the database has
been appropriately configured by the nova-db-sync pod, and then they're fine and
start serving.

When nova-db-sync exits successfully, the pod just sits in state terminated
thanks to restartPolicy: onFailure. Sometime later I can delete the
terminated nova-db-sync pod, but it's also harmless if I just leave it or even
if it gets occasionally re-run as part of some sort of update.

[2]
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-api-repcon.yaml
[3]
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml
[4] https://github.com/anguslees/kube-openstack/blob/master/kubecfg-create.sh

I'm open to whether we want to make these as lightweight/independent as
possible (every daemon in an individual container), or limit it to one per
project (eg: run nova-api, nova-conduc

Re: [openstack-dev] [kolla] on Dockerfile patterns


On 10/14/2014 04:52 PM, David Vossel wrote:


- Original Message -

Ok, why are you so down on running systemd in a container?

It goes against the grain.

 From a distributed systems view, we gain quite a bit of control by maintaining
"one service per container". Containers can be re-organised and re-purposed 
dynamically.
If we have systemd trying to manage an entire stack of resources within a 
container,
we lose this control.

 From my perspective a containerized application stack needs to be managed 
externally
by whatever is orchestrating the containers to begin with. When we take a step 
back
and look at how we actually want to deploy containers, systemd doesn't make 
much sense.
It actually limits us in the long run.

Also... recovery. Using systemd to manage a stack of resources within a single 
container
makes it difficult for whatever is externally enforcing the availability of 
that container
to detect the health of the container.  As it is now, the actual service is pid 
1 of a
container. If that service dies, the container dies. If systemd is pid 1, there 
can
be all kinds of chaos occurring within the container, but the external 
distributed
orchestration system won't have a clue (unless it invokes some custom health 
monitoring
tools within the container itself, which will likely be the case someday.)
I tend to agree systemd makes healthchecking and recovery escalation 
more difficult/impossible.  At a minimum to do escalation with systemd, 
the external orch system (k8s) needs to run code in the container to 
healthcheck the services.  This can be done today with k8s, but I fail 
to see why it is necessary to involve the complexity of systemd.  The 
systemd system is pretty sweet for multiple processes in one container, 
but systemd doesn't meld well with the one application per container 
model that appears to be best practice at this time.


Regards,
-steve


-- Vossel



Pacemaker works, but its kind of a pain to setup compared just yum installing
a few packages and setting init to systemd. There are some benefits for
sure, but if you have to force all the docker components onto the same
physical machine anyway, why bother with the extra complexity?

Thanks,
Kevin


From: David Vossel [dvos...@redhat.com]
Sent: Tuesday, October 14, 2014 3:14 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

- Original Message -

Same thing works with cloud init too...


I've been waiting on systemd working inside a container for a while. it
seems
to work now.

oh no...


The idea being its hard to write a shell script to get everything up and
running with all the interactions that may need to happen. The init
system's
already designed for that. Take a nova-compute docker container for
example,
you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
celiometer-agent all backed in. Writing a shell script to get it all
started
and shut down properly would be really ugly.

You could split it up into 4 containers and try and ensure they are
coscheduled and all the pieces are able to talk to each other, but why?
Putting them all in one container with systemd starting the subprocesses is
much easier and shouldn't have many drawbacks. The components code is
designed and tested assuming the pieces are all together.

What you need is a dependency model that is enforced outside of the
containers. Something
that manages the order containers are started/stopped/recovered in. This
allows
you to isolate your containers with 1 service per container, yet still
express that
container with service A needs to start before container with service B.

Pacemaker does this easily. There's even a docker resource-agent for
Pacemaker now.
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker

-- Vossel

ps. don't run systemd in a container... If you think you should, talk to me
first.


You can even add a ssh server in there easily too and then ansible in to do
whatever other stuff you want to do to the container like add other
monitoring and such

Ansible or puppet or whatever should work better in this arrangement too
since existing code assumes you can just systemctl start foo;

Kevin

From: Lars Kellogg-Stedman [l...@redhat.com]
Sent: Tuesday, October 14, 2014 12:10 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:

With Docker, you are limited to the operating system of whatever the
image
uses.

See, that's the part I disagree with.  What I was saying about ansible
and puppet in my email is that I think the right thing to do is take
advantage of those tools:

   FROM ubuntu

   RUN apt-get install ansible
   COP

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread Lars Kellogg-Stedman

On Wed, Oct 15, 2014 at 01:50:08PM -0400, David Vossel wrote:
> Something like LSB init scripts except tailored towards the container use 
> case.
> The primary difference would be that the 'start' action of this new standard
> wouldn't fork. Instead 'start' would be pid 1. The 'status' could be checked
> externally by calling the exact same entry point script to invoke the 'status'
> function.

With the 1.3 release, which introduces "docker exec", you could just
about get there.  Rather than attempting to introspect the container
to find the entrypoint script -- which might not even exist -- I would
say standardize on some top level paths (e.g., '/status') that can be
used to run a status check, and leave the implementation of those
paths up to the image (maybe they're scripts, maybe they're binaries,
just as long as they are executable).

Then you check would boil down to:

  docker exec  /status

The reason why I am trying to avoid assuming some specially
constructed entrypoint script is that many images will simply not have
one -- they simply provide an initial command via CMD. Adding a
/status script or similar in this case is very simple:

   FROM original_image
   ADD status /status

Doing this via an entrypoint script would be a bit more complicated:

- You would have to determine whether or not the original image had an
  existing entrypoint script.
- If so you would need to wrap it or replicate the functionality.
- Some images may have entrypoint scripts that already provide
  "subcommand" like functionality (docker run someimage ,
  where  is parsed by the entrypoint script) and might not be
  compatible with an entrypoint-based status check.

Otherwise, I think establishing a "best practice" mechanism for executing
in-container checks is a great idea.

-- 
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/

pgpD1P2Vb3KMF.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread David Vossel



- Original Message -
> Excerpts from Vishvananda Ishaya's message of 2014-10-15 07:52:34 -0700:
> > 
> > On Oct 14, 2014, at 1:12 PM, Clint Byrum  wrote:
> > 
> > > Excerpts from Lars Kellogg-Stedman's message of 2014-10-14 12:50:48
> > > -0700:
> > >> On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:
> > >>> I think the above strategy is spot on. Unfortunately, that's not how
> > >>> the
> > >>> Docker ecosystem works.
> > >> 
> > >> I'm not sure I agree here, but again nobody is forcing you to use this
> > >> tool.
> > >> 
> > >>> operating system that the image is built for. I see you didn't respond
> > >>> to my
> > >>> point that in your openstack-containers environment, you end up with
> > >>> Debian
> > >>> *and* Fedora images, since you use the "official" MySQL dockerhub
> > >>> image. And
> > >>> therefore you will end up needing to know sysadmin specifics (such as
> > >>> how
> > >>> network interfaces are set up) on multiple operating system
> > >>> distributions.
> > >> 
> > >> I missed that part, but ideally you don't *care* about the
> > >> distribution in use.  All you care about is the application.  Your
> > >> container environment (docker itself, or maybe a higher level
> > >> abstraction) sets up networking for you, and away you go.
> > >> 
> > >> If you have to perform system administration tasks inside your
> > >> containers, my general feeling is that something is wrong.
> > >> 
> > > 
> > > Speaking as a curmudgeon ops guy from "back in the day".. the reason
> > > I choose the OS I do is precisely because it helps me _when something
> > > is wrong_. And the best way an OS can help me is to provide excellent
> > > debugging tools, and otherwise move out of the way.
> > > 
> > > When something _is_ wrong and I want to attach GDB to mysqld in said
> > > container, I could build a new container with debugging tools installed,
> > > but that may lose the very system state that I'm debugging. So I need to
> > > run things inside the container like apt-get or yum to install GDB.. and
> > > at some point you start to realize that having a whole OS is actually a
> > > good thing even if it means needing to think about a few more things up
> > > front, such as "which OS will I use?" and "what tools do I need installed
> > > in my containers?"
> > > 
> > > What I mean to say is, just grabbing off the shelf has unstated
> > > consequences.
> > 
> > If this is how people are going to use and think about containers, I would
> > submit they are a huge waste of time. The performance value they offer is
> > dramatically outweighed by the flexibilty and existing tooling that exists
> > for virtual machines. As I state in my blog post[1] if we really want to
> > get value from containers, we must convert to the single application per
> > container view. This means having standard ways of doing the above either
> > on the host machine or in a debugging container that is as easy (or easier)
> > than the workflow you mention. There are not good ways to do this yet, and
> > the community hand-waves it away, saying things like, "well you could …”.
> > You could isn’t good enough. The result is that a lot of people that are
> > using containers today are doing fat containers with a full os.
> > 
> 
> I think we really agree.
> 
> What the container universe hasn't worked out is all the stuff that the
> distros have worked out for a long time now: consistency.

I agree we need consistency. I have an idea. What if we developed an entrypoint
script standard...

Something like LSB init scripts except tailored towards the container use case.
The primary difference would be that the 'start' action of this new standard
wouldn't fork. Instead 'start' would be pid 1. The 'status' could be checked
externally by calling the exact same entry point script to invoke the 'status'
function.

This standard would lock us into the 'one service per container' concept while
giving us the ability to standardize on how the container is launched and 
monitored.

If we all conformed to something like this, docker could even extend the 
standard
so health checks could be performed using the docker cli tool.

docker status 

Internally docker would just be doing a nsenter into the container and calling
the internal status function in our init script standard.

We already have docker start  and docker stop . Being able
to generically call something like docker status  and have that 
translate
into some service specific command on the inside of the container would be kind 
of
neat.

Tools like kubernetes could use this functionality to poll a container's health 
and
be able to detect issues occurring within the container that don't necessarily
involve the container's service failing.

Does anyone else have any interest in this? I have quite a bit of of init 
script type
standard experience. It would be trivial for me to define something like this 
for us
to begin discussing.

-- Vossel

> I think it would be a good id

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread Clint Byrum

Excerpts from Vishvananda Ishaya's message of 2014-10-15 07:52:34 -0700:
> 
> On Oct 14, 2014, at 1:12 PM, Clint Byrum  wrote:
> 
> > Excerpts from Lars Kellogg-Stedman's message of 2014-10-14 12:50:48 -0700:
> >> On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:
> >>> I think the above strategy is spot on. Unfortunately, that's not how the
> >>> Docker ecosystem works.
> >> 
> >> I'm not sure I agree here, but again nobody is forcing you to use this
> >> tool.
> >> 
> >>> operating system that the image is built for. I see you didn't respond to 
> >>> my
> >>> point that in your openstack-containers environment, you end up with 
> >>> Debian
> >>> *and* Fedora images, since you use the "official" MySQL dockerhub image. 
> >>> And
> >>> therefore you will end up needing to know sysadmin specifics (such as how
> >>> network interfaces are set up) on multiple operating system distributions.
> >> 
> >> I missed that part, but ideally you don't *care* about the
> >> distribution in use.  All you care about is the application.  Your
> >> container environment (docker itself, or maybe a higher level
> >> abstraction) sets up networking for you, and away you go.
> >> 
> >> If you have to perform system administration tasks inside your
> >> containers, my general feeling is that something is wrong.
> >> 
> > 
> > Speaking as a curmudgeon ops guy from "back in the day".. the reason
> > I choose the OS I do is precisely because it helps me _when something
> > is wrong_. And the best way an OS can help me is to provide excellent
> > debugging tools, and otherwise move out of the way.
> > 
> > When something _is_ wrong and I want to attach GDB to mysqld in said
> > container, I could build a new container with debugging tools installed,
> > but that may lose the very system state that I'm debugging. So I need to
> > run things inside the container like apt-get or yum to install GDB.. and
> > at some point you start to realize that having a whole OS is actually a
> > good thing even if it means needing to think about a few more things up
> > front, such as "which OS will I use?" and "what tools do I need installed
> > in my containers?"
> > 
> > What I mean to say is, just grabbing off the shelf has unstated
> > consequences.
> 
> If this is how people are going to use and think about containers, I would
> submit they are a huge waste of time. The performance value they offer is
> dramatically outweighed by the flexibilty and existing tooling that exists
> for virtual machines. As I state in my blog post[1] if we really want to
> get value from containers, we must convert to the single application per
> container view. This means having standard ways of doing the above either
> on the host machine or in a debugging container that is as easy (or easier)
> than the workflow you mention. There are not good ways to do this yet, and
> the community hand-waves it away, saying things like, "well you could …”.
> You could isn’t good enough. The result is that a lot of people that are
> using containers today are doing fat containers with a full os.
> 

I think we really agree.

What the container universe hasn't worked out is all the stuff that the
distros have worked out for a long time now: consistency.

I think it would be a good idea for containers' filesystem contents to
be a whole distro. What's at question in this thread is what should be
running. If we can just chroot into the container's FS and run apt-get/yum
install our tools, and then nsenter and attach to the running process,
then huzzah: I think we have best of both worlds.

To the container makers: consider that things can and will go wrong,
and the answer may already exist as a traditional tool, and not be
"restart the container".

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread Lars Kellogg-Stedman

On Wed, Oct 15, 2014 at 07:52:56AM -0700, Vishvananda Ishaya wrote:
> There must be a standard way
> to do this stuff or people will continue to build fat containers with
> all of their pet tools inside. This means containers will just be
> another incarnation of virtualization.

I wouldn't spend time worrying about that.  "docker exec" will be the
standard way as soon as it lands in a release version, which I think
will be happening imminently with 1.3.

-- 
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/



pgp6GOmzu0krD.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread David Vossel

- Original Message -
> On Tue, 2014-10-14 at 19:52 -0400, David Vossel wrote:
> > 
> > - Original Message -
> > > Ok, why are you so down on running systemd in a container?
> > 
> > It goes against the grain.
> > 
> > From a distributed systems view, we gain quite a bit of control by
> > maintaining
> > "one service per container". Containers can be re-organised and re-purposed
> > dynamically.
> > If we have systemd trying to manage an entire stack of resources within a
> > container,
> > we lose this control.
> > 
> > From my perspective a containerized application stack needs to be managed
> > externally
> > by whatever is orchestrating the containers to begin with. When we take a
> > step back
> > and look at how we actually want to deploy containers, systemd doesn't make
> > much sense.
> > It actually limits us in the long run.
> > 
> > Also... recovery. Using systemd to manage a stack of resources within a
> > single container
> > makes it difficult for whatever is externally enforcing the availability of
> > that container
> > to detect the health of the container.  As it is now, the actual service is
> > pid 1 of a
> > container. If that service dies, the container dies. If systemd is pid 1,
> > there can
> > be all kinds of chaos occurring within the container, but the external
> > distributed
> > orchestration system won't have a clue (unless it invokes some custom
> > health monitoring
> > tools within the container itself, which will likely be the case someday.)
> 
> I don't really think this is a good argument.  If you're using docker,
> docker is the management and orchestration system for the containers.

no, docker is a local tool for pulling images and launching containers.
Docker is not the distributed resource manager in charge of overseeing
what machines launch what containers and how those containers are linked
together.

> There's no dogmatic answer to the question should you run init in the
> container.

an init daemon might make sense to put in some containers where we have
a tightly coupled resource stack. There could be a use case where it would
make more sense to put these resources in a single container.

I don't think systemd is a good solution for the init daemon though. Systemd
attempts to handle recovery itself as if it has the entire view of the 
system. With containers, the system view exists outside of the containers.
If we put an internal init daemon within the containers, that daemon needs
to escalate internal failures. The easiest way to do this is to
have init die if it encounters a resource failure (init is pid 1, pid 1 exiting
causes container to exit, container exiting gets the attention of whatever
is managing the containers)

> The reason for not running init inside a container managed by docker is
> that you want the template to be thin for ease of orchestration and
> transfer, so you want to share as much as possible with the host.  The
> more junk you put into the container, the fatter and less agile it
> becomes, so you should probably share the init system with the host in
> this paradigm.

I don't think the local init system and containers should have anything
to do with one another.  I said this in a previous reply, I'm approaching
this problem from a distributed management perspective. The host's
init daemon only has a local view of the world. 

> 
> Conversely, containers can be used to virtualize full operating systems.
> This isn't the standard way of doing docker, but LXC and OpenVZ by
> default do containers this way.  For this type of container, because you
> have a full OS running inside the container, you have to also have
> systemd (assuming it's the init system) running within the container.

sure, if you want to do this use systemd. I don't understand the use case
where this makes any sense though. For me this falls in the "yeah you can do it,
but why?" category.

-- Vossel

> 
> James
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread David Vossel



- Original Message -
> I'm not arguing that everything should be managed by one systemd, I'm just
> saying, for certain types of containers, a single docker container with
> systemd in it might be preferable to trying to slice it unnaturally into
> several containers.
> 
> Systemd has invested a lot of time/effort to be able to relaunch failed
> services, support spawning and maintaining unix sockets and services across
> them, etc, that you'd have to push out of and across docker containers. All
> of that can be done, but why reinvent the wheel? Like you said, pacemaker
> can be made to make it all work, but I have yet to see a way to deploy
> pacemaker services anywhere near as easy as systemd+yum makes it. (Thanks be
> to redhat. :)
> 
> The answer seems to be, its not "dockerish". Thats ok. I just wanted to
> understand the issue for what it is. If there is a really good reason for
> not wanting to do it, or that its just "not the way things are done". I've
> had kind of the opposite feeling regarding docker containers. Docker use to
> do very bad things when killing the container. nasty if you wanted your
> database not to go corrupt. killing pid 1 is a bit sketchy then forcing the
> container down after 10 seconds was particularly bad. having something like
> systemd in place allows the database to be notified, then shutdown properly.
> Sure you can script up enough shell to make this work, but you have to do
> some difficult code, over and over again... Docker has gotten better more
> recently but it still makes me a bit nervous using it for statefull things.
> 
> As for recovery, systemd can do the recovery too. I'd argue at this point in
> time, I'd expect systemd recovery to probably work better then some custom

yes, systemd can do recovery and that is part of the problem. From my 
perspective
there should be one resource management system. Whether that be pacemaker, 
kubernetes,
or some other distributed system, it doesn't matter.  If you are mixing systemd
with these other external distributed orchestration/management tools you have 
containers
that are silently failing/recovering without the management layer having any 
clue.

centralized recovery. There's one tool responsible for detecting and invoking 
recovery.
Everything else in the system is designed to make that possible.

If we want to put a process in the container to manage multiple services, we'd 
need
the ability to escalate failures to the distributed management tool.  Systemd 
could
work if it was given the ability to act more as a watchdog after starting 
services than
invoke recovery. If systemd could be configured to die (or potentially 
gracefully cleanup
the containers resource's before dieing) whenever a failure is detected, then 
systemd
might make sense. 

I'm approaching this from a system management point of view. Running systemd in 
your
one off container that you're managing manually does not have the same 
drawbacks.
I don't have a vendetta against systemd or anything, I just think it's a step 
backwards
to put systemd in containers. I see little value in having containers become 
lightweight
virtual machines. Containers have much more to offer.

-- Vossel



> shell scripts when it comes to doing the right thing recovering at bring up.
> The other thing is, recovery is not just about pid1 going away. often it
> sticks around and other badness is going on. Its A way to know things are
> bad, but you can't necessarily rely on it to know the container's healty.
> You need more robust checks for that.
> 
> Thanks,
> Kevin
> 
> ____________________
> From: David Vossel [dvos...@redhat.com]
> Sent: Tuesday, October 14, 2014 4:52 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> 
> - Original Message -
> > Ok, why are you so down on running systemd in a container?
> 
> It goes against the grain.
> 
> From a distributed systems view, we gain quite a bit of control by
> maintaining
> "one service per container". Containers can be re-organised and re-purposed
> dynamically.
> If we have systemd trying to manage an entire stack of resources within a
> container,
> we lose this control.
> 
> From my perspective a containerized application stack needs to be managed
> externally
> by whatever is orchestrating the containers to begin with. When we take a
> step back
> and look at how we actually want to deploy containers, systemd doesn't make
> much sense.
> It actually limits us in the long run.
> 
> Also... recovery. Using systemd to manage a stack of resources within a
> single container
>

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread Vishvananda Ishaya


On Oct 14, 2014, at 1:12 PM, Clint Byrum  wrote:

> Excerpts from Lars Kellogg-Stedman's message of 2014-10-14 12:50:48 -0700:
>> On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:
>>> I think the above strategy is spot on. Unfortunately, that's not how the
>>> Docker ecosystem works.
>> 
>> I'm not sure I agree here, but again nobody is forcing you to use this
>> tool.
>> 
>>> operating system that the image is built for. I see you didn't respond to my
>>> point that in your openstack-containers environment, you end up with Debian
>>> *and* Fedora images, since you use the "official" MySQL dockerhub image. And
>>> therefore you will end up needing to know sysadmin specifics (such as how
>>> network interfaces are set up) on multiple operating system distributions.
>> 
>> I missed that part, but ideally you don't *care* about the
>> distribution in use.  All you care about is the application.  Your
>> container environment (docker itself, or maybe a higher level
>> abstraction) sets up networking for you, and away you go.
>> 
>> If you have to perform system administration tasks inside your
>> containers, my general feeling is that something is wrong.
>> 
> 
> Speaking as a curmudgeon ops guy from "back in the day".. the reason
> I choose the OS I do is precisely because it helps me _when something
> is wrong_. And the best way an OS can help me is to provide excellent
> debugging tools, and otherwise move out of the way.
> 
> When something _is_ wrong and I want to attach GDB to mysqld in said
> container, I could build a new container with debugging tools installed,
> but that may lose the very system state that I'm debugging. So I need to
> run things inside the container like apt-get or yum to install GDB.. and
> at some point you start to realize that having a whole OS is actually a
> good thing even if it means needing to think about a few more things up
> front, such as "which OS will I use?" and "what tools do I need installed
> in my containers?"
> 
> What I mean to say is, just grabbing off the shelf has unstated
> consequences.

If this is how people are going to use and think about containers, I would
submit they are a huge waste of time. The performance value they offer is
dramatically outweighed by the flexibilty and existing tooling that exists
for virtual machines. As I state in my blog post[1] if we really want to
get value from containers, we must convert to the single application per
container view. This means having standard ways of doing the above either
on the host machine or in a debugging container that is as easy (or easier)
than the workflow you mention. There are not good ways to do this yet, and
the community hand-waves it away, saying things like, "well you could …”.
You could isn’t good enough. The result is that a lot of people that are
using containers today are doing fat containers with a full os.

Vish

[1] 
https://medium.com/@vishvananda/standard-components-not-standard-containers-c30567f23da6
> 
>>> Sure, Docker isn't any more limiting than using a VM or bare hardware, but
>>> if you use the "official" Docker images, it is more limiting, no?
>> 
>> No more so than grabbing a virtual appliance rather than building a
>> system yourself.  
>> 
>> In other words: sure, it's less flexible, but possibly it's faster to
>> get started, which is especially useful if your primary goal is not
>> "be a database administrator" but is actually "write an application
>> that uses a database backend".
>> 
>> I think there are uses cases for both "official" and customized
>> images.
>> 
> 
> In the case of Kolla, we're deploying OpenStack, not just some new
> application that uses a database backend. I think the bar is a bit
> higher for operations than end-user applications, since it sits below
> the abstractions, much closer to the metal.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread Vishvananda Ishaya


On Oct 14, 2014, at 1:21 PM, Lars Kellogg-Stedman  wrote:

> On Tue, Oct 14, 2014 at 04:06:22PM -0400, Jay Pipes wrote:
>> I understand that general feeling, but system administration tasks like
>> debugging networking issues or determining and grepping log file locations
>> or diagnosing packaging issues for OpenStack services or performing database
>> logfile maintenance and backups don't just go away because you're using
>> containers, right?
> 
> They don't go away, but they're not necessarily things that you would
> do inside your container.
> 
> Any state (e.g., database tables) that has a lifetime different from
> that of your container should be stored outside of the container
> proper.  In docker, this would be a "volume" (in a cloud environment,
> this would be something like EBS or a Cinder volume).
> 
> Ideally, your container-optimized applications logs to stdout/stderr.
> If you have multiple processes, they each run in a separate container.
> 
> Backups take advantage of the data volumes you've associated with your
> container.  E.g., spawning a new container using the docker
> "--volumes-from" option to access that data for backup purposes.
> 
> If you really need to get inside a container for diagnostic purposes,
> then you use something like "nsenter", "nsinit", or the forthcoming
> "docker exec”.

“something like” isn’t good enough here. There must be a standard way
to do this stuff or people will continue to build fat containers with
all of their pet tools inside. This means containers will just be
another incarnation of virtualization.

Vish

> 
> 
>> they very much seem to be developed from the point of view of application
>> developers, and not so much from the point of view of operators who need to
>> maintain and support those applications.
> 
> I think it's entirely accurate to say that they are
> application-centric, much like services such as Heroku, OpenShift,
> etc.
> 
> -- 
> Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
> Cloud Engineering / OpenStack  | http://blog.oddbit.com/
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns


On 10/14/2014 01:12 PM, Clint Byrum wrote:

Excerpts from Lars Kellogg-Stedman's message of 2014-10-14 12:50:48 -0700:

On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:

I think the above strategy is spot on. Unfortunately, that's not how the
Docker ecosystem works.

I'm not sure I agree here, but again nobody is forcing you to use this
tool.


operating system that the image is built for. I see you didn't respond to my
point that in your openstack-containers environment, you end up with Debian
*and* Fedora images, since you use the "official" MySQL dockerhub image. And
therefore you will end up needing to know sysadmin specifics (such as how
network interfaces are set up) on multiple operating system distributions.

I missed that part, but ideally you don't *care* about the
distribution in use.  All you care about is the application.  Your
container environment (docker itself, or maybe a higher level
abstraction) sets up networking for you, and away you go.

If you have to perform system administration tasks inside your
containers, my general feeling is that something is wrong.


Speaking as a curmudgeon ops guy from "back in the day".. the reason
I choose the OS I do is precisely because it helps me _when something
is wrong_. And the best way an OS can help me is to provide excellent
debugging tools, and otherwise move out of the way.

When something _is_ wrong and I want to attach GDB to mysqld in said
container, I could build a new container with debugging tools installed,
but that may lose the very system state that I'm debugging. So I need to
run things inside the container like apt-get or yum to install GDB.. and
at some point you start to realize that having a whole OS is actually a
good thing even if it means needing to think about a few more things up
front, such as "which OS will I use?" and "what tools do I need installed
in my containers?"

What I mean to say is, just grabbing off the shelf has unstated
consequences.
The biggest gain of containers is they are hermetically sealed. They 
turn hundreds of packages (the dependencies and OS files) into 1 
interface with one operation:

Start with a defined variatic environment.

This idea rocks if you can put up with the pain that debugging something 
that is busted is very difficult.   Accessing logs is not tidy and 
debugging in the ways that true experienced folk know how to (with gdb 
attach for example) just isn't possible.


It also requires that you rely on a completely stateless model (until 
persistent storage is implemented in k8s I guess) and completely 
idempotent model.  I really like the idea of "time to upgrade, lets roll 
new images across the cluster".  This model is very powerful but comes 
with some pain.


Regards
-steve



Sure, Docker isn't any more limiting than using a VM or bare hardware, but
if you use the "official" Docker images, it is more limiting, no?

No more so than grabbing a virtual appliance rather than building a
system yourself.

In other words: sure, it's less flexible, but possibly it's faster to
get started, which is especially useful if your primary goal is not
"be a database administrator" but is actually "write an application
that uses a database backend".

I think there are uses cases for both "official" and customized
images.


In the case of Kolla, we're deploying OpenStack, not just some new
application that uses a database backend. I think the bar is a bit
higher for operations than end-user applications, since it sits below
the abstractions, much closer to the metal.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-15 Thread James Bottomley

On Tue, 2014-10-14 at 19:52 -0400, David Vossel wrote:
> 
> - Original Message -
> > Ok, why are you so down on running systemd in a container?
> 
> It goes against the grain.
> 
> From a distributed systems view, we gain quite a bit of control by maintaining
> "one service per container". Containers can be re-organised and re-purposed 
> dynamically.
> If we have systemd trying to manage an entire stack of resources within a 
> container,
> we lose this control.
> 
> From my perspective a containerized application stack needs to be managed 
> externally
> by whatever is orchestrating the containers to begin with. When we take a 
> step back
> and look at how we actually want to deploy containers, systemd doesn't make 
> much sense.
> It actually limits us in the long run.
> 
> Also... recovery. Using systemd to manage a stack of resources within a 
> single container
> makes it difficult for whatever is externally enforcing the availability of 
> that container
> to detect the health of the container.  As it is now, the actual service is 
> pid 1 of a
> container. If that service dies, the container dies. If systemd is pid 1, 
> there can
> be all kinds of chaos occurring within the container, but the external 
> distributed
> orchestration system won't have a clue (unless it invokes some custom health 
> monitoring
> tools within the container itself, which will likely be the case someday.)

I don't really think this is a good argument.  If you're using docker,
docker is the management and orchestration system for the containers.
There's no dogmatic answer to the question should you run init in the
container.

The reason for not running init inside a container managed by docker is
that you want the template to be thin for ease of orchestration and
transfer, so you want to share as much as possible with the host.  The
more junk you put into the container, the fatter and less agile it
becomes, so you should probably share the init system with the host in
this paradigm.

Conversely, containers can be used to virtualize full operating systems.
This isn't the standard way of doing docker, but LXC and OpenVZ by
default do containers this way.  For this type of container, because you
have a full OS running inside the container, you have to also have
systemd (assuming it's the init system) running within the container.

James

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Baohua Yang

And here is the best practice for using Dockerfile.
https://docs.docker.com/articles/dockerfile_best-practices/

On Tue, Oct 14, 2014 at 11:51 AM, Angus Lees  wrote:

> I've been reading a bunch of the existing Dockerfiles, and I have two
> humble
> requests:
>
>
> 1. It would be good if the "interesting" code came from python sdist/bdists
> rather than rpms.
>
> This will make it possible to rebuild the containers using code from a
> private
> branch or even unsubmitted code, without having to go through a redhat/rpm
> release process first.
>
> I care much less about where the python dependencies come from. Pulling
> them
> from rpms rather than pip/pypi seems like a very good idea, given the
> relative
> difficulty of caching pypi content and we also pull in the required C, etc
> libraries for free.
>
>
> With this in place, I think I could drop my own containers and switch to
> reusing kolla's for building virtual testing environments.  This would
> make me
> happy.
>
>
> 2. I think we should separate out "run the server" from "do once-off
> setup".
>
> Currently the containers run a start.sh that typically sets up the
> database,
> runs the servers, creates keystone users and sets up the keystone
> catalog.  In
> something like k8s, the container will almost certainly be run multiple
> times
> in parallel and restarted numerous times, so all those other steps go
> against
> the service-oriented k8s ideal and are at-best wasted.
>
> I suggest making the container contain the deployed code and offer a few
> thin
> scripts/commands for entrypoints.  The main replicationController/pod
> _just_
> starts the server, and then we have separate pods (or perhaps even non-k8s
> container invocations) that do initial database setup/migrate, and post-
> install keystone setup.
>
> I'm open to whether we want to make these as lightweight/independent as
> possible (every daemon in an individual container), or limit it to one per
> project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one
> container).   I think the differences are run-time scalability and
> resource-
> attribution vs upfront coding effort and are not hugely significant either
> way.
>
> Post-install catalog setup we can combine into one cross-service setup like
> tripleO does[1].  Although k8s doesn't have explicit support for batch
> tasks
> currently, I'm doing the pre-install setup in restartPolicy: onFailure pods
> currently and it seems to work quite well[2].
>
> (I'm saying "post install catalog setup", but really keystone catalog can
> happen at any point pre/post aiui.)
>
> [1]
> https://github.com/openstack/tripleo-incubator/blob/master/scripts/setup-endpoints
> [2]
> https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml
>
> --
>  - Gus
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Best wishes!
Baohua
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Clint Byrum

Excerpts from Fox, Kevin M's message of 2014-10-14 17:40:16 -0700:
> I'm not arguing that everything should be managed by one systemd, I'm
> just saying, for certain types of containers, a single docker container
> with systemd in it might be preferable to trying to slice it unnaturally
> into several containers.
> 

Can you be more concrete? Most of the time things that need to be in
the same machine tend to have some kind of controller already. Meanwhile
it is worth noting that you can have one _image_, but several containers
running from that one image. So if you're trying to run a few pieces of
Neutron, for instance, you can have multiple containers each from that
one neutron image.

> Systemd has invested a lot of time/effort to be able to relaunch failed
> services, support spawning and maintaining unix sockets and services
> across them, etc, that you'd have to push out of and across docker
> containers. All of that can be done, but why reinvent the wheel? Like you
> said, pacemaker can be made to make it all work, but I have yet to see
> a way to deploy pacemaker services anywhere near as easy as systemd+yum
> makes it. (Thanks be to redhat. :)
> 

There are some of us who are rather annoyed that systemd tries to do
this in such a naive way and assumes everyone will want that kind of
management. It's the same naiveté that leads people to think if they
make their app server systemd service depend on their mysql systemd
service that this will eliminate startup problems. Once you have more
than one server, it doesn't work.

Kubernetes adds a distributed awareness of the containers that makes it
uniquely positioned to do most of those jobs much better than systemd
can.

> The answer seems to be, its not "dockerish". Thats ok. I just wanted to
> understand the issue for what it is. If there is a really good reason for
> not wanting to do it, or that its just "not the way things are done". I've
> had kind of the opposite feeling regarding docker containers. Docker use
> to do very bad things when killing the container. nasty if you wanted
> your database not to go corrupt. killing pid 1 is a bit sketchy then
> forcing the container down after 10 seconds was particularly bad. having
> something like systemd in place allows the database to be notified, then
> shutdown properly. Sure you can script up enough shell to make this work,
> but you have to do some difficult code, over and over again... Docker
> has gotten better more recently but it still makes me a bit nervous
> using it for statefull things.
> 

What I think David was saying was that the process you want to run under
systemd is the pid 1 of the container. So if killing that would be bad,
it would also be bad to stop the systemd service, which would do the
same thing: send it SIGTERM. If that causes all hell to break loose, the
stateful thing isn't worth a dime, because it isn't crash safe.

> As for recovery, systemd can do the recovery too. I'd argue at this
> point in time, I'd expect systemd recovery to probably work better
> then some custom shell scripts when it comes to doing the right thing
> recovering at bring up. The other thing is, recovery is not just about
> pid1 going away. often it sticks around and other badness is going
> on. Its A way to know things are bad, but you can't necessarily rely on
> it to know the container's healty. You need more robust checks for that.

I think one thing people like about Kubernetes is that when a container
crashes, and needs to be brought back up, it may actually be brought
up on a different, less busy, more healthy host. I could be wrong, or
that might be in the "FUTURE" section. But the point is, recovery and
start-up are not things that always want to happen on the same box.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Angus Lees

On Tue, 14 Oct 2014 07:51:54 AM Steven Dake wrote:
> Angus,
> 
> On 10/13/2014 08:51 PM, Angus Lees wrote:
> > I've been reading a bunch of the existing Dockerfiles, and I have two
> > humble requests:
> > 
> > 
> > 1. It would be good if the "interesting" code came from python
> > sdist/bdists
> > rather than rpms.
> > 
> > This will make it possible to rebuild the containers using code from a
> > private branch or even unsubmitted code, without having to go through a
> > redhat/rpm release process first.
> > 
> > I care much less about where the python dependencies come from. Pulling
> > them from rpms rather than pip/pypi seems like a very good idea, given
> > the relative difficulty of caching pypi content and we also pull in the
> > required C, etc libraries for free.
> > 
> > 
> > With this in place, I think I could drop my own containers and switch to
> > reusing kolla's for building virtual testing environments.  This would
> > make me happy.
> 
> I've captured this requirement here:
> https://blueprints.launchpad.net/kolla/+spec/run-from-master
> 
> I also believe it would be interesting to run from master or a stable
> branch for CD.  Unfortunately I'm still working on the nova-compute
> docker code, but if someone comes along and picks up that blueprint, i
> expect it will get implemented :)  Maybe that could be you.

Yeah I've already got a bunch of working containers that pull from master[1], 
but I've been thinking I should change that to use an externally supplied 
bdist.  The downside is you quickly end up wanting a docker container to build 
your deployment docker container.  I gather this is quite a common thing to 
do, but I haven't found the time to script it up yet.

[1] https://github.com/anguslees/kube-openstack/tree/master/docker

I could indeed work on this, and I guess I was gauging the level of enthusiasm 
within kolla for such a change.  I don't want to take time away from the 
alternative I have that already does what I need only to push uphill to get it 
integrated :/

> > 2. I think we should separate out "run the server" from "do once-off
> > setup".
> > 
> > Currently the containers run a start.sh that typically sets up the
> > database, runs the servers, creates keystone users and sets up the
> > keystone catalog.  In something like k8s, the container will almost
> > certainly be run multiple times in parallel and restarted numerous times,
> > so all those other steps go against the service-oriented k8s ideal and
> > are at-best wasted.
> > 
> > I suggest making the container contain the deployed code and offer a few
> > thin scripts/commands for entrypoints.  The main
> > replicationController/pod _just_ starts the server, and then we have
> > separate pods (or perhaps even non-k8s container invocations) that do
> > initial database setup/migrate, and post- install keystone setup.
> 
> The server may not start before the configuration of the server is
> complete.  I guess I don't quite understand what you indicate here when
> you say we have separate pods that do initial database setup/migrate.
> Do you mean have dependencies in some way, or for eg:
> 
> glance-registry-setup-pod.yaml - the glance registry pod descriptor
> which sets up the db and keystone
> glance-registry-pod.yaml - the glance registry pod descriptor which
> starts the application and waits for db/keystone setup
> 
> and start these two pods as part of the same selector (glance-registry)?
> 
> That idea sounds pretty appealing although probably won't be ready to go
> for milestone #1.

So the way I do it now, I have a replicationController that starts/manages 
(eg) nova-api pods[2].  I separately have a nova-db-sync pod[3] that basically 
just runs "nova-manage db sync".

I then have a simple shell script[4] that starts them all at the same time.  
The nova-api pods crash and get restarted a few times until the database has 
been appropriately configured by the nova-db-sync pod, and then they're fine 
and 
start serving.

When nova-db-sync exits successfully, the pod just sits in state terminated 
thanks to restartPolicy: onFailure.  Sometime later I can delete the 
terminated nova-db-sync pod, but it's also harmless if I just leave it or even 
if it gets occasionally re-run as part of some sort of update.


[2] 
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-api-repcon.yaml
[3] 
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml
[4] https://github.com/anguslees/kube-openstack/blob/master/kubecfg-create.sh


> > I'm open to whether we want to make these as lightweight/independent as
> > possible (every daemon in an individual container), or limit it to one per
> > project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one
> > container).   I think the differences are run-time scalability and
> > resource- attribution vs upfront coding effort and are not hugely
> > significant either way.
> > 
> > Post-install catalog set

Re: [openstack-dev] [kolla] on Dockerfile patterns

I'm not arguing that everything should be managed by one systemd, I'm just 
saying, for certain types of containers, a single docker container with systemd 
in it might be preferable to trying to slice it unnaturally into several 
containers.

Systemd has invested a lot of time/effort to be able to relaunch failed 
services, support spawning and maintaining unix sockets and services across 
them, etc, that you'd have to push out of and across docker containers. All of 
that can be done, but why reinvent the wheel? Like you said, pacemaker can be 
made to make it all work, but I have yet to see a way to deploy pacemaker 
services anywhere near as easy as systemd+yum makes it. (Thanks be to redhat. :)

The answer seems to be, its not "dockerish". Thats ok. I just wanted to 
understand the issue for what it is. If there is a really good reason for not 
wanting to do it, or that its just "not the way things are done". I've had kind 
of the opposite feeling regarding docker containers. Docker use to do very bad 
things when killing the container. nasty if you wanted your database not to go 
corrupt. killing pid 1 is a bit sketchy then forcing the container down after 
10 seconds was particularly bad. having something like systemd in place allows 
the database to be notified, then shutdown properly. Sure you can script up 
enough shell to make this work, but you have to do some difficult code, over 
and over again... Docker has gotten better more recently but it still makes me 
a bit nervous using it for statefull things.

As for recovery, systemd can do the recovery too. I'd argue at this point in 
time, I'd expect systemd recovery to probably work better then some custom 
shell scripts when it comes to doing the right thing recovering at bring up. 
The other thing is, recovery is not just about pid1 going away. often it sticks 
around and other badness is going on. Its A way to know things are bad, but you 
can't necessarily rely on it to know the container's healty. You need more 
robust checks for that.

Thanks,
Kevin


From: David Vossel [dvos...@redhat.com]
Sent: Tuesday, October 14, 2014 4:52 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

- Original Message -
> Ok, why are you so down on running systemd in a container?

It goes against the grain.

>From a distributed systems view, we gain quite a bit of control by maintaining
"one service per container". Containers can be re-organised and re-purposed 
dynamically.
If we have systemd trying to manage an entire stack of resources within a 
container,
we lose this control.

>From my perspective a containerized application stack needs to be managed 
>externally
by whatever is orchestrating the containers to begin with. When we take a step 
back
and look at how we actually want to deploy containers, systemd doesn't make 
much sense.
It actually limits us in the long run.

Also... recovery. Using systemd to manage a stack of resources within a single 
container
makes it difficult for whatever is externally enforcing the availability of 
that container
to detect the health of the container.  As it is now, the actual service is pid 
1 of a
container. If that service dies, the container dies. If systemd is pid 1, there 
can
be all kinds of chaos occurring within the container, but the external 
distributed
orchestration system won't have a clue (unless it invokes some custom health 
monitoring
tools within the container itself, which will likely be the case someday.)

-- Vossel


> Pacemaker works, but its kind of a pain to setup compared just yum installing
> a few packages and setting init to systemd. There are some benefits for
> sure, but if you have to force all the docker components onto the same
> physical machine anyway, why bother with the extra complexity?
>
> Thanks,
> Kevin
>
> 
> From: David Vossel [dvos...@redhat.com]
> Sent: Tuesday, October 14, 2014 3:14 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
>
> - Original Message -
> > Same thing works with cloud init too...
> >
> >
> > I've been waiting on systemd working inside a container for a while. it
> > seems
> > to work now.
>
> oh no...
>
> > The idea being its hard to write a shell script to get everything up and
> > running with all the interactions that may need to happen. The init
> > system's
> > already designed for that. Take a nova-compute docker container for
> > example,
> > you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
> > celiometer-agent all backed in. Writing a

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Charles Crouch



- Original Message -
> Ok, why are you so down on running systemd in a container?
> 
> Pacemaker works, but its kind of a pain to setup compared just yum installing
> a few packages and setting init to systemd. There are some benefits for
> sure, but if you have to force all the docker components onto the same
> physical machine anyway, why bother with the extra complexity?

The other benefit of running systemd in a container is that, at least
conceptually, it makes the migration path from VMs to docker containers
that much easier. If you've got an existing investment in using system
tools to run your services, it would be awfully nice to be able to keep
using that, at least in the short term, as you get your feet wet with
docker. Once you've "seen the light" and decided that pacemaker is a much
better choice than systemd then you can go port everything over.

I know that using systemd is not the "docker-way" but it is a way. 
And I think enabling people to gain some benefits of using containers without 
making them completely re-architect how they manage their services
has some value. 

To me this is akin to letting people run their legacy/non-cloudy apps on 
an IaaS. Are you going to get all the benefits that come from re-architecting 
your app to be autoscaleable and robust in the face of instance outages?
Nope. But are you going to get some benefits? Yep

Cheers
Charles

> 
> Thanks,
> Kevin
> 
> 
> From: David Vossel [dvos...@redhat.com]
> Sent: Tuesday, October 14, 2014 3:14 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> 
> - Original Message -
> > Same thing works with cloud init too...
> >
> >
> > I've been waiting on systemd working inside a container for a while. it
> > seems
> > to work now.
> 
> oh no...
> 
> > The idea being its hard to write a shell script to get everything up and
> > running with all the interactions that may need to happen. The init
> > system's
> > already designed for that. Take a nova-compute docker container for
> > example,
> > you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
> > celiometer-agent all backed in. Writing a shell script to get it all
> > started
> > and shut down properly would be really ugly.
> >
> > You could split it up into 4 containers and try and ensure they are
> > coscheduled and all the pieces are able to talk to each other, but why?
> > Putting them all in one container with systemd starting the subprocesses is
> > much easier and shouldn't have many drawbacks. The components code is
> > designed and tested assuming the pieces are all together.
> 
> What you need is a dependency model that is enforced outside of the
> containers. Something
> that manages the order containers are started/stopped/recovered in. This
> allows
> you to isolate your containers with 1 service per container, yet still
> express that
> container with service A needs to start before container with service B.
> 
> Pacemaker does this easily. There's even a docker resource-agent for
> Pacemaker now.
> https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker
> 
> -- Vossel
> 
> ps. don't run systemd in a container... If you think you should, talk to me
> first.
> 
> >
> > You can even add a ssh server in there easily too and then ansible in to do
> > whatever other stuff you want to do to the container like add other
> > monitoring and such
> >
> > Ansible or puppet or whatever should work better in this arrangement too
> > since existing code assumes you can just systemctl start foo;
> >
> > Kevin
> > 
> > From: Lars Kellogg-Stedman [l...@redhat.com]
> > Sent: Tuesday, October 14, 2014 12:10 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> >
> > On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> > > With Docker, you are limited to the operating system of whatever the
> > > image
> > > uses.
> >
> > See, that's the part I disagree with.  What I was saying about ansible
> > and puppet in my email is that I think the right thing to do is take
> > advantage of those tools:
> >
> >   FROM ubuntu
> >
> >   RUN apt-get install ansible
> >   COPY my_ansible_config.yaml /my_ansible_config.yaml
> >   RUN ansible /my_ansible_config.yaml
> >
> > Or:
> >
> &

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread David Vossel



- Original Message -
> Ok, why are you so down on running systemd in a container?

It goes against the grain.

>From a distributed systems view, we gain quite a bit of control by maintaining
"one service per container". Containers can be re-organised and re-purposed 
dynamically.
If we have systemd trying to manage an entire stack of resources within a 
container,
we lose this control.

>From my perspective a containerized application stack needs to be managed 
>externally
by whatever is orchestrating the containers to begin with. When we take a step 
back
and look at how we actually want to deploy containers, systemd doesn't make 
much sense.
It actually limits us in the long run.

Also... recovery. Using systemd to manage a stack of resources within a single 
container
makes it difficult for whatever is externally enforcing the availability of 
that container
to detect the health of the container.  As it is now, the actual service is pid 
1 of a
container. If that service dies, the container dies. If systemd is pid 1, there 
can
be all kinds of chaos occurring within the container, but the external 
distributed
orchestration system won't have a clue (unless it invokes some custom health 
monitoring
tools within the container itself, which will likely be the case someday.)

-- Vossel


> Pacemaker works, but its kind of a pain to setup compared just yum installing
> a few packages and setting init to systemd. There are some benefits for
> sure, but if you have to force all the docker components onto the same
> physical machine anyway, why bother with the extra complexity?
>
> Thanks,
> Kevin
> 
> 
> From: David Vossel [dvos...@redhat.com]
> Sent: Tuesday, October 14, 2014 3:14 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> 
> - Original Message -
> > Same thing works with cloud init too...
> >
> >
> > I've been waiting on systemd working inside a container for a while. it
> > seems
> > to work now.
> 
> oh no...
> 
> > The idea being its hard to write a shell script to get everything up and
> > running with all the interactions that may need to happen. The init
> > system's
> > already designed for that. Take a nova-compute docker container for
> > example,
> > you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
> > celiometer-agent all backed in. Writing a shell script to get it all
> > started
> > and shut down properly would be really ugly.
> >
> > You could split it up into 4 containers and try and ensure they are
> > coscheduled and all the pieces are able to talk to each other, but why?
> > Putting them all in one container with systemd starting the subprocesses is
> > much easier and shouldn't have many drawbacks. The components code is
> > designed and tested assuming the pieces are all together.
> 
> What you need is a dependency model that is enforced outside of the
> containers. Something
> that manages the order containers are started/stopped/recovered in. This
> allows
> you to isolate your containers with 1 service per container, yet still
> express that
> container with service A needs to start before container with service B.
> 
> Pacemaker does this easily. There's even a docker resource-agent for
> Pacemaker now.
> https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker
> 
> -- Vossel
> 
> ps. don't run systemd in a container... If you think you should, talk to me
> first.
> 
> >
> > You can even add a ssh server in there easily too and then ansible in to do
> > whatever other stuff you want to do to the container like add other
> > monitoring and such
> >
> > Ansible or puppet or whatever should work better in this arrangement too
> > since existing code assumes you can just systemctl start foo;
> >
> > Kevin
> > 
> > From: Lars Kellogg-Stedman [l...@redhat.com]
> > Sent: Tuesday, October 14, 2014 12:10 PM
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> >
> > On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> > > With Docker, you are limited to the operating system of whatever the
> > > image
> > > uses.
> >
> > See, that's the part I disagree with.  What I was saying about ansible
> > and puppet in my email is that I think the right thing to do is take
> > advantage of those tools:
> >
> >   FROM ubuntu
> &g

Re: [openstack-dev] [kolla] on Dockerfile patterns

Ok, why are you so down on running systemd in a container?

Pacemaker works, but its kind of a pain to setup compared just yum installing a 
few packages and setting init to systemd. There are some benefits for sure, but 
if you have to force all the docker components onto the same physical machine 
anyway, why bother with the extra complexity?

Thanks,
Kevin


From: David Vossel [dvos...@redhat.com]
Sent: Tuesday, October 14, 2014 3:14 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

- Original Message -
> Same thing works with cloud init too...
>
>
> I've been waiting on systemd working inside a container for a while. it seems
> to work now.

oh no...

> The idea being its hard to write a shell script to get everything up and
> running with all the interactions that may need to happen. The init system's
> already designed for that. Take a nova-compute docker container for example,
> you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
> celiometer-agent all backed in. Writing a shell script to get it all started
> and shut down properly would be really ugly.
>
> You could split it up into 4 containers and try and ensure they are
> coscheduled and all the pieces are able to talk to each other, but why?
> Putting them all in one container with systemd starting the subprocesses is
> much easier and shouldn't have many drawbacks. The components code is
> designed and tested assuming the pieces are all together.

What you need is a dependency model that is enforced outside of the containers. 
Something
that manages the order containers are started/stopped/recovered in. This allows
you to isolate your containers with 1 service per container, yet still express 
that
container with service A needs to start before container with service B.

Pacemaker does this easily. There's even a docker resource-agent for Pacemaker 
now.
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker

-- Vossel

ps. don't run systemd in a container... If you think you should, talk to me 
first.

>
> You can even add a ssh server in there easily too and then ansible in to do
> whatever other stuff you want to do to the container like add other
> monitoring and such
>
> Ansible or puppet or whatever should work better in this arrangement too
> since existing code assumes you can just systemctl start foo;
>
> Kevin
> 
> From: Lars Kellogg-Stedman [l...@redhat.com]
> Sent: Tuesday, October 14, 2014 12:10 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
>
> On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> > With Docker, you are limited to the operating system of whatever the image
> > uses.
>
> See, that's the part I disagree with.  What I was saying about ansible
> and puppet in my email is that I think the right thing to do is take
> advantage of those tools:
>
>   FROM ubuntu
>
>   RUN apt-get install ansible
>   COPY my_ansible_config.yaml /my_ansible_config.yaml
>   RUN ansible /my_ansible_config.yaml
>
> Or:
>
>   FROM Fedora
>
>   RUN yum install ansible
>   COPY my_ansible_config.yaml /my_ansible_config.yaml
>   RUN ansible /my_ansible_config.yaml
>
> Put the minimal instructions in your dockerfile to bootstrap your
> preferred configuration management tool. This is exactly what you
> would do when booting, say, a Nova instance into an openstack
> environment: you can provide a shell script to cloud-init that would
> install whatever packages are required to run your config management
> tool, and then run that tool.
>
> Once you have bootstrapped your cm environment you can take advantage
> of all those distribution-agnostic cm tools.
>
> In other words, using docker is no more limiting than using a vm or
> bare hardware that has been installed with your distribution of
> choice.
>
> > [1] Is there an official MySQL docker image? I found 553 Dockerhub
> > repositories for MySQL images...
>
> Yes, it's called "mysql".  It is in fact one of the official images
> highlighted on https://registry.hub.docker.com/.
>
> > >I have looked into using Puppet as part of both the build and runtime
> > >configuration process, but I haven't spent much time on it yet.
> >
> > Oh, I don't think Puppet is any better than Ansible for these things.
>
> I think it's pretty clear that I was not suggesting it was better than
> ansible.  That is hardly relevant to this discussion.  I was only
> saying that is what *I*

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread David Vossel



- Original Message -
> Same thing works with cloud init too...
> 
> 
> I've been waiting on systemd working inside a container for a while. it seems
> to work now.

oh no...

> The idea being its hard to write a shell script to get everything up and
> running with all the interactions that may need to happen. The init system's
> already designed for that. Take a nova-compute docker container for example,
> you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the
> celiometer-agent all backed in. Writing a shell script to get it all started
> and shut down properly would be really ugly.
>
> You could split it up into 4 containers and try and ensure they are
> coscheduled and all the pieces are able to talk to each other, but why?
> Putting them all in one container with systemd starting the subprocesses is
> much easier and shouldn't have many drawbacks. The components code is
> designed and tested assuming the pieces are all together.

What you need is a dependency model that is enforced outside of the containers. 
Something
that manages the order containers are started/stopped/recovered in. This allows
you to isolate your containers with 1 service per container, yet still express 
that
container with service A needs to start before container with service B.

Pacemaker does this easily. There's even a docker resource-agent for Pacemaker 
now.
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker

-- Vossel

ps. don't run systemd in a container... If you think you should, talk to me 
first.

> 
> You can even add a ssh server in there easily too and then ansible in to do
> whatever other stuff you want to do to the container like add other
> monitoring and such
> 
> Ansible or puppet or whatever should work better in this arrangement too
> since existing code assumes you can just systemctl start foo;
> 
> Kevin
> 
> From: Lars Kellogg-Stedman [l...@redhat.com]
> Sent: Tuesday, October 14, 2014 12:10 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns
> 
> On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> > With Docker, you are limited to the operating system of whatever the image
> > uses.
> 
> See, that's the part I disagree with.  What I was saying about ansible
> and puppet in my email is that I think the right thing to do is take
> advantage of those tools:
> 
>   FROM ubuntu
> 
>   RUN apt-get install ansible
>   COPY my_ansible_config.yaml /my_ansible_config.yaml
>   RUN ansible /my_ansible_config.yaml
> 
> Or:
> 
>   FROM Fedora
> 
>   RUN yum install ansible
>   COPY my_ansible_config.yaml /my_ansible_config.yaml
>   RUN ansible /my_ansible_config.yaml
> 
> Put the minimal instructions in your dockerfile to bootstrap your
> preferred configuration management tool. This is exactly what you
> would do when booting, say, a Nova instance into an openstack
> environment: you can provide a shell script to cloud-init that would
> install whatever packages are required to run your config management
> tool, and then run that tool.
> 
> Once you have bootstrapped your cm environment you can take advantage
> of all those distribution-agnostic cm tools.
> 
> In other words, using docker is no more limiting than using a vm or
> bare hardware that has been installed with your distribution of
> choice.
> 
> > [1] Is there an official MySQL docker image? I found 553 Dockerhub
> > repositories for MySQL images...
> 
> Yes, it's called "mysql".  It is in fact one of the official images
> highlighted on https://registry.hub.docker.com/.
> 
> > >I have looked into using Puppet as part of both the build and runtime
> > >configuration process, but I haven't spent much time on it yet.
> >
> > Oh, I don't think Puppet is any better than Ansible for these things.
> 
> I think it's pretty clear that I was not suggesting it was better than
> ansible.  That is hardly relevant to this discussion.  I was only
> saying that is what *I* have looked at, and I was agreeing that *any*
> configuration management system is probably better than writing shells
> cript.
> 
> > How would I go about essentially transferring the ownership of the RPC
> > exchanges that the original nova-conductor container managed to the new
> > nova-conductor container? Would it be as simple as shutting down the old
> > container and starting up the new nova-conductor container using things
> > like
> > --link rabbitmq:rabbitmq in the startup docker line?
> 
> I think that you would not necessa

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Ian Main

Angus Lees wrote:
> I've been reading a bunch of the existing Dockerfiles, and I have two humble 
> requests:
> 
> 
> 1. It would be good if the "interesting" code came from python sdist/bdists 
> rather than rpms.
> 
> This will make it possible to rebuild the containers using code from a 
> private 
> branch or even unsubmitted code, without having to go through a redhat/rpm 
> release process first.
> 
> I care much less about where the python dependencies come from. Pulling them 
> from rpms rather than pip/pypi seems like a very good idea, given the 
> relative 
> difficulty of caching pypi content and we also pull in the required C, etc 
> libraries for free.
> 
> 
> With this in place, I think I could drop my own containers and switch to 
> reusing kolla's for building virtual testing environments.  This would make 
> me 
> happy.
> 
> 
> 2. I think we should separate out "run the server" from "do once-off setup".
> 
> Currently the containers run a start.sh that typically sets up the database, 
> runs the servers, creates keystone users and sets up the keystone catalog.  
> In 
> something like k8s, the container will almost certainly be run multiple times 
> in parallel and restarted numerous times, so all those other steps go against 
> the service-oriented k8s ideal and are at-best wasted.
> 
> I suggest making the container contain the deployed code and offer a few thin 
> scripts/commands for entrypoints.  The main replicationController/pod _just_ 
> starts the server, and then we have separate pods (or perhaps even non-k8s 
> container invocations) that do initial database setup/migrate, and post-
> install keystone setup.
> 
> I'm open to whether we want to make these as lightweight/independent as 
> possible (every daemon in an individual container), or limit it to one per 
> project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one 
> container).   I think the differences are run-time scalability and resource-
> attribution vs upfront coding effort and are not hugely significant either 
> way.
> 
> Post-install catalog setup we can combine into one cross-service setup like 
> tripleO does[1].  Although k8s doesn't have explicit support for batch tasks 
> currently, I'm doing the pre-install setup in restartPolicy: onFailure pods 
> currently and it seems to work quite well[2].
> 
> (I'm saying "post install catalog setup", but really keystone catalog can 
> happen at any point pre/post aiui.)
> 
> [1] 
> https://github.com/openstack/tripleo-incubator/blob/master/scripts/setup-endpoints
> [2] 
> https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml
> 
> -- 
>  - Gus

One thing I've learned is to not perform software updates within a container.
A number of the containers I've seen do software updates on startup but I've
seen this break dependencies in containers a few times making them unusable.
This detracts from the ability to have a completely controlled environment
within a container with proven software versions that play nicely together.

Ian

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, Oct 14, 2014 at 04:06:22PM -0400, Jay Pipes wrote:
> I understand that general feeling, but system administration tasks like
> debugging networking issues or determining and grepping log file locations
> or diagnosing packaging issues for OpenStack services or performing database
> logfile maintenance and backups don't just go away because you're using
> containers, right?

They don't go away, but they're not necessarily things that you would
do inside your container.

Any state (e.g., database tables) that has a lifetime different from
that of your container should be stored outside of the container
proper.  In docker, this would be a "volume" (in a cloud environment,
this would be something like EBS or a Cinder volume).

Ideally, your container-optimized applications logs to stdout/stderr.
If you have multiple processes, they each run in a separate container.

Backups take advantage of the data volumes you've associated with your
container.  E.g., spawning a new container using the docker
"--volumes-from" option to access that data for backup purposes.

If you really need to get inside a container for diagnostic purposes,
then you use something like "nsenter", "nsinit", or the forthcoming
"docker exec".

> they very much seem to be developed from the point of view of application
> developers, and not so much from the point of view of operators who need to
> maintain and support those applications.

I think it's entirely accurate to say that they are
application-centric, much like services such as Heroku, OpenShift,
etc.

-- 
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/

pgp29hOhB_K2x.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Clint Byrum

Excerpts from Lars Kellogg-Stedman's message of 2014-10-14 12:50:48 -0700:
> On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:
> > I think the above strategy is spot on. Unfortunately, that's not how the
> > Docker ecosystem works.
> 
> I'm not sure I agree here, but again nobody is forcing you to use this
> tool.
> 
> > operating system that the image is built for. I see you didn't respond to my
> > point that in your openstack-containers environment, you end up with Debian
> > *and* Fedora images, since you use the "official" MySQL dockerhub image. And
> > therefore you will end up needing to know sysadmin specifics (such as how
> > network interfaces are set up) on multiple operating system distributions.
> 
> I missed that part, but ideally you don't *care* about the
> distribution in use.  All you care about is the application.  Your
> container environment (docker itself, or maybe a higher level
> abstraction) sets up networking for you, and away you go.
> 
> If you have to perform system administration tasks inside your
> containers, my general feeling is that something is wrong.
> 

Speaking as a curmudgeon ops guy from "back in the day".. the reason
I choose the OS I do is precisely because it helps me _when something
is wrong_. And the best way an OS can help me is to provide excellent
debugging tools, and otherwise move out of the way.

When something _is_ wrong and I want to attach GDB to mysqld in said
container, I could build a new container with debugging tools installed,
but that may lose the very system state that I'm debugging. So I need to
run things inside the container like apt-get or yum to install GDB.. and
at some point you start to realize that having a whole OS is actually a
good thing even if it means needing to think about a few more things up
front, such as "which OS will I use?" and "what tools do I need installed
in my containers?"

What I mean to say is, just grabbing off the shelf has unstated
consequences.

> > Sure, Docker isn't any more limiting than using a VM or bare hardware, but
> > if you use the "official" Docker images, it is more limiting, no?
> 
> No more so than grabbing a virtual appliance rather than building a
> system yourself.  
> 
> In other words: sure, it's less flexible, but possibly it's faster to
> get started, which is especially useful if your primary goal is not
> "be a database administrator" but is actually "write an application
> that uses a database backend".
> 
> I think there are uses cases for both "official" and customized
> images.
> 

In the case of Kolla, we're deploying OpenStack, not just some new
application that uses a database backend. I think the bar is a bit
higher for operations than end-user applications, since it sits below
the abstractions, much closer to the metal.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns


On 10/14/2014 03:50 PM, Lars Kellogg-Stedman wrote:

On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:

I think the above strategy is spot on. Unfortunately, that's not how the
Docker ecosystem works.


I'm not sure I agree here, but again nobody is forcing you to use this
tool.


I know that. I'm not slamming Docker. I'm trying to better understand 
the Docker and Kubernetes systems.



operating system that the image is built for. I see you didn't respond to my
point that in your openstack-containers environment, you end up with Debian
*and* Fedora images, since you use the "official" MySQL dockerhub image. And
therefore you will end up needing to know sysadmin specifics (such as how
network interfaces are set up) on multiple operating system distributions.


I missed that part, but ideally you don't *care* about the
distribution in use.  All you care about is the application.  Your
container environment (docker itself, or maybe a higher level
abstraction) sets up networking for you, and away you go.

If you have to perform system administration tasks inside your
containers, my general feeling is that something is wrong.


I understand that general feeling, but system administration tasks like 
debugging networking issues or determining and grepping log file 
locations or diagnosing packaging issues for OpenStack services or 
performing database logfile maintenance and backups don't just go away 
because you're using containers, right? If there are multiple operating 
systems in use in these containers, it makes the life of an admin more 
cumbersome, IMO.


I guess all I'm saying is that, from what I can tell, Docker and 
Kubernetes and all the application/service-centric worldviews are cool 
and all, but they very much seem to be developed from the point of view 
of application developers, and not so much from the point of view of 
operators who need to maintain and support those applications.


I'm still super-interested in the topic, just thought I'd point that out.

Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:
> I think the above strategy is spot on. Unfortunately, that's not how the
> Docker ecosystem works.

I'm not sure I agree here, but again nobody is forcing you to use this
tool.

> operating system that the image is built for. I see you didn't respond to my
> point that in your openstack-containers environment, you end up with Debian
> *and* Fedora images, since you use the "official" MySQL dockerhub image. And
> therefore you will end up needing to know sysadmin specifics (such as how
> network interfaces are set up) on multiple operating system distributions.

I missed that part, but ideally you don't *care* about the
distribution in use.  All you care about is the application.  Your
container environment (docker itself, or maybe a higher level
abstraction) sets up networking for you, and away you go.

If you have to perform system administration tasks inside your
containers, my general feeling is that something is wrong.

> Sure, Docker isn't any more limiting than using a VM or bare hardware, but
> if you use the "official" Docker images, it is more limiting, no?

No more so than grabbing a virtual appliance rather than building a
system yourself.  

In other words: sure, it's less flexible, but possibly it's faster to
get started, which is especially useful if your primary goal is not
"be a database administrator" but is actually "write an application
that uses a database backend".

I think there are uses cases for both "official" and customized
images.

-- 
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/

pgpKaSDODdjVy.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

Same thing works with cloud init too...

I've been waiting on systemd working inside a container for a while. it seems 
to work now.

The idea being its hard to write a shell script to get everything up and 
running with all the interactions that may need to happen. The init system's 
already designed for that. Take a nova-compute docker container for example, 
you probably need nova-compute, libvirt, neutron-openvswitch-agent, and the 
celiometer-agent all backed in. Writing a shell script to get it all started 
and shut down properly would be really ugly.

You could split it up into 4 containers and try and ensure they are coscheduled 
and all the pieces are able to talk to each other, but why? Putting them all in 
one container with systemd starting the subprocesses is much easier and 
shouldn't have many drawbacks. The components code is designed and tested 
assuming the pieces are all together.

You can even add a ssh server in there easily too and then ansible in to do 
whatever other stuff you want to do to the container like add other monitoring 
and such

Ansible or puppet or whatever should work better in this arrangement too since 
existing code assumes you can just systemctl start foo;

Kevin

From: Lars Kellogg-Stedman [l...@redhat.com]
Sent: Tuesday, October 14, 2014 12:10 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> With Docker, you are limited to the operating system of whatever the image
> uses.

See, that's the part I disagree with.  What I was saying about ansible
and puppet in my email is that I think the right thing to do is take
advantage of those tools:

  FROM ubuntu

  RUN apt-get install ansible
  COPY my_ansible_config.yaml /my_ansible_config.yaml
  RUN ansible /my_ansible_config.yaml

Or:

  FROM Fedora

  RUN yum install ansible
  COPY my_ansible_config.yaml /my_ansible_config.yaml
  RUN ansible /my_ansible_config.yaml

Put the minimal instructions in your dockerfile to bootstrap your
preferred configuration management tool. This is exactly what you
would do when booting, say, a Nova instance into an openstack
environment: you can provide a shell script to cloud-init that would
install whatever packages are required to run your config management
tool, and then run that tool.

Once you have bootstrapped your cm environment you can take advantage
of all those distribution-agnostic cm tools.

In other words, using docker is no more limiting than using a vm or
bare hardware that has been installed with your distribution of
choice.

> [1] Is there an official MySQL docker image? I found 553 Dockerhub
> repositories for MySQL images...

Yes, it's called "mysql".  It is in fact one of the official images
highlighted on https://registry.hub.docker.com/.

> >I have looked into using Puppet as part of both the build and runtime
> >configuration process, but I haven't spent much time on it yet.
>
> Oh, I don't think Puppet is any better than Ansible for these things.

I think it's pretty clear that I was not suggesting it was better than
ansible.  That is hardly relevant to this discussion.  I was only
saying that is what *I* have looked at, and I was agreeing that *any*
configuration management system is probably better than writing shells
cript.

> How would I go about essentially transferring the ownership of the RPC
> exchanges that the original nova-conductor container managed to the new
> nova-conductor container? Would it be as simple as shutting down the old
> container and starting up the new nova-conductor container using things like
> --link rabbitmq:rabbitmq in the startup docker line?

I think that you would not necessarily rely on --link for this sort of
thing.  Under kubernetes, you would use a "service" definition, in
which kubernetes maintains a proxy that directs traffic to the
appropriate place as containers are created and destroyed.

Outside of kubernetes, you would use some other service discovery
mechanism; there are many available (etcd, consul, serf, etc).

But this isn't particularly a docker problem.  This is the same
problem you would face running the same software on top of a cloud
environment in which you cannot predict things like ip addresses a
priori.

--
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Chris Dent


On Tue, 14 Oct 2014, Jay Pipes wrote:

This means you now have to know the system administrative comments and setup 
for two operating systems ... or go find a Fedora20 image for mysql 
somewhere.


For sake of conversation and devil's advocacy let me ask, in
response to this paragraph, "why [do you] have to know [...]?"

If you've got a docker container that is running mysql, IME that's _all_
it should be doing and any (post-setup) management you are doing of it
should be happening by creating a new container and trashing the one
that is not right, not manipulating the existing container. The
operating system should be as close to invisible as you can get it.

Everything in the Dockerfile should be about getting the service to
a fully installe and configured state and it should close with the
one command or entrypoint it is going to run/use, in the mysql case
that's one of the various ways to get mysqld happening.

If the goal is to use Docker and to have automation I think it would
be better to automate the generation of suitably
layered/hierarchical Dockerfiles (using whatever tool of choice),
not having Dockerfiles which then use config tools to populate out
the image. If such config tools are necessary in the container it is
pretty much guaranteed that the container is being overburdened with
too many responsibilities.

[1] Is there an official MySQL docker image? I found 553 Dockerhub 
repositories for MySQL images...


  https://github.com/docker-library/mysql

Docker-library appears to be the place for official things.

By layered model, are you referring to the bottom layer being the Docker 
image and then upper layers being stuff managed by a configuration management 
system?


I assume it's the layering afforded by union file systems: Makes
building images based on other images cheap and fast. The cheapness
and fastness is one of the reasons why expressive Dockerfiles[1] are
important: each line is a separate checkpointed image.


[1] That is, Dockerfiles which do the install and config work rather
than calling on other stuff to do the work.

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns


On 10/14/2014 03:10 PM, Lars Kellogg-Stedman wrote:

On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:

With Docker, you are limited to the operating system of whatever the image
uses.


See, that's the part I disagree with.  What I was saying about ansible
and puppet in my email is that I think the right thing to do is take
advantage of those tools:

   FROM ubuntu

   RUN apt-get install ansible
   COPY my_ansible_config.yaml /my_ansible_config.yaml
   RUN ansible /my_ansible_config.yaml

Or:

   FROM Fedora

   RUN yum install ansible
   COPY my_ansible_config.yaml /my_ansible_config.yaml
   RUN ansible /my_ansible_config.yaml

Put the minimal instructions in your dockerfile to bootstrap your
preferred configuration management tool. This is exactly what you
would do when booting, say, a Nova instance into an openstack
environment: you can provide a shell script to cloud-init that would
install whatever packages are required to run your config management
tool, and then run that tool.


I think the above strategy is spot on. Unfortunately, that's not how the 
Docker ecosystem works. Everything is based on images, and you get the 
operating system that the image is built for. I see you didn't respond 
to my point that in your openstack-containers environment, you end up 
with Debian *and* Fedora images, since you use the "official" MySQL 
dockerhub image. And therefore you will end up needing to know sysadmin 
specifics (such as how network interfaces are set up) on multiple 
operating system distributions.



Once you have bootstrapped your cm environment you can take advantage
of all those distribution-agnostic cm tools.

In other words, using docker is no more limiting than using a vm or
bare hardware that has been installed with your distribution of
choice.


Sure, Docker isn't any more limiting than using a VM or bare hardware, 
but if you use the "official" Docker images, it is more limiting, no?



How would I go about essentially transferring the ownership of the RPC
exchanges that the original nova-conductor container managed to the new
nova-conductor container? Would it be as simple as shutting down the old
container and starting up the new nova-conductor container using things like
--link rabbitmq:rabbitmq in the startup docker line?


I think that you would not necessarily rely on --link for this sort of
thing.  Under kubernetes, you would use a "service" definition, in
which kubernetes maintains a proxy that directs traffic to the
appropriate place as containers are created and destroyed.

Outside of kubernetes, you would use some other service discovery
mechanism; there are many available (etcd, consul, serf, etc).

But this isn't particularly a docker problem.  This is the same
problem you would face running the same software on top of a cloud
environment in which you cannot predict things like ip addresses a
priori.


Gotcha. I'm still reading through the Kubernetes docs to better 
understand the role that it plays in relation to Docker. Hopefully I'll 
have a better grip on this stuff by summit time :) Thanks again for your 
very helpful replies.


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, Oct 14, 2014 at 02:45:30PM -0400, Jay Pipes wrote:
> With Docker, you are limited to the operating system of whatever the image
> uses.

See, that's the part I disagree with.  What I was saying about ansible
and puppet in my email is that I think the right thing to do is take
advantage of those tools:

  FROM ubuntu

  RUN apt-get install ansible
  COPY my_ansible_config.yaml /my_ansible_config.yaml
  RUN ansible /my_ansible_config.yaml

Or:

  FROM Fedora

  RUN yum install ansible
  COPY my_ansible_config.yaml /my_ansible_config.yaml
  RUN ansible /my_ansible_config.yaml

Put the minimal instructions in your dockerfile to bootstrap your
preferred configuration management tool. This is exactly what you
would do when booting, say, a Nova instance into an openstack
environment: you can provide a shell script to cloud-init that would
install whatever packages are required to run your config management
tool, and then run that tool.

Once you have bootstrapped your cm environment you can take advantage
of all those distribution-agnostic cm tools.

In other words, using docker is no more limiting than using a vm or
bare hardware that has been installed with your distribution of
choice.

> [1] Is there an official MySQL docker image? I found 553 Dockerhub
> repositories for MySQL images...

Yes, it's called "mysql".  It is in fact one of the official images
highlighted on https://registry.hub.docker.com/.

> >I have looked into using Puppet as part of both the build and runtime
> >configuration process, but I haven't spent much time on it yet.
> 
> Oh, I don't think Puppet is any better than Ansible for these things.

I think it's pretty clear that I was not suggesting it was better than
ansible.  That is hardly relevant to this discussion.  I was only
saying that is what *I* have looked at, and I was agreeing that *any*
configuration management system is probably better than writing shells
cript.

> How would I go about essentially transferring the ownership of the RPC
> exchanges that the original nova-conductor container managed to the new
> nova-conductor container? Would it be as simple as shutting down the old
> container and starting up the new nova-conductor container using things like
> --link rabbitmq:rabbitmq in the startup docker line?

I think that you would not necessarily rely on --link for this sort of
thing.  Under kubernetes, you would use a "service" definition, in
which kubernetes maintains a proxy that directs traffic to the
appropriate place as containers are created and destroyed.

Outside of kubernetes, you would use some other service discovery
mechanism; there are many available (etcd, consul, serf, etc).

But this isn't particularly a docker problem.  This is the same
problem you would face running the same software on top of a cloud
environment in which you cannot predict things like ip addresses a
priori.

-- 
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/

pgp9SM_Y1OfTe.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns


On 10/14/2014 01:28 PM, Lars Kellogg-Stedman wrote:

On Tue, Oct 14, 2014 at 12:33:42PM -0400, Jay Pipes wrote:

Can I use your Dockerfiles to build Ubuntu/Debian images instead of only
Fedora images?


Not easily, no.


Seems to me that the image-based Docker system makes the
resulting container quite brittle -- since a) you can't use configuration
management systems like Ansible to choose which operating system or package
management tools you wish to use...


While that's true, it seems like a non-goal.  You're not starting with
a virtual machine and a blank disk here, you're starting from an
existing filesystem.

I'm not sure I understand your use case enough to give you a more
useful reply.


Sorry, I'm trying hard to describe some of this. I have limited 
vocabulary to use in this new space :)


I guess what I am saying is that there is a wealth of existing 
configuration management provenance that installs and configures 
application packages. These configuration management modules/tools are 
written to entirely take away the multi-operating-system, 
multi-package-manager problems.


Instead of having two Dockerfiles, one that only works on Fedora that 
does something like:


FROM fedora20
RUN yum -y install python-pbr

and one that only works on Debian:

FROM debian:wheezy
RUN apt-get install -y python-pbr

Configuration management tools like Ansible already work cross-operating 
system, and allow you to express what gets installed regardless of the 
operating system of the disk image:


tasks:
  - name: install PBR Debian
apt: name=python-pbr state=present
when: ansible_os_family == "Debian"
  - name: install PBR RH
yum: name=python-pbr state=present
when: ansible_os_family == "RedHat"

Heck, in Chef, you wouldn't even need the when: switch logic, since Chef 
knows which package management system to use depending on the operating 
system.


With Docker, you are limited to the operating system of whatever the 
image uses.


This means, for things like your openstack-containers Ansible+Docker 
environment (which is wicked cool, BTW), you have containers running 
Fedora20 for everything except MySQL, which due to the fact that you are 
using the "official" [1] MySQL image on dockerhub, is only a 
Debian:Wheezy image.


This means you now have to know the system administrative comments and 
setup for two operating systems ... or go find a Fedora20 image for 
mysql somewhere.


It just seems to me that Docker is re-inventing a whole bunch of stuff 
that configuration management tools like Ansible, Puppet, Chef, and 
Saltstack have gotten good at over the years.


[1] Is there an official MySQL docker image? I found 553 Dockerhub 
repositories for MySQL images...



So... what am I missing with this? What makes Docker images more ideal than
straight up LXC containers and using Ansible to control upgrades/changes to
configuration of the software on those containers?


I think that in general that Docker images are more share-able, and
the layered model makes building components on top of a base image
both easy and reasonably efficient in terms of time and storage.


By layered model, are you referring to the bottom layer being the Docker 
image and then upper layers being stuff managed by a configuration 
management system?



I think that Ansible makes a great tool for managing configuration
inside Docker containers, and you could easily use it as part of the
image build process.  Right now, people using Docker are basically
writing shell scripts to perform system configuration, which is like a
20 year step back in time.


Right, I've noticed that :)

>  Using a more structured mechanism for

doing this is a great idea, and one that lots of people are pursuing.
I have looked into using Puppet as part of both the build and runtime
configuration process, but I haven't spent much time on it yet.


Oh, I don't think Puppet is any better than Ansible for these things.


A key goal for Docker images is generally that images are "immutable",
or at least "stateless".  You don't "yum upgrade" or "apt-get upgrade"
in a container; you generate a new image with new packages/code/etc.
This makes it trivial to revert to a previous version of a deployment,
and clearly separates the "build the image" process from the "run the
application" process.


OK, so bear with me more on this, please... :)

Let's say I build Docker images for, say, my nova-conductor container, 
and I build the image by doing a devstack-style "install from git repos" 
method. Then, I build another nova-conductor container from a newer 
revision in source control.


How would I go about essentially transferring the ownership of the RPC 
exchanges that the original nova-conductor container managed to the new 
nova-conductor container? Would it be as simple as shutting down the old 
container and starting up the new nova-conductor container using things 
like --link rabbitmq:rabbitmq in the startup docker line?


Genuinely curious and

Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, Oct 14, 2014 at 12:33:42PM -0400, Jay Pipes wrote:
> Can I use your Dockerfiles to build Ubuntu/Debian images instead of only
> Fedora images?

Not easily, no.

> Seems to me that the image-based Docker system makes the
> resulting container quite brittle -- since a) you can't use configuration
> management systems like Ansible to choose which operating system or package
> management tools you wish to use...

While that's true, it seems like a non-goal.  You're not starting with
a virtual machine and a blank disk here, you're starting from an
existing filesystem.

I'm not sure I understand your use case enough to give you a more
useful reply.

> So... what am I missing with this? What makes Docker images more ideal than
> straight up LXC containers and using Ansible to control upgrades/changes to
> configuration of the software on those containers?

I think that in general that Docker images are more share-able, and
the layered model makes building components on top of a base image
both easy and reasonably efficient in terms of time and storage.

I think that Ansible makes a great tool for managing configuration
inside Docker containers, and you could easily use it as part of the
image build process.  Right now, people using Docker are basically
writing shell scripts to perform system configuration, which is like a
20 year step back in time.  Using a more structured mechanism for
doing this is a great idea, and one that lots of people are pursuing.
I have looked into using Puppet as part of both the build and runtime
configuration process, but I haven't spent much time on it yet.

A key goal for Docker images is generally that images are "immutable",
or at least "stateless".  You don't "yum upgrade" or "apt-get upgrade"
in a container; you generate a new image with new packages/code/etc.
This makes it trivial to revert to a previous version of a deployment,
and clearly separates the "build the image" process from the "run the
application" process.

I like this model.

-- 
Lars Kellogg-Stedman  | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack  | http://blog.oddbit.com/

pgpF7zc0dsOpi.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns


On 10/14/2014 10:49 AM, Lars Kellogg-Stedman wrote:

On Tue, Oct 14, 2014 at 02:51:15PM +1100, Angus Lees wrote:

1. It would be good if the "interesting" code came from python sdist/bdists
rather than rpms.


I agree in principal, although starting from packages right now lets
us ignore a whole host of issues.  Possibly we'll hit that change down
the road.


2. I think we should separate out "run the server" from "do once-off setup".

Currently the containers run a start.sh that typically sets up the database,
runs the servers, creates keystone users and sets up the keystone catalog.  In
something like k8s, the container will almost certainly be run multiple times
in parallel and restarted numerous times, so all those other steps go against
the service-oriented k8s ideal and are at-best wasted.


All the existing containers [*] are designed to be idempotent, which I
think is not a bad model.  Even if we move initial configuration out
of the service containers I think that is a goal we want to preserve.

I pursued exactly the model you suggest on my own when working on an
ansible-driven workflow for setting things up:

   https://github.com/larsks/openstack-containers

Ansible made it easy to support one-off "batch" containers which, as
you say, aren't exactly supported in Kubernetes.  I like your
(ab?)use of restartPolicy; I think that's worth pursuing.


I agree that Ansible makes it easy to support one-off batch containers. 
Ansible rocks.


Which brings me to my question (admittedly, I am a Docker n00b, so 
please forgive me for the question)...


Can I use your Dockerfiles to build Ubuntu/Debian images instead of only 
Fedora images? Seems to me that the image-based Docker system makes the 
resulting container quite brittle -- since a) you can't use 
configuration management systems like Ansible to choose which operating 
system or package management tools you wish to use, and b) any time you 
may a change to the image, you need to regenerate the image from a new 
Dockerfile and, presumably, start a new container with the new image, 
shut down the old container, and then change all the other containers 
that were linked with the old container to point to the new one. All of 
this would be a simple apt-get upgrade -y for things like security 
updates, which for the most part, wouldn't require any such container 
rebuilds.


So... what am I missing with this? What makes Docker images more ideal 
than straight up LXC containers and using Ansible to control 
upgrades/changes to configuration of the software on those containers?


Again, sorry for the n00b question!

Best,
-jay


[*] That work, which includes rabbitmq, mariadb, keystone, and glance.


I'm open to whether we want to make these as lightweight/independent as
possible (every daemon in an individual container), or limit it to one per
project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one
container).


My goal is one-service-per-container, because that generally makes the
question of process supervision and log collection a *host* problem
rather than a *container* problem. It also makes it easier to scale an
individual service, if that becomes necessary.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

Absolutely this needs splitting out. I ran into an issue a few years ago with 
this antipattern with the mythtv folks. The myth client on my laptop got 
upgraded and it was overly helpful in that it connected directly to the 
database and upgraded the schema for me, breaking the server, and all the other 
clients, and forcing an unplanned upgrade of everything else. :/

From: Chris Dent [chd...@redhat.com]
Sent: Tuesday, October 14, 2014 3:54 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [kolla] on Dockerfile patterns

On Tue, 14 Oct 2014, Angus Lees wrote:

> 2. I think we should separate out "run the server" from "do once-off setup".

Yes! Otherwise it feels like the entire point of using containers
and dockerfiles is rather lost.

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

2014-10-14 Thread Steven Dake

Angus,

On 10/13/2014 08:51 PM, Angus Lees wrote:

I've been reading a bunch of the existing Dockerfiles, and I have two humble
requests:

1. It would be good if the "interesting" code came from python sdist/bdists
rather than rpms.

This will make it possible to rebuild the containers using code from a private
branch or even unsubmitted code, without having to go through a redhat/rpm
release process first.

I care much less about where the python dependencies come from. Pulling them
from rpms rather than pip/pypi seems like a very good idea, given the relative
difficulty of caching pypi content and we also pull in the required C, etc
libraries for free.

With this in place, I think I could drop my own containers and switch to
reusing kolla's for building virtual testing environments. This would make me
happy.

I've captured this requirement here:
https://blueprints.launchpad.net/kolla/+spec/run-from-master

2. I think we should separate out "run the server" from "do once-off setup".

Currently the containers run a start.sh that typically sets up the database,
runs the servers, creates keystone users and sets up the keystone catalog. In
something like k8s, the container will almost certainly be run multiple times
in parallel and restarted numerous times, so all those other steps go against
the service-oriented k8s ideal and are at-best wasted.

I suggest making the container contain the deployed code and offer a few thin
scripts/commands for entrypoints. The main replicationController/pod _just_
starts the server, and then we have separate pods (or perhaps even non-k8s
container invocations) that do initial database setup/migrate, and post-
install keystone setup.
The server may not start before the configuration of the server is
complete. I guess I don't quite understand what you indicate here when
you say we have separate pods that do initial database setup/migrate.
Do you mean have dependencies in some way, or for eg:

and start these two pods as part of the same selector (glance-registry)?

That idea sounds pretty appealing although probably won't be ready to go
for milestone #1.

Regards,
-steve

I'm open to whether we want to make these as lightweight/independent as
possible (every daemon in an individual container), or limit it to one per
project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one
container). I think the differences are run-time scalability and resource-
attribution vs upfront coding effort and are not hugely significant either way.

Post-install catalog setup we can combine into one cross-service setup like
tripleO does[1]. Although k8s doesn't have explicit support for batch tasks
currently, I'm doing the pre-install setup in restartPolicy: onFailure pods
currently and it seems to work quite well[2].

(I'm saying "post install catalog setup", but really keystone catalog can
happen at any point pre/post aiui.)

[1]
https://github.com/openstack/tripleo-incubator/blob/master/scripts/setup-endpoints
[2]
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns