Re: [openstack-dev] [magnum]storage for docker-bootstrap

[王华]

Adrian,

flannel_network_cidr and flannel_network_subnetlen are two parameters
needed for flannel. flannel_network_cidr is the network range for flannel
overlay network. flannel_network is the size of subnet assigned to each
minion. When flannel starts, it needs the two parameters. Flannel will
allocate a subnet from flannel_network_cidr for each minion. THe subnets on
different minions are different. The data of flannel is stored in etcd. BIP
is equal to the subnet created by flannel. MTU depends on whether we use
vxlan in flannel.

If we use one docker daemon, we need to start the docker daemon without BIP
first, then run flannel and etcd to generate BIP. After that, we need to
kill the previous docker daemon and start a new docker daemon with BIP,
then run etcd and flannel on it.

Regards,
Wanghua

On Sat, Dec 19, 2015 at 2:19 AM, Adrian Otto <adrian.o...@rackspace.com>
wrote:

> Wanghua,
>
> I see. The circular dependency you described does sound like a formidable
> challenge. Having multiple docker daemons violates the principle of least
> surprise. I worry that when it comes time to perform troubleshooting, an
> engineer would be surprised to find multiple dockers running at the same
> time within the same compute instance.
>
> Perhaps there is a way to generate the BIP and MTU before the docker
> daemon is started, then use those while starting docker, and start both
> flannel and etcd containers so all containers on the instance can share a
> single docker daemon? Would that work at all? I guess I’d need a better
> understanding of exactly how the BIP and MTU are generated before judging
> if this is a good idea.
>
> Adrian
>
> On Dec 16, 2015, at 11:40 PM, 王华 <wanghua.hum...@gmail.com> wrote:
>
> Adrian,
>
> When the docker daemon starts, it needs to know the bip and mtu which are
> generated by flannel. So flannel and etcd should start before docker
> daemon, but if flannel and etcd run in the same daemon, it introduces a
> circle. We need another docker daemon which is dedicated to flannel and
> etcd.
>
> Regards
> wanghua
>
> On Mon, Dec 14, 2015 at 11:45 AM, Steven Dake (stdake) <std...@cisco.com>
> wrote:
>
>> Adrian,
>>
>> Its a real shame Atomic can't execute its mission -  serve as a container
>> operating system.  If you need some guidance on image building find
>> experienced developers on #kolla – we have extensive experience in
>> producing containers for various runtime environments focused around
>> OpenStack.
>>
>> Regards
>> -steve
>>
>>
>> From: Adrian Otto <adrian.o...@rackspace.com>
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> <openstack-dev@lists.openstack.org>
>> Date: Monday, December 7, 2015 at 1:16 PM
>> To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev@lists.openstack.org>
>> Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap
>>
>> Until I see evidence to the contrary, I think adding some bootstrap
>> complexity to simplify the process of bay node image management and
>> customization is worth it. Think about where most users will focus
>> customization efforts. My guess is that it will be within these docker
>> images. We should ask our team to keep things as simple as possible while
>> working to containerize components where that makes sense. That may take
>> some creativity and a few iterations to achieve.
>>
>> We can pivot on this later if we try it and hate it.
>>
>> Thanks,
>>
>> Adrian
>>
>> On Dec 7, 2015, at 1:57 AM, Kai Qiang Wu <wk...@cn.ibm.com> wrote:
>>
>> HI Hua,
>>
>> From my point of view, not everything needed to be put in container.
>> Let's make the initial start (be simple)to work and then discussed other
>> options if needed in IRC or weekly meeting.
>>
>>
>> Thanks
>>
>> Best Wishes,
>>
>> 
>> Kai Qiang Wu (吴开强 Kennan）
>> IBM China System and Technology Lab, Beijing
>>
>> E-mail: wk...@cn.ibm.com
>> Tel: 86-10-82451647
>> Address: Building 28(Ring Building), ZhongGuanCun Software Park,
>> No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193
>>
>> 
>> Follow your heart. You are miracle!
>>
>> 王华 ---07/12/2015 10:10:38 am---Hi all, If we want to run
>> etcd and flannel in container, we will introduce
>>
>> From: 王华 <wanghua.hum...@gmail.com>
>> 

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Adrian Otto]

Wanghua,

I see. The circular dependency you described does sound like a formidable 
challenge. Having multiple docker daemons violates the principle of least 
surprise. I worry that when it comes time to perform troubleshooting, an 
engineer would be surprised to find multiple dockers running at the same time 
within the same compute instance.

Perhaps there is a way to generate the BIP and MTU before the docker daemon is 
started, then use those while starting docker, and start both flannel and etcd 
containers so all containers on the instance can share a single docker daemon? 
Would that work at all? I guess I’d need a better understanding of exactly how 
the BIP and MTU are generated before judging if this is a good idea.

Adrian

On Dec 16, 2015, at 11:40 PM, 王华 
<wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>> wrote:

Adrian,

When the docker daemon starts, it needs to know the bip and mtu which are 
generated by flannel. So flannel and etcd should start before docker daemon, 
but if flannel and etcd run in the same daemon, it introduces a circle. We need 
another docker daemon which is dedicated to flannel and etcd.

Regards
wanghua

On Mon, Dec 14, 2015 at 11:45 AM, Steven Dake (stdake) 
<std...@cisco.com<mailto:std...@cisco.com>> wrote:
Adrian,

Its a real shame Atomic can't execute its mission -  serve as a container 
operating system.  If you need some guidance on image building find experienced 
developers on #kolla – we have extensive experience in producing containers for 
various runtime environments focused around OpenStack.

Regards
-steve


From: Adrian Otto <adrian.o...@rackspace.com<mailto:adrian.o...@rackspace.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, December 7, 2015 at 1:16 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap

Until I see evidence to the contrary, I think adding some bootstrap complexity 
to simplify the process of bay node image management and customization is worth 
it. Think about where most users will focus customization efforts. My guess is 
that it will be within these docker images. We should ask our team to keep 
things as simple as possible while working to containerize components where 
that makes sense. That may take some creativity and a few iterations to achieve.

We can pivot on this later if we try it and hate it.

Thanks,

Adrian

On Dec 7, 2015, at 1:57 AM, Kai Qiang Wu 
<wk...@cn.ibm.com<mailto:wk...@cn.ibm.com>> wrote:


HI Hua,

From my point of view, not everything needed to be put in container. Let's make 
the initial start (be simple)to work and then discussed other options if needed 
in IRC or weekly meeting.


Thanks

Best Wishes,

Kai Qiang Wu (吴开强 Kennan）
IBM China System and Technology Lab, Beijing

E-mail: wk...@cn.ibm.com<mailto:wk...@cn.ibm.com>
Tel: 86-10-82451647
Address: Building 28(Ring Building), ZhongGuanCun Software Park,
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193

Follow your heart. You are miracle!

王华 ---07/12/2015 10:10:38 am---Hi all, If we want to run etcd and 
flannel in container, we will introduce

From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
To: Egor Guz <e...@walmartlabs.com<mailto:e...@walmartlabs.com>>
Cc: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: 07/12/2015 10:10 am
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap





Hi all,

If we want to run etcd and flannel in container, we will introduce 
docker-bootstrap which makes setup become more complex as Egor pointed out. 
Should we pay for the price?

On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz 
<e...@walmartlabs.com<mailto:e...@walmartlabs.com>> wrote:

Wanghua,

I don’t think moving flannel to the container is good idea. This is setup great 
for dev environment, but become too complex from operator point of view (you 
add extra Docker daemon and need extra Cinder volume for this daemon, also
keep in mind it makes sense to keep etcd data folder at Cinder storage as well 
because etcd is database). flannel has just there files without extra 
dependencies and it’s much easy to download it during cloud-init ;)

I agree that we have pain with building Fedora Atomic images, but instead of 
simplify this process we should switch to another

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[王华]

Adrian,

When the docker daemon starts, it needs to know the bip and mtu which are
generated by flannel. So flannel and etcd should start before docker
daemon, but if flannel and etcd run in the same daemon, it introduces a
circle. We need another docker daemon which is dedicated to flannel and
etcd.

Regards
wanghua

On Mon, Dec 14, 2015 at 11:45 AM, Steven Dake (stdake) <std...@cisco.com>
wrote:

> Adrian,
>
> Its a real shame Atomic can't execute its mission -  serve as a container
> operating system.  If you need some guidance on image building find
> experienced developers on #kolla – we have extensive experience in
> producing containers for various runtime environments focused around
> OpenStack.
>
> Regards
> -steve
>
>
> From: Adrian Otto <adrian.o...@rackspace.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Monday, December 7, 2015 at 1:16 PM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap
>
> Until I see evidence to the contrary, I think adding some bootstrap
> complexity to simplify the process of bay node image management and
> customization is worth it. Think about where most users will focus
> customization efforts. My guess is that it will be within these docker
> images. We should ask our team to keep things as simple as possible while
> working to containerize components where that makes sense. That may take
> some creativity and a few iterations to achieve.
>
> We can pivot on this later if we try it and hate it.
>
> Thanks,
>
> Adrian
>
> On Dec 7, 2015, at 1:57 AM, Kai Qiang Wu <wk...@cn.ibm.com> wrote:
>
> HI Hua,
>
> From my point of view, not everything needed to be put in container. Let's
> make the initial start (be simple)to work and then discussed other options
> if needed in IRC or weekly meeting.
>
>
> Thanks
>
> Best Wishes,
>
> 
> Kai Qiang Wu (吴开强 Kennan）
> IBM China System and Technology Lab, Beijing
>
> E-mail: wk...@cn.ibm.com
> Tel: 86-10-82451647
> Address: Building 28(Ring Building), ZhongGuanCun Software Park,
> No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193
>
> 
> Follow your heart. You are miracle!
>
> 王华 ---07/12/2015 10:10:38 am---Hi all, If we want to run
> etcd and flannel in container, we will introduce
>
> From: 王华 <wanghua.hum...@gmail.com>
> To: Egor Guz <e...@walmartlabs.com>
> Cc: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org
> >
> Date: 07/12/2015 10:10 am
> Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap
> --
>
>
>
> Hi all,
>
> If we want to run etcd and flannel in container, we will
> introduce docker-bootstrap which makes setup become more complex as Egor
> pointed out. Should we pay for the price?
>
> On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz <*e...@walmartlabs.com*
> <e...@walmartlabs.com>> wrote:
>
>Wanghua,
>
>I don’t think moving flannel to the container is good idea. This is
>setup great for dev environment, but become too complex from operator point
>of view (you add extra Docker daemon and need extra Cinder volume for this
>daemon, also
>keep in mind it makes sense to keep etcd data folder at Cinder storage
>as well because etcd is database). flannel has just there files without
>extra dependencies and it’s much easy to download it during cloud-init ;)
>
>I agree that we have pain with building Fedora Atomic images, but
>instead of simplify this process we should switch to another more
>“friendly” images (e.g. Fedora/CentOS/Ubuntu) which we can easy build with
>disk builder.
>Also we can fix CoreOS template (I believe people more asked about it
>instead of Atomic), but we may face similar to Atomic issues when we will
>try to integrate not CoreOS products (e.g. Calico or Weave)
>
>—
>Egor
>
>From: 王华 <*wanghua.hum...@gmail.com* <wanghua.hum...@gmail.com>*wanghua.hum...@gmail.com* <wanghua.hum...@gmail.com>>>
>Reply-To: "OpenStack Development Mailing List (not for usage
>questions)" <*openstack-dev@lists.openstack.org*
>    <openstack-dev@lists.openstack.org>*openstack-dev@lists.openstack.org* <openstack-dev@lists.openstack.org>
>>>
>

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Steven Dake (stdake)]

Adrian,

Its a real shame Atomic can't execute its mission -  serve as a container 
operating system.  If you need some guidance on image building find experienced 
developers on #kolla – we have extensive experience in producing containers for 
various runtime environments focused around OpenStack.

Regards
-steve


From: Adrian Otto <adrian.o...@rackspace.com<mailto:adrian.o...@rackspace.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Monday, December 7, 2015 at 1:16 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap

Until I see evidence to the contrary, I think adding some bootstrap complexity 
to simplify the process of bay node image management and customization is worth 
it. Think about where most users will focus customization efforts. My guess is 
that it will be within these docker images. We should ask our team to keep 
things as simple as possible while working to containerize components where 
that makes sense. That may take some creativity and a few iterations to achieve.

We can pivot on this later if we try it and hate it.

Thanks,

Adrian

On Dec 7, 2015, at 1:57 AM, Kai Qiang Wu 
<wk...@cn.ibm.com<mailto:wk...@cn.ibm.com>> wrote:


HI Hua,

From my point of view, not everything needed to be put in container. Let's make 
the initial start (be simple)to work and then discussed other options if needed 
in IRC or weekly meeting.


Thanks

Best Wishes,

Kai Qiang Wu (吴开强 Kennan）
IBM China System and Technology Lab, Beijing

E-mail: wk...@cn.ibm.com<mailto:wk...@cn.ibm.com>
Tel: 86-10-82451647
Address: Building 28(Ring Building), ZhongGuanCun Software Park,
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193

Follow your heart. You are miracle!

王华 ---07/12/2015 10:10:38 am---Hi all, If we want to run etcd and 
flannel in container, we will introduce

From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
To: Egor Guz <e...@walmartlabs.com<mailto:e...@walmartlabs.com>>
Cc: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: 07/12/2015 10:10 am
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap





Hi all,

If we want to run etcd and flannel in container, we will introduce 
docker-bootstrap which makes setup become more complex as Egor pointed out. 
Should we pay for the price?

On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz 
<e...@walmartlabs.com<mailto:e...@walmartlabs.com>> wrote:

Wanghua,

I don’t think moving flannel to the container is good idea. This is setup great 
for dev environment, but become too complex from operator point of view (you 
add extra Docker daemon and need extra Cinder volume for this daemon, also
keep in mind it makes sense to keep etcd data folder at Cinder storage as well 
because etcd is database). flannel has just there files without extra 
dependencies and it’s much easy to download it during cloud-init ;)

I agree that we have pain with building Fedora Atomic images, but instead of 
simplify this process we should switch to another more “friendly” images (e.g. 
Fedora/CentOS/Ubuntu) which we can easy build with disk builder.
Also we can fix CoreOS template (I believe people more asked about it instead 
of Atomic), but we may face similar to Atomic issues when we will try to 
integrate not CoreOS products (e.g. Calico or Weave)

—
Egor

From: 王华 
<wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com><mailto:wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org><mailto:openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>>
Date: Thursday, November 26, 2015 at 00:15
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org><mailto:openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap

Hi Hongbin,

The docker in master node stores data in /dev/mapper/atomicos-docker--data and 
metadata in /dev/mapper/atomicos-docker--meta. 
/dev/mapper/atomicos-docker--data and /dev

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Kai Qiang Wu]

HI Hua,

From my point of view, not everything needed to be put in container. Let's
make the initial start (be simple)to work and then discussed other options
if needed in IRC or weekly meeting.


Thanks

Best Wishes,

Kai Qiang Wu (吴开强  Kennan）
IBM China System and Technology Lab, Beijing

E-mail: wk...@cn.ibm.com
Tel: 86-10-82451647
Address: Building 28(Ring Building), ZhongGuanCun Software Park,
 No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China
100193

Follow your heart. You are miracle!



From:   王华 <wanghua.hum...@gmail.com>
To: Egor Guz <e...@walmartlabs.com>
Cc: "openstack-dev@lists.openstack.org"
<openstack-dev@lists.openstack.org>
Date:   07/12/2015 10:10 am
Subject:    Re: [openstack-dev] [magnum]storage for docker-bootstrap



Hi all,

If we want to run etcd and flannel in container, we will
introduce docker-bootstrap which makes setup become more complex as Egor
pointed out. Should we pay for the price?

On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz <e...@walmartlabs.com> wrote:
  Wanghua,

  I don’t think moving flannel to the container is good idea. This is setup
  great for dev environment, but become too complex from operator point of
  view (you add extra Docker daemon and need extra Cinder volume for this
  daemon, also
  keep in mind it makes sense to keep etcd data folder at Cinder storage as
  well because etcd is database). flannel has just there files without
  extra dependencies and it’s much easy to download it during cloud-init ;)

  I agree that we have pain with building Fedora Atomic images, but instead
  of simplify this process we should switch to another more “friendly”
  images (e.g. Fedora/CentOS/Ubuntu) which we can easy build with disk
  builder.
  Also we can fix CoreOS template (I believe people more asked about it
  instead of Atomic), but we may face similar to Atomic issues when we will
  try to integrate not CoreOS products (e.g. Calico or Weave)

  —
  Egor

  From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
  Reply-To: "OpenStack Development Mailing List (not for usage questions)"
  <openstack-dev@lists.openstack.org>
  Date: Thursday, November 26, 2015 at 00:15
  To: "OpenStack Development Mailing List (not for usage questions)" <
  openstack-dev@lists.openstack.org>
  Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap

  Hi Hongbin,

  The docker in master node stores data
  in /dev/mapper/atomicos-docker--data and metadata
  in /dev/mapper/atomicos-docker--meta. /dev/mapper/atomicos-docker--data
  and /dev/mapper/atomicos-docker--meta are logic volumes. The docker in
  minion node store data in the cinder volume,
  but /dev/mapper/atomicos-docker--meta
  and /dev/mapper/atomicos-docker--meta are not used. If we want to
  leverage Cinder volume for docker in master, should we
  drop /dev/mapper/atomicos-docker--meta
  and /dev/mapper/atomicos-docker--meta? I think it is not necessary to
  allocate a Cinder volume. It is enough to allocate two logic volumes for
  docker, because only etcd, flannel, k8s run in the docker daemon which
  need not a large amount of storage.

  Best regards,
  Wanghua

  On Thu, Nov 26, 2015 at 12:40 AM, Hongbin Lu <hongbin...@huawei.com
  <mailto:hongbin...@huawei.com>> wrote:
  Here is a bit more context.

  Currently, at k8s and swarm bay, some required binaries (i.e. etcd and
  flannel) are built into image and run at host. We are exploring the
  possibility to containerize some of these system components. The
  rationales are (i) it is infeasible to build custom packages into an
  atomic image and (ii) it is infeasible to upgrade individual component.
  For example, if there is a bug in current version of flannel and we know
  the bug was fixed in the next version, we need to upgrade flannel by
  building a new image, which is a tedious process.

  To containerize flannel, we need a second docker daemon, called
  docker-bootstrap [1]. In this setup, pods are running on the main docker
  daemon, and flannel and etcd are running on the second docker daemon. The
  reason is that flannel needs to manage the network of the main docker
  daemon, so it needs to run on a separated daemon.

  Daneyon, I think it requires separated storage because it needs to run a
  separated docker daemon (unless there is a way to make two docker daemons
  share the same storage).

  Wanghua, is it possible to leverage Cinder volume for that. Leveraging
  external storage is always preferred [2].

  [1]
  
http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker

  [2] http://www.projectatomic.io/docs/docker-storage-recommendation/

  Best regards,
  Hongbin

  From: Daneyon Hansen (danehan

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Adrian Otto]

Until I see evidence to the contrary, I think adding some bootstrap complexity 
to simplify the process of bay node image management and customization is worth 
it. Think about where most users will focus customization efforts. My guess is 
that it will be within these docker images. We should ask our team to keep 
things as simple as possible while working to containerize components where 
that makes sense. That may take some creativity and a few iterations to achieve.

We can pivot on this later if we try it and hate it.

Thanks,

Adrian

On Dec 7, 2015, at 1:57 AM, Kai Qiang Wu 
<wk...@cn.ibm.com<mailto:wk...@cn.ibm.com>> wrote:


HI Hua,

From my point of view, not everything needed to be put in container. Let's make 
the initial start (be simple)to work and then discussed other options if needed 
in IRC or weekly meeting.


Thanks

Best Wishes,

Kai Qiang Wu (吴开强 Kennan）
IBM China System and Technology Lab, Beijing

E-mail: wk...@cn.ibm.com<mailto:wk...@cn.ibm.com>
Tel: 86-10-82451647
Address: Building 28(Ring Building), ZhongGuanCun Software Park,
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193

Follow your heart. You are miracle!

王华 ---07/12/2015 10:10:38 am---Hi all, If we want to run etcd and 
flannel in container, we will introduce

From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
To: Egor Guz <e...@walmartlabs.com<mailto:e...@walmartlabs.com>>
Cc: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: 07/12/2015 10:10 am
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap





Hi all,

If we want to run etcd and flannel in container, we will introduce 
docker-bootstrap which makes setup become more complex as Egor pointed out. 
Should we pay for the price?

On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz 
<e...@walmartlabs.com<mailto:e...@walmartlabs.com>> wrote:

Wanghua,

I don’t think moving flannel to the container is good idea. This is setup great 
for dev environment, but become too complex from operator point of view (you 
add extra Docker daemon and need extra Cinder volume for this daemon, also
keep in mind it makes sense to keep etcd data folder at Cinder storage as well 
because etcd is database). flannel has just there files without extra 
dependencies and it’s much easy to download it during cloud-init ;)

I agree that we have pain with building Fedora Atomic images, but instead of 
simplify this process we should switch to another more “friendly” images (e.g. 
Fedora/CentOS/Ubuntu) which we can easy build with disk builder.
Also we can fix CoreOS template (I believe people more asked about it instead 
of Atomic), but we may face similar to Atomic issues when we will try to 
integrate not CoreOS products (e.g. Calico or Weave)

—
Egor

From: 王华 
<wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com><mailto:wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org><mailto:openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>>
Date: Thursday, November 26, 2015 at 00:15
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org><mailto:openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap

Hi Hongbin,

The docker in master node stores data in /dev/mapper/atomicos-docker--data and 
metadata in /dev/mapper/atomicos-docker--meta. 
/dev/mapper/atomicos-docker--data and /dev/mapper/atomicos-docker--meta are 
logic volumes. The docker in minion node store data in the cinder volume, but 
/dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta are not 
used. If we want to leverage Cinder volume for docker in master, should we drop 
/dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta? I 
think it is not necessary to allocate a Cinder volume. It is enough to allocate 
two logic volumes for docker, because only etcd, flannel, k8s run in the docker 
daemon which need not a large amount of storage.

Best regards,
Wanghua

On Thu, Nov 26, 2015 at 12:40 AM, Hongbin Lu 
<hongbin...@huawei.com<mailto:hongbin...@huawei.com><mailto:hongbin...@huawei.com<mailto:hongbin...@huawei.com>>>
 wrote:
Here is a bit more context.

Currently, at k8s and swarm bay, some required binari

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[王华]

Hi all,

If we want to run etcd and flannel in container, we will introduce
docker-bootstrap
which makes setup become more complex as Egor pointed out. Should we pay
for the price?

On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz <e...@walmartlabs.com> wrote:

> Wanghua,
>
> I don’t think moving flannel to the container is good idea. This is setup
> great for dev environment, but become too complex from operator point of
> view (you add extra Docker daemon and need extra Cinder volume for this
> daemon, also
> keep in mind it makes sense to keep etcd data folder at Cinder storage as
> well because etcd is database). flannel has just there files without extra
> dependencies and it’s much easy to download it during cloud-init ;)
>
> I agree that we have pain with building Fedora Atomic images, but instead
> of simplify this process we should switch to another more “friendly” images
> (e.g. Fedora/CentOS/Ubuntu) which we can easy build with disk builder.
> Also we can fix CoreOS template (I believe people more asked about it
> instead of Atomic), but we may face similar to Atomic issues when we will
> try to integrate not CoreOS products (e.g. Calico or Weave)
>
> —
> Egor
>
> From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org
> >>
> Date: Thursday, November 26, 2015 at 00:15
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org
> >>
> Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap
>
> Hi Hongbin,
>
> The docker in master node stores data in /dev/mapper/atomicos-docker--data
> and metadata in /dev/mapper/atomicos-docker--meta.
> /dev/mapper/atomicos-docker--data and /dev/mapper/atomicos-docker--meta are
> logic volumes. The docker in minion node store data in the cinder volume,
> but /dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta
> are not used. If we want to leverage Cinder volume for docker in master,
> should we drop /dev/mapper/atomicos-docker--meta and
> /dev/mapper/atomicos-docker--meta? I think it is not necessary to allocate
> a Cinder volume. It is enough to allocate two logic volumes for docker,
> because only etcd, flannel, k8s run in the docker daemon which need not a
> large amount of storage.
>
> Best regards,
> Wanghua
>
> On Thu, Nov 26, 2015 at 12:40 AM, Hongbin Lu <hongbin...@huawei.com
> <mailto:hongbin...@huawei.com>> wrote:
> Here is a bit more context.
>
> Currently, at k8s and swarm bay, some required binaries (i.e. etcd and
> flannel) are built into image and run at host. We are exploring the
> possibility to containerize some of these system components. The rationales
> are (i) it is infeasible to build custom packages into an atomic image and
> (ii) it is infeasible to upgrade individual component. For example, if
> there is a bug in current version of flannel and we know the bug was fixed
> in the next version, we need to upgrade flannel by building a new image,
> which is a tedious process.
>
> To containerize flannel, we need a second docker daemon, called
> docker-bootstrap [1]. In this setup, pods are running on the main docker
> daemon, and flannel and etcd are running on the second docker daemon. The
> reason is that flannel needs to manage the network of the main docker
> daemon, so it needs to run on a separated daemon.
>
> Daneyon, I think it requires separated storage because it needs to run a
> separated docker daemon (unless there is a way to make two docker daemons
> share the same storage).
>
> Wanghua, is it possible to leverage Cinder volume for that. Leveraging
> external storage is always preferred [2].
>
> [1]
> http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker
> [2] http://www.projectatomic.io/docs/docker-storage-recommendation/
>
> Best regards,
> Hongbin
>
> From: Daneyon Hansen (danehans) [mailto:daneh...@cisco.com daneh...@cisco.com>]
> Sent: November-25-15 11:10 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap
>
>
>
> From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org
> >>
> Date: Wednesday, November 25, 2015 at 5:00 AM
> To: "OpenStack Development Mailing List (no

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Egor Guz]

Wanghua,

I don’t think moving flannel to the container is good idea. This is setup great 
for dev environment, but become too complex from operator point of view (you 
add extra Docker daemon and need extra Cinder volume for this daemon, also
keep in mind it makes sense to keep etcd data folder at Cinder storage as well 
because etcd is database). flannel has just there files without extra 
dependencies and it’s much easy to download it during cloud-init ;)

I agree that we have pain with building Fedora Atomic images, but instead of 
simplify this process we should switch to another more “friendly” images (e.g. 
Fedora/CentOS/Ubuntu) which we can easy build with disk builder.
Also we can fix CoreOS template (I believe people more asked about it instead 
of Atomic), but we may face similar to Atomic issues when we will try to 
integrate not CoreOS products (e.g. Calico or Weave)

―
Egor

From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Thursday, November 26, 2015 at 00:15
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap

Hi Hongbin,

The docker in master node stores data in /dev/mapper/atomicos-docker--data and 
metadata in /dev/mapper/atomicos-docker--meta. 
/dev/mapper/atomicos-docker--data and /dev/mapper/atomicos-docker--meta are 
logic volumes. The docker in minion node store data in the cinder volume, but 
/dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta are not 
used. If we want to leverage Cinder volume for docker in master, should we drop 
/dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta? I 
think it is not necessary to allocate a Cinder volume. It is enough to allocate 
two logic volumes for docker, because only etcd, flannel, k8s run in the docker 
daemon which need not a large amount of storage.

Best regards,
Wanghua

On Thu, Nov 26, 2015 at 12:40 AM, Hongbin Lu 
<hongbin...@huawei.com<mailto:hongbin...@huawei.com>> wrote:
Here is a bit more context.

Currently, at k8s and swarm bay, some required binaries (i.e. etcd and flannel) 
are built into image and run at host. We are exploring the possibility to 
containerize some of these system components. The rationales are (i) it is 
infeasible to build custom packages into an atomic image and (ii) it is 
infeasible to upgrade individual component. For example, if there is a bug in 
current version of flannel and we know the bug was fixed in the next version, 
we need to upgrade flannel by building a new image, which is a tedious process.

To containerize flannel, we need a second docker daemon, called 
docker-bootstrap [1]. In this setup, pods are running on the main docker 
daemon, and flannel and etcd are running on the second docker daemon. The 
reason is that flannel needs to manage the network of the main docker daemon, 
so it needs to run on a separated daemon.

Daneyon, I think it requires separated storage because it needs to run a 
separated docker daemon (unless there is a way to make two docker daemons share 
the same storage).

Wanghua, is it possible to leverage Cinder volume for that. Leveraging external 
storage is always preferred [2].

[1] 
http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker
[2] http://www.projectatomic.io/docs/docker-storage-recommendation/

Best regards,
Hongbin

From: Daneyon Hansen (danehans) 
[mailto:daneh...@cisco.com<mailto:daneh...@cisco.com>]
Sent: November-25-15 11:10 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap



From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, November 25, 2015 at 5:00 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [magnum]storage for docker-bootstrap

Hi all,

I am working on containerizing etcd and flannel. But I met a problem. As 
described in [1], we need a docker-bootstrap. Docker and docker-bootstrap can 
not use the same storage, so we need some disk space for it.

I reviewed [1] and I do not see where the bootstrap docker instance requires 
separate storage.

The docker in master node stores data in /dev/mapper/atomicos-docker--data and 
metadata in /dev/mapper/atomicos-docker--meta. The disk space left is too same 
for docker-

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[王华]

Hi Hongbin,

The docker in master node stores data in /dev/mapper/atomicos-docker--data
and metadata in /dev/mapper/atomicos-docker--meta.
/dev/mapper/atomicos-docker--data
and /dev/mapper/atomicos-docker--meta are logic volumes. The docker in
minion node store data in the cinder volume, but
/dev/mapper/atomicos-docker--meta
and /dev/mapper/atomicos-docker--meta are not used. If we want to leverage
Cinder volume for docker in master, should we drop
/dev/mapper/atomicos-docker--meta
and /dev/mapper/atomicos-docker--meta? I think it is not necessary to
allocate a Cinder volume. It is enough to allocate two logic volumes for
docker, because only etcd, flannel, k8s run in the docker daemon which need
not a large amount of storage.

Best regards,
Wanghua

On Thu, Nov 26, 2015 at 12:40 AM, Hongbin Lu <hongbin...@huawei.com> wrote:

> Here is a bit more context.
>
>
>
> Currently, at k8s and swarm bay, some required binaries (i.e. etcd and
> flannel) are built into image and run at host. We are exploring the
> possibility to containerize some of these system components. The rationales
> are (i) it is infeasible to build custom packages into an atomic image and
> (ii) it is infeasible to upgrade individual component. For example, if
> there is a bug in current version of flannel and we know the bug was fixed
> in the next version, we need to upgrade flannel by building a new image,
> which is a tedious process.
>
>
>
> To containerize flannel, we need a second docker daemon, called
> docker-bootstrap [1]. In this setup, pods are running on the main docker
> daemon, and flannel and etcd are running on the second docker daemon. The
> reason is that flannel needs to manage the network of the main docker
> daemon, so it needs to run on a separated daemon.
>
>
>
> Daneyon, I think it requires separated storage because it needs to run a
> separated docker daemon (unless there is a way to make two docker daemons
> share the same storage).
>
>
>
> Wanghua, is it possible to leverage Cinder volume for that. Leveraging
> external storage is always preferred [2].
>
>
>
> [1]
> http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker
>
> [2] http://www.projectatomic.io/docs/docker-storage-recommendation/
>
>
>
> Best regards,
>
> Hongbin
>
>
>
> *From:* Daneyon Hansen (danehans) [mailto:daneh...@cisco.com]
> *Sent:* November-25-15 11:10 AM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [magnum]storage for docker-bootstrap
>
>
>
>
>
>
>
> *From: *王华 <wanghua.hum...@gmail.com>
> *Reply-To: *"OpenStack Development Mailing List (not for usage
> questions)" <openstack-dev@lists.openstack.org>
> *Date: *Wednesday, November 25, 2015 at 5:00 AM
> *To: *"OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> *Subject: *[openstack-dev] [magnum]storage for docker-bootstrap
>
>
>
> Hi all,
>
>
>
> I am working on containerizing etcd and flannel. But I met a problem. As
> described in [1], we need a docker-bootstrap. Docker and docker-bootstrap
> can not use the same storage, so we need some disk space for it.
>
>
>
> I reviewed [1] and I do not see where the bootstrap docker instance
> requires separate storage.
>
>
>
> The docker in master node stores data in /dev/mapper/atomicos-docker--data
> and metadata in /dev/mapper/atomicos-docker--meta. The disk space left is
> too same for docker-bootstrap. Even if the root_gb of the instance flavor
> is 20G, only 8G can be used in our image. I want to make it bigger. One way
> is we can add the disk space left in the vda as vda3 into atomicos vg after
> the instance starts and we allocate two logic volumes for docker-bootstrap.
> Another way is when we create the image, we allocate two logic volumes for
> docker-bootstrap. The second way has a advantage. It doesn't have to make
> filesystem when the instance is created which is time consuming.
>
>
>
> What is your opinion?
>
>
>
> Best Regards
>
> Wanghua
>
>
>
> [1]
> http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode/master.html
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Daneyon Hansen (danehans)]

From: 王华 >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Wednesday, November 25, 2015 at 5:00 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: [openstack-dev] [magnum]storage for docker-bootstrap

Hi all,

I am working on containerizing etcd and flannel. But I met a problem. As 
described in [1], we need a docker-bootstrap. Docker and docker-bootstrap can 
not use the same storage, so we need some disk space for it.

I reviewed [1] and I do not see where the bootstrap docker instance requires 
separate storage.

The docker in master node stores data in /dev/mapper/atomicos-docker--data and 
metadata in /dev/mapper/atomicos-docker--meta. The disk space left is too same 
for docker-bootstrap. Even if the root_gb of the instance flavor is 20G, only 
8G can be used in our image. I want to make it bigger. One way is we can add 
the disk space left in the vda as vda3 into atomicos vg after the instance 
starts and we allocate two logic volumes for docker-bootstrap. Another way is 
when we create the image, we allocate two logic volumes for docker-bootstrap. 
The second way has a advantage. It doesn't have to make filesystem when the 
instance is created which is time consuming.

What is your opinion?

Best Regards
Wanghua

[1] 
http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode/master.html
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum]storage for docker-bootstrap

[Hongbin Lu]

Here is a bit more context.

Currently, at k8s and swarm bay, some required binaries (i.e. etcd and flannel) 
are built into image and run at host. We are exploring the possibility to 
containerize some of these system components. The rationales are (i) it is 
infeasible to build custom packages into an atomic image and (ii) it is 
infeasible to upgrade individual component. For example, if there is a bug in 
current version of flannel and we know the bug was fixed in the next version, 
we need to upgrade flannel by building a new image, which is a tedious process.

To containerize flannel, we need a second docker daemon, called 
docker-bootstrap [1]. In this setup, pods are running on the main docker 
daemon, and flannel and etcd are running on the second docker daemon. The 
reason is that flannel needs to manage the network of the main docker daemon, 
so it needs to run on a separated daemon.

Daneyon, I think it requires separated storage because it needs to run a 
separated docker daemon (unless there is a way to make two docker daemons share 
the same storage).

Wanghua, is it possible to leverage Cinder volume for that. Leveraging external 
storage is always preferred [2].

[1] 
http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker
[2] http://www.projectatomic.io/docs/docker-storage-recommendation/

Best regards,
Hongbin

From: Daneyon Hansen (danehans) [mailto:daneh...@cisco.com]
Sent: November-25-15 11:10 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap



From: 王华 <wanghua.hum...@gmail.com<mailto:wanghua.hum...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, November 25, 2015 at 5:00 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [magnum]storage for docker-bootstrap

Hi all,

I am working on containerizing etcd and flannel. But I met a problem. As 
described in [1], we need a docker-bootstrap. Docker and docker-bootstrap can 
not use the same storage, so we need some disk space for it.

I reviewed [1] and I do not see where the bootstrap docker instance requires 
separate storage.

The docker in master node stores data in /dev/mapper/atomicos-docker--data and 
metadata in /dev/mapper/atomicos-docker--meta. The disk space left is too same 
for docker-bootstrap. Even if the root_gb of the instance flavor is 20G, only 
8G can be used in our image. I want to make it bigger. One way is we can add 
the disk space left in the vda as vda3 into atomicos vg after the instance 
starts and we allocate two logic volumes for docker-bootstrap. Another way is 
when we create the image, we allocate two logic volumes for docker-bootstrap. 
The second way has a advantage. It doesn't have to make filesystem when the 
instance is created which is time consuming.

What is your opinion?

Best Regards
Wanghua

[1] 
http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode/master.html
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev