Re: [openstack-dev] [Nova][Docker] Environment variables
On 17 December 2013 12:53, Daniel P. Berrange berra...@redhat.com wrote: On Mon, Dec 16, 2013 at 01:04:33PM -0800, Dan Smith wrote: eg use a 'env_' prefix for glance image attributes We've got a couple of cases now where we want to overrides these same things on a per-instance basis. Kernel command line args is one other example. Other hardware overrides like disk/net device types are another possibility Rather than invent new extensions for each, I think we should have a way to pass arbitrary attributes alon with the boot API call, that a driver would handle in much the same way as they do for glance image properties. Basically think of it as a way to custom any image property per instance created. Personally, I think having a bunch of special case magic namespaces (even if documented) is less desirable than a proper API to do something like this. Especially a namespace that someone else could potentially use legitimately that would conflict. To me, this feels a lot like what I'm worried this effort will turn into, which is making containers support in Nova look like a bolt-on thing with a bunch of specialness required to make it behave. NB I'm not saying that everything related to containers should be done with metadata properties. I just feel that this is appropriate for setting of environment variables and some other things like kernel command line args, since it gives a consistent approach to use for setting those per-image vs per-instance. +1 it seems a fairly nice mapping for kernel args and environment variables. Cloud-Init could add the environment variable inside VMs if we felt that way inclined. Discoverability isn't awesome though. Anyone remember this bolt-on gem? nova boot --block-device-mapping vda=965453c9-02b5-4d5b-8ec0-3164a89bf6f4:::0 --flavor=m1.tiny --image=6415797a-7c03-45fe-b490-f9af99d2bae0 BFV I found that one amidst hundreds of forum threads of people confused about what incantation of magic they were supposed to do to make it actually boot from volume. Everything about the way you use block device mapping is plain awful - even the bits that were done as proper API extensions. I don't think the design failures there apply in this case. If we do env variables as metadata properties, then you may well not end up even needing to pass them to 'nova boot' in the common case, since it'll likely be sufficient to have them just set against the image in glance most of the time. +1 Going further, we set PV vs HVM via image properties. It would be nice to override that on a per boot basis that matches these other cases. Some generic way of setting a per-boot equivalent of an image property might be the best approach? Going back to glance protected properties, we would need a Nova equivalent. But prehaps a whitelist of properties you can override on boot would be best? John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
Actually Daniel P. Barrange comment is interesting. He states that a configuration per instance would also be beneficial for Cinder. The configuration is essentially needed to change the bootstrap of the image. If you look at a docker image in abstract way then that is the same thing - a configuration to bootstrap the image differently. From user point of view it could look like: nova boot --image-meta key=value Daniel On Mon, Dec 16, 2013 at 10:04 PM, Dan Smith d...@danplanet.com wrote: eg use a 'env_' prefix for glance image attributes We've got a couple of cases now where we want to overrides these same things on a per-instance basis. Kernel command line args is one other example. Other hardware overrides like disk/net device types are another possibility Rather than invent new extensions for each, I think we should have a way to pass arbitrary attributes alon with the boot API call, that a driver would handle in much the same way as they do for glance image properties. Basically think of it as a way to custom any image property per instance created. Personally, I think having a bunch of special case magic namespaces (even if documented) is less desirable than a proper API to do something like this. Especially a namespace that someone else could potentially use legitimately that would conflict. To me, this feels a lot like what I'm worried this effort will turn into, which is making containers support in Nova look like a bolt-on thing with a bunch of specialness required to make it behave. Anyone remember this bolt-on gem? nova boot --block-device-mapping vda=965453c9-02b5-4d5b-8ec0-3164a89bf6f4:::0 --flavor=m1.tiny --image=6415797a-7c03-45fe-b490-f9af99d2bae0 BFV I found that one amidst hundreds of forum threads of people confused about what incantation of magic they were supposed to do to make it actually boot from volume. Just MHO. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
On Mon, Dec 16, 2013 at 01:04:33PM -0800, Dan Smith wrote: eg use a 'env_' prefix for glance image attributes We've got a couple of cases now where we want to overrides these same things on a per-instance basis. Kernel command line args is one other example. Other hardware overrides like disk/net device types are another possibility Rather than invent new extensions for each, I think we should have a way to pass arbitrary attributes alon with the boot API call, that a driver would handle in much the same way as they do for glance image properties. Basically think of it as a way to custom any image property per instance created. Personally, I think having a bunch of special case magic namespaces (even if documented) is less desirable than a proper API to do something like this. Especially a namespace that someone else could potentially use legitimately that would conflict. To me, this feels a lot like what I'm worried this effort will turn into, which is making containers support in Nova look like a bolt-on thing with a bunch of specialness required to make it behave. NB I'm not saying that everything related to containers should be done with metadata properties. I just feel that this is appropriate for setting of environment variables and some other things like kernel command line args, since it gives a consistent approach to use for setting those per-image vs per-instance. Anyone remember this bolt-on gem? nova boot --block-device-mapping vda=965453c9-02b5-4d5b-8ec0-3164a89bf6f4:::0 --flavor=m1.tiny --image=6415797a-7c03-45fe-b490-f9af99d2bae0 BFV I found that one amidst hundreds of forum threads of people confused about what incantation of magic they were supposed to do to make it actually boot from volume. Everything about the way you use block device mapping is plain awful - even the bits that were done as proper API extensions. I don't think the design failures there apply in this case. If we do env variables as metadata properties, then you may well not end up even needing to pass them to 'nova boot' in the common case, since it'll likely be sufficient to have them just set against the image in glance most of the time. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
On 12/16/2013 09:27 AM, Daniel Kuffner wrote: Hi All, I have submitted a new blueprint which addresses the a common pattern in the docker world. A usual pattern in the docker world is to use environment variables to configure a container. docker run -e SQL_URL=postgres://user:password@/db my-app The nova docker driver doesn't support to set any environment variables. To work around this issue I used cloud-init which works fine. But this approach has of course the drawback that a) I have to install the cloud init service. and b) my docker container doesn't work outside of openstack. I propose to allow a user to set docker environment variables via nova instance metadata. The metadata key should have a prefix like ENV_ which can be used to determine all environment variables. The prefix should be removed and the remainder key and vaule will be injected. The metadata can unfortunately not be set in horizon but can be used from the nova command line tool and from heat. Example heat: myapp: Type: OS::Nova::Server Properties: flavor: m1.small image: my-app:latest meta-data: - ENV_SQL_URL: postgres://user:password@/db - ENV_SOMETHING_ELSE: Value Let me know what you think about that. Blueprint: https://blueprints.launchpad.net/nova/+spec/docker-env-via-meta-data Thanks for starting the discussion. More people should do this for their blueprints. :-) One of the things we should be striving for is to provide as consistent of an experience as we can across drivers. Right now, we have the metadata service and config drive, and neither of those are driver specific. In the case of config drive, whether it's used or not is exposed through the API. As you point out, the meta-data service does technically work with the docker driver. I don't think we should support environment variables like this automatically. Instead, I think it would be more appropriate to add an API extension for specifying env vars. That way the behavior is more explicit and communicated through the API. The env vars would be passed through all of the appropriate plumbing and down to drivers that are able to support it. This is all also assuming that containers support is staying in Nova and not a new service. That discussion seems to have stalled. Is anyone still pushing on that? Any updates? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
Hi Russel, I have something that is pushing it for to stay in nova (at least the compute drivers). I should have a gerrit branch for people to review soon. Regards chuck On Mon, Dec 16, 2013 at 10:07 AM, Russell Bryant rbry...@redhat.com wrote: On 12/16/2013 09:27 AM, Daniel Kuffner wrote: Hi All, I have submitted a new blueprint which addresses the a common pattern in the docker world. A usual pattern in the docker world is to use environment variables to configure a container. docker run -e SQL_URL=postgres://user:password@/db my-app The nova docker driver doesn't support to set any environment variables. To work around this issue I used cloud-init which works fine. But this approach has of course the drawback that a) I have to install the cloud init service. and b) my docker container doesn't work outside of openstack. I propose to allow a user to set docker environment variables via nova instance metadata. The metadata key should have a prefix like ENV_ which can be used to determine all environment variables. The prefix should be removed and the remainder key and vaule will be injected. The metadata can unfortunately not be set in horizon but can be used from the nova command line tool and from heat. Example heat: myapp: Type: OS::Nova::Server Properties: flavor: m1.small image: my-app:latest meta-data: - ENV_SQL_URL: postgres://user:password@/db - ENV_SOMETHING_ELSE: Value Let me know what you think about that. Blueprint: https://blueprints.launchpad.net/nova/+spec/docker-env-via-meta-data Thanks for starting the discussion. More people should do this for their blueprints. :-) One of the things we should be striving for is to provide as consistent of an experience as we can across drivers. Right now, we have the metadata service and config drive, and neither of those are driver specific. In the case of config drive, whether it's used or not is exposed through the API. As you point out, the meta-data service does technically work with the docker driver. I don't think we should support environment variables like this automatically. Instead, I think it would be more appropriate to add an API extension for specifying env vars. That way the behavior is more explicit and communicated through the API. The env vars would be passed through all of the appropriate plumbing and down to drivers that are able to support it. This is all also assuming that containers support is staying in Nova and not a new service. That discussion seems to have stalled. Is anyone still pushing on that? Any updates? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
On 12/16/2013 10:12 AM, Chuck Short wrote: I have something that is pushing it for to stay in nova (at least the compute drivers). I should have a gerrit branch for people to review soon. OK. Do you have any design notes for whatever you're proposing? That would probably be easier to review and discuss. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
Hi Russell, You actually propose to extend the whole nova stack to support environment variables. Would any other driver benefit from this API extension? Is that what you imagine? nova --env SQL_URL=postgres://user:password --image Regarding the discussion you mentioned. Are there any public resources to read. I kind of missed it. Most likely it was before I was part of this community :) thanks, Daniel On Mon, Dec 16, 2013 at 4:07 PM, Russell Bryant rbry...@redhat.com wrote: On 12/16/2013 09:27 AM, Daniel Kuffner wrote: Hi All, I have submitted a new blueprint which addresses the a common pattern in the docker world. A usual pattern in the docker world is to use environment variables to configure a container. docker run -e SQL_URL=postgres://user:password@/db my-app The nova docker driver doesn't support to set any environment variables. To work around this issue I used cloud-init which works fine. But this approach has of course the drawback that a) I have to install the cloud init service. and b) my docker container doesn't work outside of openstack. I propose to allow a user to set docker environment variables via nova instance metadata. The metadata key should have a prefix like ENV_ which can be used to determine all environment variables. The prefix should be removed and the remainder key and vaule will be injected. The metadata can unfortunately not be set in horizon but can be used from the nova command line tool and from heat. Example heat: myapp: Type: OS::Nova::Server Properties: flavor: m1.small image: my-app:latest meta-data: - ENV_SQL_URL: postgres://user:password@/db - ENV_SOMETHING_ELSE: Value Let me know what you think about that. Blueprint: https://blueprints.launchpad.net/nova/+spec/docker-env-via-meta-data Thanks for starting the discussion. More people should do this for their blueprints. :-) One of the things we should be striving for is to provide as consistent of an experience as we can across drivers. Right now, we have the metadata service and config drive, and neither of those are driver specific. In the case of config drive, whether it's used or not is exposed through the API. As you point out, the meta-data service does technically work with the docker driver. I don't think we should support environment variables like this automatically. Instead, I think it would be more appropriate to add an API extension for specifying env vars. That way the behavior is more explicit and communicated through the API. The env vars would be passed through all of the appropriate plumbing and down to drivers that are able to support it. This is all also assuming that containers support is staying in Nova and not a new service. That discussion seems to have stalled. Is anyone still pushing on that? Any updates? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
Hi Chuck, yes please, I'm eager to see what you have. :) Daniel On Mon, Dec 16, 2013 at 4:12 PM, Chuck Short chuck.sh...@canonical.com wrote: Hi Russel, I have something that is pushing it for to stay in nova (at least the compute drivers). I should have a gerrit branch for people to review soon. Regards chuck On Mon, Dec 16, 2013 at 10:07 AM, Russell Bryant rbry...@redhat.com wrote: On 12/16/2013 09:27 AM, Daniel Kuffner wrote: Hi All, I have submitted a new blueprint which addresses the a common pattern in the docker world. A usual pattern in the docker world is to use environment variables to configure a container. docker run -e SQL_URL=postgres://user:password@/db my-app The nova docker driver doesn't support to set any environment variables. To work around this issue I used cloud-init which works fine. But this approach has of course the drawback that a) I have to install the cloud init service. and b) my docker container doesn't work outside of openstack. I propose to allow a user to set docker environment variables via nova instance metadata. The metadata key should have a prefix like ENV_ which can be used to determine all environment variables. The prefix should be removed and the remainder key and vaule will be injected. The metadata can unfortunately not be set in horizon but can be used from the nova command line tool and from heat. Example heat: myapp: Type: OS::Nova::Server Properties: flavor: m1.small image: my-app:latest meta-data: - ENV_SQL_URL: postgres://user:password@/db - ENV_SOMETHING_ELSE: Value Let me know what you think about that. Blueprint: https://blueprints.launchpad.net/nova/+spec/docker-env-via-meta-data Thanks for starting the discussion. More people should do this for their blueprints. :-) One of the things we should be striving for is to provide as consistent of an experience as we can across drivers. Right now, we have the metadata service and config drive, and neither of those are driver specific. In the case of config drive, whether it's used or not is exposed through the API. As you point out, the meta-data service does technically work with the docker driver. I don't think we should support environment variables like this automatically. Instead, I think it would be more appropriate to add an API extension for specifying env vars. That way the behavior is more explicit and communicated through the API. The env vars would be passed through all of the appropriate plumbing and down to drivers that are able to support it. This is all also assuming that containers support is staying in Nova and not a new service. That discussion seems to have stalled. Is anyone still pushing on that? Any updates? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
That would be great, I have also a couple of change request waiting for approval. Would be good to know if it has any relevance in the future. https://review.openstack.org/#/c/59824/ https://review.openstack.org/#/c/62182/ https://review.openstack.org/#/c/62183/ https://review.openstack.org/#/c/62220/ Daniel On Mon, Dec 16, 2013 at 4:17 PM, Russell Bryant rbry...@redhat.com wrote: On 12/16/2013 10:12 AM, Chuck Short wrote: I have something that is pushing it for to stay in nova (at least the compute drivers). I should have a gerrit branch for people to review soon. OK. Do you have any design notes for whatever you're proposing? That would probably be easier to review and discuss. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
On 12/16/2013 10:18 AM, Daniel Kuffner wrote: Hi Russell, You actually propose to extend the whole nova stack to support environment variables. Would any other driver benefit from this API extension? Is that what you imagine? nova --env SQL_URL=postgres://user:password --image Yes. Regarding the discussion you mentioned. Are there any public resources to read. I kind of missed it. Most likely it was before I was part of this community :) It started here back in November: http://lists.openstack.org/pipermail/openstack-dev/2013-November/019637.html and then there have been a few messages on that thread this month, too. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
On Mon, Dec 16, 2013 at 04:18:52PM +0100, Daniel Kuffner wrote: Hi Russell, You actually propose to extend the whole nova stack to support environment variables. Would any other driver benefit from this API extension? Is that what you imagine? nova --env SQL_URL=postgres://user:password --image Regarding the discussion you mentioned. Are there any public resources to read. I kind of missed it. Most likely it was before I was part of this community :) With glance images we have a way to associate arbitrary metadata attributes with the image. I could see using this mechanism to associate some default set of environment variables. eg use a 'env_' prefix for glance image attributes We've got a couple of cases now where we want to overrides these same things on a per-instance basis. Kernel command line args is one other example. Other hardware overrides like disk/net device types are another possibility Rather than invent new extensions for each, I think we should have a way to pass arbitrary attributes alon with the boot API call, that a driver would handle in much the same way as they do for glance image properties. Basically think of it as a way to custom any image property per instance created. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
On 12/16/2013 10:39 AM, Daniel P. Berrange wrote: On Mon, Dec 16, 2013 at 04:18:52PM +0100, Daniel Kuffner wrote: Hi Russell, You actually propose to extend the whole nova stack to support environment variables. Would any other driver benefit from this API extension? Is that what you imagine? nova --env SQL_URL=postgres://user:password --image Regarding the discussion you mentioned. Are there any public resources to read. I kind of missed it. Most likely it was before I was part of this community :) With glance images we have a way to associate arbitrary metadata attributes with the image. I could see using this mechanism to associate some default set of environment variables. eg use a 'env_' prefix for glance image attributes We've got a couple of cases now where we want to overrides these same things on a per-instance basis. Kernel command line args is one other example. Other hardware overrides like disk/net device types are another possibility Rather than invent new extensions for each, I think we should have a way to pass arbitrary attributes alon with the boot API call, that a driver would handle in much the same way as they do for glance image properties. Basically think of it as a way to custom any image property per instance created. That's a pretty nice idea. I like it. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Docker] Environment variables
eg use a 'env_' prefix for glance image attributes We've got a couple of cases now where we want to overrides these same things on a per-instance basis. Kernel command line args is one other example. Other hardware overrides like disk/net device types are another possibility Rather than invent new extensions for each, I think we should have a way to pass arbitrary attributes alon with the boot API call, that a driver would handle in much the same way as they do for glance image properties. Basically think of it as a way to custom any image property per instance created. Personally, I think having a bunch of special case magic namespaces (even if documented) is less desirable than a proper API to do something like this. Especially a namespace that someone else could potentially use legitimately that would conflict. To me, this feels a lot like what I'm worried this effort will turn into, which is making containers support in Nova look like a bolt-on thing with a bunch of specialness required to make it behave. Anyone remember this bolt-on gem? nova boot --block-device-mapping vda=965453c9-02b5-4d5b-8ec0-3164a89bf6f4:::0 --flavor=m1.tiny --image=6415797a-7c03-45fe-b490-f9af99d2bae0 BFV I found that one amidst hundreds of forum threads of people confused about what incantation of magic they were supposed to do to make it actually boot from volume. Just MHO. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
